How to effectively list 100,000 parts on 30bz

[dynambee]

Soon I will need to list ~100,000 parts on a 30bz site. This will be for ~1100 products that have ±100 parts each. If this goes well the number of parts will eventually reach about 500,000.

I want to list all ~100 parts for one product on one page with an exploded parts diagram of that product (ideally zoomable) so customers can figure out what part they need to order. The parts themselves have no description, only a short name (1 to 3 words -- think "washer", "bolt", "spring retainer", etc) and a 10 digit part number.

I do not want these 100,000 parts to appear in search results as it would make it impossible to find anything that isn't a part. That seems easy enough to do by setting "catalog only" for each part in the back office.

I have come up with two possible solutions for this but I may be missing something and would like some feedback:

1. Create a separate category for each product and put the ~100 parts for that product into that category. Then use @lesley's "Flex Layout Module" to enforce a custom layout for these category pages. This has an added benefit that there are a few key parts (1 or 2 per product) that we actually would like to appear in search results and this would allow us to do this.

2. Use product attributes. Create a "parts product" for each product we sell parts for. ~1100 "parts products" at first, eventually perhaps up to 5000. Each of these "parts products" would contain a single attribute ("part type") with ~100 different options to choose from. Once again use @lesley's "Flex Layout Module" to design a custom layout for these products so instead of showing a huge dropdown list of 100 different choices it breaks them out onto the page itself with an "add to cart" button beside each attribute choice.

I'm not sure if I am explaining the above properly and I'm also not sure if either option is even possible. My questions though:

• Assuming both options are actually possible is one better than the other?
• Is having 5000 categories a problem?
• How about having 5000 different products each with one product attribute containing 100 different options?
• Is it realistic to use custom layouts like I described above?
• Is there a better way to do this that I haven't considered?

Help appreciated.

[moy2010]

Hi, Dynambee. Could you please be more explicit? I mean, could you give us an example of "a product" and "a part"? Like if a product is a car engine, and a part is a bolt, etc.

[dynambee]

Thanks for your reply!

The products are fishing reels. Each reel has somewhere between 50 and 200 parts. Some of the reels have been discontinued already but parts are still available. In this case we wouldn't be selling the reel itself, only the parts. In other cases we sell both the reel and the parts.

Typical parts might be "spring", "bolt", "washer", "spool", "crank", "drag knob", "crank handle", "click pawl", "level wind", "electric motor", etc. There are about 1800 unique part names that we are currently translating from Japanese to English but there could be 100s of different parts with exactly the same name. Part numbers are of course unique.

There is no problem managing the parts themselves as this will be automated, and there is no need for individual photos of the parts (which is good because I'd go insane trying to manage that...)

The issue I face is finding the best way to present these parts to customers. Customers often buy multiple parts from one reel at the same time so having all the parts for one reel on one customer-facing page is important. Being able to quickly add different parts to the cart is important too.

As far as the items we would like to appear in search this would basically be spools and maybe cranks. Spools in particular are things that people often buy for various reasons besides repairs. When someone searches for a given reel, seeing that they can also buy a spare spool is quite valuable to us. If using variations is a better solution than using categories then I can either list the spool as both a variation and then again as a separate product or it could just be a separate product and not listed as a part.

I hope this helps clarify things. Happy to provide more details as needed, of course.

[moy2010]

Got it. Here are a few questions:

• Do products share the parts? I mean, can the same part be found on more that 1 product?
• Will you need to handle stock quantities for the parts?
• How will you calculate the stock for the products?

[dynambee]

Thank you for your reply and assistance.

Do products share the parts? I mean, can the same part be found on more that 1 product?

Well, yes and no.

Yes, there are many parts shared between different reels. However the way the part numbers are provided to us, a dealer, it appears that each one is unique. So on 30bz each part will be unique and there will be no parts that appear in more than one reel.

Will you need to handle stock quantities for the parts?

Due to the extremely large variety of parts we do not stock these items -- no dealer in Japan does, actually. The manufacturer stocks them and ships them out when orders come in. The manufacturer is very good about keeping items in stock but really terrible about sharing exact stock information. Therefore we sell these items "blind" and explain this to customers. About 1% of orders can't be filled, in our experience. In this case the customer gets the option to wait until stock comes in or they can cancel their order and receive a refund.

How will you calculate the stock for the products?

We'll likely just set available stock to be "5" or "10" for each part and automatically reset it after we ship each order. This will be an automated process.

[moy2010]

Consider the following:

• You cannot handle stock quantities for attributes, nor to sell them separately. You'll have to handle the parts as products. The same goes for the prices.
• You can decide what will and will not be indexed with ES for search purposes.
• As for displaying the parts within a reel product page, there are several ways for achieving this.

@dynambee said in How to effectively list 100,000 parts on 30bz:

Is having 5000 categories a problem? How about having 5000 different products each with one product attribute containing 100 different options?

Having that many categories is not a problem, but I suggest to have a clear products taxonomy before launching the ecommerce store. Otherwise, you'll suffer much, much more. I.e., it's easier to decide which product categories will be indexed or queried for full text search.

On the other side, having that many attributes per product will undoubtedly lead to a combinatorial explosion that will hurt your website performance really bad. Again, my recommendation is to upload the parts as products.

[Havouza]

One detail in this is in my humble opinion not correct I can sell an attribute only but if it is good here I dont know I sell a book in 2 different formats. paper, ebook two attributes 1. Format: paperbook / no paperbook 2 Ebook: kindle/epub/pdf/no ebook

Here you can buy only the attribute

I had it setup in a test store but unfortunately it is gone

[lesley]

Your only real option is to sell them as products. They are not an attribute of a product.

As far as the amount, yes, it can handle it, that is not to say you can handle it on a 2 core vps server, but a setup can be made to handle it.

I can show you how to bring back the scenes area which will allow for having schematics with links, but it is not responsive or zoomable, that would need to be custom written into the theme.

As far as excluding the products from search, that would take something custom as well. A module would need to be developed that overrode the search and limited the categories that it would index.

It is all possible though.

[dynambee]

@lesley, @moy2010, thank you very much for your kind assistance & knowledge.

I'd heard that too many different attributes & combinations is bad for performance but i wasn't sure just how bad it might be.

@lesley said in How to effectively list 100,000 parts on 30bz:

Your only real option is to sell them as products. They are not an attribute of a product.

Well that is definitely settled then!

As far as the amount, yes, it can handle it, that is not to say you can handle it on a 2 core vps server, but a setup can be made to handle it.

Would all these parts really place much load on the server? The parts will have no photos, no descriptions, very short names, a part number, and won't be indexed for search. All 500,000 parts items combined seem unlikely to add even 10MB to the database. I could easily be missing something here, even something obvious, so any information & ideas would be appreciated.

I can show you how to bring back the scenes area which will allow for having schematics with links, but it is not responsive or zoomable, that would need to be custom written into the theme.

Due to the number of products & parts I don't plan to do anything fancy. The system just needs to show the schematic image at the top and then all the parts listed numerically below it, probably in two columns. The parts diagrams are labelled numerically starting with 001 so sorting won't be big deal. Beside each part I'd like the customer to be able to choose the quantity to purchase and beside that an add-to-cart button.

As far as excluding the products from search, that would take something custom as well. A module would need to be developed that overrode the search and limited the categories that it would index.

Are items that are marked as "catalog only" in the back office still indexed for search? Looking at the tb_product database table, changing a product's visibility setting from "both" to "catalog only" changes two fields in the db: indexed changes from 1 to 0 and visibility changes from 'both' to 'catalog'.

It is all possible though.

Well I'm certainly going to make a site that will be both a test case and a future example!

[dynambee]

@moy2010 said in How to effectively list 100,000 parts on 30bz:

Consider the following:

• You cannot handle stock quantities for attributes, nor to sell them separately. You'll have to handle the parts as products. The same goes for the prices.
• You can decide what will and will not be indexed with ES for search purposes.

Definitely clear now that the individual parts need to be listed as separate products on the 30bz site. This works well for me as the way PS/30bz is set up it is easier to manage products than it is to manage variations.

• As for displaying the parts within a reel product page, there are several ways for achieving this.

Many of the reels we sell parts for are no longer in production and as such won't have a product page on our website. With this in mind I would sooner have reel parts grouped by category and use custom category page layouts so customers can see the list of parts and quickly add them to their shopping cart.

Here is a link to a reel parts page from one of my competitors: http://nullrefer.com/?http://bit.ly/2s94mrz (Please excuse the redirected & shortened link, I would sooner their logs not point back to this thread. ;))

Their site design leaves quite a lot to be desired but that is the general basic idea that I am considering. Of course what I want to do will be to make it look nicer and to be a responsive layout for mobile users too.

Of course what I am thinking of does not have to be done as a category page and I would be very happy with other potential solutions too.

One thing I have thought about but I suspect may be a PITA to implement would be to have a separate search for parts. When customers enter the "parts area" of the website they get a second search box where they can search for a reel to buy parts for it.

• One way I can think of to accomplish this would be to create a separate ES index just for the names of the parts categories (rather than the parts themselves.) Users could then proceed to the category of the reel they are interested in and find the right part from the exploded diagram and parts list. All parts categories would be contained within one sub tree of categories so it would be easy to index them.

• Another option would be to create "dummy products" in the tb_product database. This is a little difficult to describe well, but the idea would be that it's probably easier to index products than it is to index category names, as far as being able to use search in a standard way and without making modifications.
  \ So to use the same product linked to above on the competitor's site, I would create a dummy product called "15 Saltiga 4500". This product would have no stock and would use a custom page layout that removes things like "out of stock", "price", "add to cart", etc. Instead when a customer visits this product page they would see the zoomable parts diagram and a list of parts more nicely formatted than what is on the competitor's page. The customer can then add the needed parts to their cart and proceed with checkout. \ Another advantage to doing this is that the list of parts to include on the page could be stored statically for each product. This list can be generated by our automation system and stored in the product description area. The custom product page layout would then use this list of products to build the page with prices, add-to-cart buttons, etc. \ Anyway, these "dummy products" could be put in a separate "parts category" that is only searched from the "parts search" box. The regular site search box would ignore this category so that parts products results don't clutter up search.

Overall I like the second idea as it allows for search without cluttering the entire site and it seems that overall it would require the least number of unusual hacks to make work. Any ideas? Is this total insanity?

Having that many categories is not a problem, but I suggest to have a clear products taxonomy before launching the ecommerce store. Otherwise, you'll suffer much, much more. I.e., it's easier to decide which product categories will be indexed or queried for full text search.

No problem, we have a well laid out category tree already existing. We would like a way for customers to be able to search for product parts but not if the results will clutter regular search results.

[Guest]

Could you not use combinations of a product - for example fishing rod has part 'washer' and in the combination information put the unique reference number for the washer for that fishing rod, for example WSH0001. In another fishing rod with part 'washer' make it's code be WSH002. You could disable the stock management so you don't have to worry what you do / don't have, you just sell it. I know it's not ideal for the drop down but you could use the Flex Layout module for it.

We sell a lot of products that have common attributes and use the unique ref number field to distinguish them, this info goes through to your invoice so you do know what is sold, it's not just listed as 'washer'.

[dynambee]

@DavidP, thank you for your reply.

One issue is that we wish to keep the parts separate from the actual products themselves. Many discontinued items still have parts available. The discontinued items don't exist on our site so we would then have to have a different way to list parts for those items. To keep things simple and easy to automate I would like to list all parts using one system rather than have a few different systems to manage.

The other issue is that PS (and thus 30bz for now) has the reputation of getting very slow if there are too many combinations on the site. I was't sure if this was actually correct or not but based on other replies in this thread it does seem that having 500,000 different combinations would kill the site's performance.

So I think I will be listing each part as its own product on the website. I just need to figure out the best way to group all a reel's parts onto one page so customers can order what they need with ease.

[moy2010]

Ok, here's an idea based on your example for the dummy product called "15 Saltiga 4500":

Now that's clear that the parts need to be uploaded as products, let's say that you create the product page for the "bolt 0027". Then, we could use the product features functionality as a base for compatibility/display.

If you create a feature named "Compatibility" with a feature value "15 Saltiga 4500" and assign this to the product "bolt 0027" you could achieve awesome things such as:

• Display within the "bolt 0027" product page its compatibility list. Even a link to each of the listed reels ("15 Saltiga 4500" in this case).
• Display the "bolt 0027" within the "15 Saltiga 4500" product page as a part, just as you want it.
• Give the customer the option to search for parts for his reel. Let's say that you give them a link to "purchase reel parts", which will take them to a landing page with a dropdown menu and/or a search bar.

The dropdown menu would list all the parts categories such as bolts, spring retainers, washers, etc. If they select bolts, it owuld redirect them to the category page for "bolts" where you could have a search bar so that customers search for their reel. If they type "15 Saltiga 4500", the system would return the bolt "0027".

The other search bar in the landing page would do basically the same, but it would perform the search across all the parts categories (return all the parts compatible with the "15 Saltiga 4500").

There are free and paid modules that allow you to have several feature values, so that you could assign the following feature values to the "bolt 0027" for the "compatibility" feature:

15 Saltiga 4500 15 Saltiga 4300 95 Saltiga Ultra Pro Saltiga auto-fishing-3000

And so on.

Yes you would need a custom development to achieve this but, due to its simplicity, it would be the best and cheapest solution if you value your peace of mind :P.

I've seen this implemented in different in several ecommerce websites, including mine. I'll use it for printer supplies, for example.

[dynambee]

There are two main makers of reels in Japan, Shimano and Daiwa.

With Daiwa we actually see their true unique internal part number for each part. We can build a cross-reference database for their parts and see exactly which parts are used in 50 different reels.

With Shimano however they do not provide their true unique part number for each part. All we get is the reel's part number followed by a generally sequential number. So we send them an order for a Scorpion 200HG Star Drag we send them the part number of the reel (03225) followed by 016, like this: 03225-016. That 016 is the sequential number on the exploded parts diagram for this reel. There are four different reels in the Scorpion 200 series and they probably all use the same Star Drag but we have absolutely no way to know. :(

To be consistent between the two brands, for ease of customer understanding, and for ease of automation we will list Shimano and Daiwa parts in the same way. This means even if a part might be shared by 20 different reels the part will be listed in the system 20 times, with the part number for each one being made up of the reel's part number followed by the parts diagram sequential number.

We also have another problem: Shimano has about 600 currently produced reels but has parts available for about 1100 different reels. Those 500 discontinued reels (1100-600=500) won't be listed on our website but we will still need a way to sell those parts. This is why I thought the dummy product and listing parts on the dummy product's page might be a good way to make this work and be consistent.

[dynambee]

So this is what I am considering now, after the discussion in this thread:

1. Create each fishing reel part as a separate product in 30bz. There will be ~100,000 to start, later potentially rising as high as 500,000. These will not be shown in search results or in the standard catalog.

2. Create a dummy product for each fishing reel that we sell parts for. There will be about 1,100 of these dummy products at first, later potentially rising as high as 5,000 products. These will not be shown in search results but will be shown in the catalog. They will be in their own category sub-tree. Eventually I would like to create a custom search for just this subtree so customers can find their parts more easily.

3. In our in-house software we will also create all these products and we will associate the "dummy product reels" with the parts for that reel.

4. This will allow us to auto-generate the descriptions for the dummy products to be used on the 30bz site (and of course allow us to automate the upload all 100,000 parts and 1100 dummy products). These descriptions will consist of a list of parts, each with the number that matches the exploded parts diagram and the 2-3 word name for that part. There will also be a button to add the part to the shopping cart.

5. The exploded parts diagram will be shown at the top of each of these pages for customer reference.

6. We will use @lesley's “Flex Layout Module” to apply a custom layout to these pages. I'll probably pay someone (perhaps @lesley if he has time, is interested, and if I can afford his time!) to create a responsive page layout for these products. The layout will have to remove things like the main "add to cart" button for the dummy product, the "out of stock" message for the dummy product, etc. as well as display the products list in an attractive & easy to use way across devices.

Does this seem like a reasonable solution?