Let's talk about Search!

[dynambee]

The standard search functions of most shopping carts are pretty disappointing. I haven't looked into this in fine detail with TB but my expectation & understanding is that TB 1.0.x search is the same as PS 1.6 search, and as such it's slow and overall just not very good. No "instant search" with results as users type. No "fuzzy search" that gives results even if the user makes spelling error(s). No ability to filter searches by price / category / brand / supplier / attributes / etc.

Worst of all is that the results themselves don't seem to be particularly good and as you add more products to a store it takes a lot longer for searches to return results. If the site is at all busy and has a higher number of SKUs then standard search will grind to a halt completely.

Understandably all of this is very frustrating for visitors and frustrated visitors are less likely to convert into customers.

So, what to do?

There are some incredibly powerful yet rather pricey options out there such as Algolia hosted cloud search. @lesley has a very slick Algolia module available and this is, IMO, the best PS/TB solution for those who have the budget.

Unfortunately I fall into the group of people with no monthly budget for search services so I've been looking into free high performance search options that are compatible with TB. A while back I came across a potential solution that I think might be of interest to other TB users: Elasticsearch + Brad.

Elasticsearch is a free high performance full-text search engine that is used by some of the largest & busiest sites online. Facebook, Wikipedia, Netflix, GitHub, eBay, and the New York Times all use Elasticsearch to power their indexing & searching.

Brad is a free module for PS 1.6 that enables searching through Elasticsearch on a PS 1.6 website. Brad has been around for a while and is currently at version 2.1. I installed Brad on a TB development server (a Cloudways VULTR server) that has Elasticsearch 2.4.4 running and Brad seems to work with TB -- mostly. There are some things I haven't figured out yet and I'm not sure if these are caused by differences between PS and TB or if I just don't have things working properly yet.

What works out-of-the-box with Brad on TB:

1. The module installs with no errors as long as you follow the directions on the GitHub download page.

2. The back office seems to work fine. The Re-index buttons work and all the settings I tried can be adjusted and saved.

3. In the left hand column of the back office there are various menu choices for Brad when looking at the module configuration. Settings, Advanced Settings, Filters, Filter templates, Info. These all seem to work as well and changes made are saved correctly.

4. Searching for items. Once Brad is turned on searching is an entirely different experience. Results start to appear as the visitor types their search terms (this can be turned off in the BO if desired.) Type "sumer" instead of "summer" and you still get good results. Other spelling errors also usually still produce quite good results. Search results are fast, but it's difficult to gauge speeds on a dev site with few items.

What seems a little wonky:

1. My dev site is using the standard TB theme. The standard TB theme search box at the top, to the right of the logo, doesn't change to a Bradsearch box. Instead a new search box appears just below the logo. This is likely due to a difference in hooks or ID, something like that. I'll look into it but I'm not a web developer so my abilities are limited.

2. Brad supports search filters out of the box but I can't get them to appear in TB. I'm guessing this is similar to the above issue and is caused by template differences but it could equally be something I'm doing wrong.

While this system isn't completely plug-and-play at this point it improves the standard search even without the filters working properly. I will continue to monkey around with Brad in the hopes that I can figure out what is going on with the extra search box and the missing filters. I would be most appreciative if any other interested parties could do likewise. Perhaps working together we can figure things out and have a fantastic free search solution for TB!

[dynambee]

Thanks for the replies and the information! It's good to know that I (probably) wasn't doing something stupid that resulted in the missing filters.

Many of the Brad files include a "you can do what you want with this" type of pseudo-license message. I'm not sure why they didn't just go with a known license that would accomplish the same thing while making things clearer for everyone. Perhaps they'll do this if asked about it... Invertus seems to be heavily wedded to PS and I'm sure they are feeling the pain of 1.7 as well. They might end up being quite interested in TB as a better solution for their own clients.

---

The below text is causing a lot of confusion so I have edited it with strikethrough. Elasticsearch for a normal web store does not need anywhere near 8GB. If you have a busy site then you may find a separate server helps but for many sites it will be very possible to run ES on the same server as your web & database are already running. I fully intend to test this as I get sites operational and will add information here as I figure more things out.

~~That's interesting about the Elasticsearch memory requirements, I wasn't aware that it was that intensive. Looking at the Elasticsearch site it seems they recommend Elasticsearch servers have at least 8GB but not more than 64GB, and they do mention it's very memory intensive.~~

~~My first goal is to get a site up and running, and then I will work on Brad and see what it can do with an 8GB or 16GB server and a few websites. Algolia is definitely interesting, if it is a big enough time saver without costing too much then it would be the better choice.~~

[Traumflug]

FWIW, PS 1.7 does show results as one types, so one option would be to grab code from there.

(Theme is Classic with heavily modified CSS)

[moy2010]

Onsite search is much more complex than most users realise and, to be honest, thirtybees default search is good enough for like 99.9% of merchants.

Though going beyond that would be welcome, I don't know if it's the right time to do so (I think there are some more important subjets for thirtybees developers).

In the meantime, we can discuss search and how would you like to get integrated into an ecommerce platform :).

[ssimard]

@Traumflug said in Let's talk about Search!:

FWIW, PS 1.7 does show results as one types, so one option would be to grab code from there.

Thirtybees does too, am I missing something ?

[Traumflug]

All the better, @ssimard ! @dynambee wrote about this lack of functionality in the opening post, so I didn't even try. Now I tried and see this working in TB, too.

[dynambee]

Hmm, interesting about the "instant search" feature with standard search function.

I type pretty quickly (~120wpm when I was young, ~80wpm today) and I actually finished typing search terms and hit enter to search before anything came up.

When I do the same thing in the bradsearch box the results start to appear nearly instantly and then continue to be refined very quickly as I continue to add letters and words. I have the minimum search length set to 3 characters on my dev box and by the time I strike the fourth key the results are there.

So the feature is there with the standard search, it's just not all that...instant.

[ssimard]

I have yet to test with all my products added but should be able to do it soon. (1000 books with their pdf and epub variants so close to 3000 products)

Hopefully speed can stay decent but I'm a bit worried at the moment...

[dynambee]

I have just "enabled" my dev domain so people here can try bradsearch.

Notes:

1. The server is in Tokyo because I'm in Osaka and it's really just for my own dev & testing. I have Cloudflare set up but I have no idea what the performance will be like in other parts of the world. I'd be interested in hearing how it feels speedwise, both for page loading times and for the search. (I am working with webservice integration right now so the site itself is bog standard.)

2. The search box at the top of the display with the blue search button is the standard search. This has not been changed by bradsearch at all.

3. The search box just below that with the black button is the bradsearch box.

4. There are only the standard 7 products on the site right now so this isn't a proper performance test, just a proof of concept. Good test terms will be things like "dress", "summer", etc. Stuff that will turn up the standard sample products. Try with spelling mistakes too, in the bradsearch box.

5. This is a dev box. Dual cores, 2GB memory. Not exactly a fast machine and far from an ideal Elasticsearch system.

6. Please play nicely. :)

The domain is https://inxonline.com/

Edit: I'll be able to keep the site enabled for a day or so, after that I will have to put it back into maintenance mode for further internal dev work & testing.

[Traumflug]

Works nicely and with decent speed from Germany. These pictures in the immediate search result are certainly a good idea!

[dynambee]

@ssimard said in Let's talk about Search!:

I have yet to test with all my products added but should be able to do it soon. (1000 books with their pdf and epub variants so close to 3000 products)

I should also have a site with ~1500 products soonish. A few more things to prepare before I can remove the existing products from the site and start sending my actual products through the webservice. I'll be sure to try out bradsearch again at that time and see how it performs with more items, as well as if it needs a more powerful server to work.

Hopefully speed can stay decent but I’m a bit worried at the moment…

I think you should be okay with ~3000 SKUs. The website for bradsearch has some performance stats in a basic chart and their tests show 480ms for standard search results with 10,000 SKUs, vs 170ms for bradsearch. No idea what sort of hardware they were testing on, unfortunately. Things really start to go haywire at 100,000 SKUs and will go downhill dramatically from there. It will of course also depend on how busy your site is, busier sites will benefit more from a dedicated search system.

[dynambee]

@Traumflug said in Let's talk about Search!:

Works nicely and with decent speed from Germany. These pictures in the immediate search result are certainly a good idea!

Thanks for the feedback!

I'll be able to keep the dev box open for bradsearch testing for another day or so and would certainly appreciate more feedback & thoughts if anyone has a moment to try it. Sometime tomorrow I hope to be back to testing webservice stuff and will have to put the website back into maintenance mode.

[ssimard]

@dynambee said in Let's talk about Search!:

I think you should be okay with ~3000 SKUs. The website for bradsearch has some performance stats in a basic chart and their tests show 480ms for standard search results with 10,000 SKUs, vs 170ms for bradsearch. No idea what sort of hardware they were testing on, unfortunately. Things really start to go haywire at 100,000 SKUs and will go downhill dramatically from there. It will of course also depend on how busy your site is, busier sites will benefit more from a dedicated search system.

Sounds good, will report back once I have something online, development is still few weeks away from having a working inventory.

[musicmaster]

A lot of improvement can already been achieved by having a good look at how indexation works in Prestashop.

If you look in classes\search.php you see that for supplierreference the field of the psproduct table is used instead of the fields of the psproductsupplier table. Also - when a product is in more than one category only the name of the default category is used.

[dynambee]

I have found a live production site that uses an older version of BRAD search that used Elasticsearch 1.7. I tested this older version BRAD search using Elasticsearch 1.7 about a year ago and the biggest difference I noticed was that the instant search results were much slower than with the newer versions.

For me, the most interesting thing to be seen on this site with the older version of BRAD search is the ability to refine subcategory results, shown in the left column of these pages: https://www.deelat.com/vehicle-fleet-maintenance/drones/ https://www.deelat.com/fasteners/flange-screws/hex-flange/

The sliders and check boxes can be configured within the BRAD module, but I have no ability to figure out how to get them to show up within a template or how to get different sets of options for different categories.

[dynambee]

I've found another production site using BRAD search but it appears to be a newer version. It's hosted in (of all places!) Iran:

Site in Persian

They use Cloudflare for DNS but they don't have Cloudflare's caching turned on, so I could see their IP and hosting location.

Google Translate does a decent job with the site but BRAD doesn't seem to work on the post-translate site. Not too surprising really since the translation is passing through Google's servers which will likely cause problems with AJAX functions.

Persian is a right-to-left language so the site is a little odd to use but the search performance is impressive, especially considering Iran isn't thought to have particularly fast Internet. Some sample search terms to try include:

• kardon

• portable

• digital

• bluetooth

The site does not have fuzzy search turned on so spelling mistakes will not return results.

They do have category result options enabled though:

Original Persian

Google Translate English

Again the Google Translate version doesn't seem to work with AJAX updates but it gives you an idea what the options are. Clicking on various options on the Persian site does give results. There seems to be a bug with the color selection setup, once I chose a color I couldn't deselect it any more.

Anyway, it's interesting to see what's possible, I hope to have a live sample site I can post later this month. I'll certainly share whatever I manage to learn about setup and implementation with the community.

[dynambee]

I don't have a functioning site yet, but I'm getting close!

I like the idea of Elasticsearch mostly because of the number of sites I want to launch using the automation I'm currently putting in place. Between automation and a large available catalog, having to pay $$$ to $$$$ per month for any specific aspect of each site will be financially difficult. This makes my situation rather unusual as I'm willing to put in more work to save that $$$ to $$$$ than most other store owners would be. $$$ x 1 isn't so bad. $$$ x 10+ is too much for me right now.

On the other hand, I do fully understand that you, @lesley, and other people building 30bz need to make some money off the project at some point or the project will die. I hope to find a way to contribute in both time and with finances as things improve financially for me.

[alwayspaws]

@mdekker said in Let's talk about Search!:

Hmmz, you seem to have gone the ElasticSearch way haven't you? We are pretty close to modularizing @lesley 's Algolia module for thirty bees. Would you be interested in a beta test? Or is it all ElasticSearch now? :P

Is Algolia search going to be part of thirty bees? (in other words, no cost?)

[lesley]

Yeah, likely it will be soon. I think it might be one the next major modules we release. It likely will not be compatible with all themes at this point, but should be a good base. In the future we have talked about having a deeper Algolia integration as well, it just makes sense for large shops.

[alwayspaws]

Very nice, @lesley. Much appreciated.

[dynambee]

I am getting quite close to uploading ~3000 items to a development store. Just working on the final bits of the upload process via the API and then I will be doing some testing. Once I have the items uploaded I'll open the development store for people to try out search again.

[dynambee]

@alwayspaws said in Let's talk about Search!:

Is Algolia search going to be part of thirty bees? (in other words, no cost?)

It sounds like the module will be available for 30bz, and 30bz have said that in-house modules will be free, something I am deeply appreciative of. There would still be monthly costs for the Algolia service itself.

Paying monthly for Algolia is not unreasonable for most stores as there is a cost to running Elasticsearch as well. Elasticsearch is memory hungry so for effective production use you need at least a VPS with 8GB of memory but it will be happier in most stores if there is 16GB of memory. This is a lot more expensive than the cost of a basic VPS or shared hosting platform that can happily run 30bz without Elasticsearch.

So for a single site it will generally be cheaper to use Algolia than to pay for a server that can effectively host Elasticsearch. Things change when you start looking at running 10 sites or 100 sites, or if you have a very large catalog. In these cases it will generally be cheaper to run Elasticsearch, sometimes dramatically cheaper. There are other differences as well of course, some in favor of Algolia and others in favor of locally hosted search. There's no one-size-fits-all perfect solution.

[lesley]

Yes, when we release it, it will be a free module.

[dynambee]

I uploaded about 1000 products to my dev site and then gave brad/elasticsearch a more serious trial. Unfortunately the results were not great, but I'm not sure exactly why. I've contacted the company that made the free module but as it's a free module I definitely don't expect a speedy reply or a lot of free support from them. They do use the module for their own customers though so it would be in their own best interests to fix any problems it may have.

My dev server has only 2GB of memory which really isn't enough for a proper elasticsearch system. I may try boosting it up to 8GB or 16GB and see if that works, or I may try putting Elasticsearch onto a different server and seeing how that works.

In better news, the 30bz search doesn't seem any different in speed with 1000 products than it does with 10 products. My next step will be to push towards 10,000 products and see how search works then.

I also look forward to the Algolia module and seeing how it works and how much it would cost on a monthly basis. Even if it is more per month (within reason of course) it could be worth it just to have a turnkey & supported system.

[Havouza]

Following this thread with interest. Looked on the Algolia web today and on prices.

Can anyone of you experts explain what 10000 records mean? 100000 operations?