Jump to content
thirty bees forum

Let's talk about Search!


dynambee

Recommended Posts

  • Replies 208
  • Created
  • Last Reply

Top Posters In This Topic

Feel free to visit https://inxonline.com, my dev site, and test it out. The search box with the standard blue button is the original 30bz search. The search box under that with the black button uses Elasticsearch hosted on a separate server in the same datacenter. If accessing the site results in the "site closed" page just reply to this thread and I'll open it up again.

A few points about the setup that might be of interest (or not...)

  • These are both VULTR VPS boxes.
  • The web/db server is 2GB and 2 cores, hosted on VULTR through Cloudways.
  • The Elasticsearch box is 8GB and 4 cores, hosted directly on VULTR.
  • Both servers are in VULTR's Tokyo datacenter because it's the closest one to me here in Osaka.
  • Elasticsearch is v5.4, I downloaded a bitnami installer to get it up and running with no fuss.
  • I had to increase max file descriptors and vm.max_map_count from the defaults that came with CENTOS 7, but this was detailed in the ES log file when it failed to start.
  • Cloudways doesn't give users private IP addresses so I'm running this over the public IP addresses. I'll check with Cloudways to see if I can get a private IP on that box as it would be safer. For now I have firewall rules to only allow ES access from the web server IP and my office's fixed IP.
  • If anyone here is thinking about getting a VULTR VPS they have a great promo running right now, they match your advance payment deposit dollar-for-dollar up to $100 deposited. I sent them $100 to cover hosting, they credited my new account for $200. Not bad! You can see the details here: https://www.vultr.com/coupons/

A few points about BRAD that I have noticed from testing without fuzzy search turned on: * BRAD (without fuzzy search) seems to be an exact phrase search. If I search for duel hardcore I get great results. However if I search for hardcore duel I get nothing. I expect this is a problem with the way BRAD has been coded. * The thumbnails that appear in the instant search results are supposed to appear to the left of the search results. For some reason the box isn't wide enough right now so they are appearing on a separate line. I'll try to fix this as it looks much nicer when it's all on one line. * I'm very much a keyboard guy (I guess most programmers are?) and when I get drop-down instant results like BRAD provides I want to be able to hit the down arrow to move down and then select a result by hitting ENTER. The standard 30bz search does this but right now BRAD does not.

Link to comment
Share on other sites

The problem with default ecommerce software search is that it suffers from performance issues, because it's not suited for big shops with thousands of products.

Once merchants realice this, they will definitely look for something better (and might another way to attract users to 30bs, @lesley @mdekker ). I would really like to get a module named "Thirtybees Advanced Search" that would, once and for all, solve the issues with faceted and text search from Prestashop. We all know the horror stories about PS Layered, right?

Now, IMHO when users are in such a situation where the standard search is no longer enough and start looking somewhere else for solutions, soon realice that BRAD is that it's a very basic implementation and requires a lot of tweaking.

Just to quote dynambee's issues:

@dynambee said in Let's talk about Search!:

A few points about BRAD that I have noticed from testing without fuzzy search turned on: * BRAD (without fuzzy search) seems to be an exact phrase search. If I search for duel hardcore I get great results. However if I search for hardcore duel I get nothing. I expect this is a problem with the way BRAD has been coded. * The thumbnails that appear in the instant search results are supposed to appear to the left of the search results. For some reason the box isn't wide enough right now so they are appearing on a separate line. I'll try to fix this as it looks much nicer when it's all on one line. * I'm very much a keyboard guy (I guess most programmers are?) and when I get drop-down instant results like BRAD provides I want to be able to hit the down arrow to move down and then select a result by hitting ENTER. The standard 30bz search does this but right now BRAD does not.

I hope that Thirtybees' Advanced Search will get these cases solved!

Link to comment
Share on other sites

@mdekker said in Let's talk about Search!:

I think that's a bit biased, but I could be wrong ^^

Yes, that comparison does come across as a little bit biased. ;)

I think Algolia's biggest benefit starts with their $299/month plan and that is distributed search. This is great for "as you type" instant search results as the results come directly from the Algolia datacenter closest to the customer. This will be faster than if a customer is searching on a website+ES setup that is on the other side of the world. Importantly though distributed search is not offered with Algolia's $59/month plan so you do pay a big premium to get that functionality.

It's also possible to have distributed search with ElasticSearch, and in fact it's one of the reasons ElasticSearch was created in the first place. I don't think it's necessary to take this complexity into consideration with any possible 30bz ES module though. If a site is big enough that they are thinking about global distributed search then they can afford to either pay Algolia or to build their own ES search module.

So IMO for shops that have fewer than 30,000 products, ES is a better overall option. Algolia would be attractive to those who don't want to do anything themselves, but I suspect most users like that would just end up on Shopify anyway. (Funnily enough, Shopify also uses ElasticSearch!)

For shops that exceed 35,000 records and need the $299/month plan Algolia will feel faster to customers in far away places because of the distributed search feature. Customers who are nearby probably won't notice any difference, as long as ES is set up on appropriate hardware. ES will certainly be cheaper, and it's very easy to set up an ES server.

I think a lot of the points in the Algolia blog are FUD-like. Issues like sharding do need to be given some consideration when setting up an ES system but for the typical 30bz webshop owner it's not going to be a problem. You can't have more ES servers in a cluster than you have shards so if you start with 5 shards (the standard) you can't have more than 5 servers without reindexing everything. This is something that is talked about a lot in the ES community because many ES clusters are absorbing gigabytes or 10s of gigabytes of new data every day and they need to be able to grow. The typical webshop is not going to ever need more than a single ES server. This is a problem for sites like eBay (which uses ElasticSearch) but not for most smaller storefronts.

@moy2010 said in Let's talk about Search!:

"If you pay, Algolia does everything for you" :P

Yes, that about sums it up.

As an aside, I noticed that Algolia claims customers save 25% if they prepay for a year at a time. However the prices they quote are not 25% reductions. Savings on the two smaller plans are ~17% and on their largest standard plan only 10%. I realize no one is perfect but that seems kind of misleading. :(

Link to comment
Share on other sites

@dynambee I agree with all your points.

I'm surprised the focus is so much on performance, when the low volumes we're talking about here is just a joke both for ES and Algolia.

A couple of years ago I setup a ES-based search system that needed to search a database of 150 million complex records. The performance was excellent and the memory- and CPU-consumption was nothing to talk of. We ran it all on two cheap 16GB servers. Just to put things in perspective...

The focus here should really be on functionality, ease of installation and ease of use, not on all the technical issues.

Link to comment
Share on other sites

@roband7 said in Let's talk about Search!:

@dynambee I agree with all your points.

I'm surprised the focus is so much on performance, when the low volumes we're talking about here is just a joke both for ES and Algolia.

The performance of "as you type instant search results" is in large part about latency. PS & 30bz both provide "instant" search but the latency is terrible so it's not really functional. Nothing against 30bz of course, MySQL full text indexing just isn't fast enough for instant search.

This performance bottleneck can be solved with either ES or Algolia. However for websites the distance from the site to the user is another issue and is why distributed search is such a cool thing. One very fair point in the Algolia blog post that was linked to above is that latency higher than ~100ms starts to feel laggy for instant search. If your customers are all within one geographic area then 100ms isn't a big problem. For global search it is though, a customer in Australia searching on a site in Europe probably won't have a great "instant search" experience.

A couple of years ago I setup a ES-based search system that needed to search a database of 150 million complex records. The performance was excellent and the memory- and CPU-consumption was nothing to talk of. We ran it all on two cheap 16GB servers. Just to put things in perspective...

This is very nice to hear. There's lots of talk online about large scale ES clusters but not a lot of information about smaller scale performance & hardware requirements.

The focus here should really be on functionality, ease of installation and ease of use, not on all the technical issues.

Installation & configuration will always be easier with a SaaS system. Shopify is easier to set up and use than 30bz. Algolia is easier than ES. However Shopify and Algolia are expensive, especially if you want their plans that provide better service options.

Want to use 3rd party calculated shipping on Shopify? You're forced to use their $300/month plan, per website of course. Live outside the few countries that support Shopify Payments? You get to pay transaction fees for sales on your own website! Want to add any features that Shopify itself doesn't provide? Almost all are monthly subscriptions rather than one-time purchases. The costs add up in a hurry.

Algolia is the same. If you have a large shop or you want lag-free search for global customers then you get to pay $300 a month.

Convenience, at a price. For some people perhaps the costs aren't a big deal but unfortunately I don't fall into that group of people. :(

From a functionality standpoint I think the two are similar except for Algolia's easy (but expensive) distributed search. Both provide fast search results with the ability to filter those results. A lot of the functionality depends on the quality of the search module doing the querying. It would be possible to have a fantastic ES module and a poor Algolia one, and the other way around of course too.

For installation there are hosts that offer one-click ElasticSearch, like Cloudways. As long as your server has the capacity you can enable ES by clicking a checkbox and saving the choice. For hosts that don't offer this or for people who want to have a separate ES server there are already options available for relatively easy installation. I don't think it's harder to set up basic ES than any other typical web-facing service.

Link to comment
Share on other sites

This thread has been very interesting to follow. But again most of the discussion is for the nerds that is members of this forum, not for all of the ones that is merchants only, not tech freaks. What shoud they choose, They ofc want a very good system. Not all of them will go to Shopify or Bigcommerce or whatever, its very expensive. So for them it reall does not matter, Algolia is there quite easy to setup but can cost a lot. But that is really the same for Elastic. If you want to use the Cloudservice they provide, it quite quickly get very high priced. Why is it so? Computer power and bandwidth is cheap nowadays. I would like to see a very good TB module that can connect to a good priced Elastic provider. Not one that charge hundreds and hundreds per month. It is fully possible, someone just have to start

Link to comment
Share on other sites

@Havouza said in Let's talk about Search!:

This thread has been very interesting to follow. But again most of the discussion is for the nerds that is members of this forum, not for all of the ones that is merchants only, not tech freaks.

Search is really something a person can geek out on. It's always interested me because good search makes a system/store/website a joy to use while slow or non-functional search leads to immense frustration, at least for me.

What shoud they choose, They ofc want a very good system. Not all of them will go to Shopify or Bigcommerce or whatever, its very expensive.

I'm definitely not going with a packaged system. They can be a good fit for some people but it's not for me.

So for them it reall does not matter, Algolia is there quite easy to setup but can cost a lot. But that is really the same for Elastic. If you want to use the Cloudservice they provide, it quite quickly get very high priced.

Algolia has a free option that should be fine for stores with 1000 to 2000 items as long a they aren't too busy with visitors. I don't know if they limit features or performance but they do force a logo to be displayed on your site which is not ideal.

Elasticsearch also doesn't have to be expensive. For a similarly sized store it shouldn't be a problem to run Elasticsearch on a smallish VPS together with the web and db servers. If you site gets busier or your catalog grows then something else will need to be done but a separate small VPS should be enough for even a relatively large (~100,000 products) store.

I don't know about other hosts but I know that Cloudways offers Elasticsearch on their plans. Just check a box to turn it on, no configuration necessary.

Why is it so? Computer power and bandwidth is cheap nowadays. I would like to see a very good TB module that can connect to a good priced Elastic provider. Not one that charge hundreds and hundreds per month. It is fully possible, someone just have to start

Most of the hosted ES cloud providers are focused on larger installations and are priced to match. They also tend to be on AWS which is expensive compared to many cloud hosts.

I think that with 30bz being open source it would be nice if the standard "advanced fast search" module used an open source search engine, but unfortunately I don't have the skill to code the search module myself. :(

Link to comment
Share on other sites

All this is good but again we talk tech guys, not my cousin that is born totally tech zero but is a premium merchant. He has no clue about i-click install or vps or.... But he is super also on economy and realize the trap with platforms like Shopify.

About hosted elastic search. I have only looked at their own cloud pricing and a setup with 2 Gb ram and 48 Gb storage cost 79 $ per month. Same time they state that less than 8 Gb is very bad

Link to comment
Share on other sites

@Havouza said in Let's talk about Search!:

All this is good but again we talk tech guys, not my cousin that is born totally tech zero but is a premium merchant. He has no clue about i-click install or vps or.... But he is super also on economy and realize the trap with platforms like Shopify.

There is always going to be a trade-off. The costly solutions (Shopify, Algolia, etc) do everything even if the shop owner doesn't know much about tech. All those fees go to pay a large staff of tech people and all the hardware & software they use to make the magic happen. The same for hosted Elasticsearch and all other SaaS solutions.

On the other hand there are a selection of free software solutions that require some time & knowledge to use. That includes 30bz of course, it requires much more knowledge to effectively use 30bz than it does to effectively use Shopify, for example.

Elasticsearch is no different in this regard. The level of power provided by Elasticsearch is staggering, and it is available for free. That's incredible if you think about it for a bit. Yes, some time & effort may be required to set it up, or in the case of Cloudways, the ability to click a checkbox and then choose "save" (that's seriously all that is required).

If clicking a checkbox and choosing save is too much to ask of someone then they are going to have a hard time installing & configuring the PayPal module, the Shopify module, and just about everything else, too. For these people services like Shopify exist. That's not meant in any sort of negative way, some people will just find it very worthwhile to use a hosted SaaS where the back end things are taken care of for them. We all do this for some things. I don't do my own car repairs, I take it to a shop. However I can do my own Elasticsearch setup, so I do that.

About hosted elastic search. I have only looked at their own cloud pricing and a setup with 2 Gb ram and 48 Gb storage cost 79 $ per month. Same time they state that less than 8 Gb is very bad

As I learn more about Elasticsearch it is becoming clear that vast amounts of memory are not needed for the type of index required by most web shops. That memory is used to store indexes for ultra high speed searching. If the indexes can't be stored in memory then fast search won't work. For the volume of data that ES usually manages a large server (or cluster of servers) is required. However for a web shop with a few thousand products it seems very possible to use something like what Cloudways provides. It even worked fine on my tiny 2GB server even though ES was sharing memory with MySQL, Apache, redis, nginx, and some other services. I thought the search problems I experienced may have been because of not enough memory but it turns out they were because of limitations of the BRAD module and because of limitations in my own understanding of ES.

Btw most discussion has been around Elastic/Brad. I see there is an expensive polish PD module. Anyone has tested?

I tried to figure out what you mean here but unfortunately I'm not sure. Can you link to it or give more details?

Link to comment
Share on other sites

Hi,

First sorry for my english

I test the brad module and it is really fast.

I changed order by when fuszzy is enabled.

In the index options I add stopword section and synonim section.

In my activty tapis = chabraque ; bombe = casque .. ...

I actualy test operator AND

Link to comment
Share on other sites

@Havouza said in Let's talk about Search!:

@dynambee https://addons.prestashop.com/en/search-filters/19527-elasticsearchconnector.html

Wow, that's a very well put together Elasticsearch Module. It does pretty much everything I would put on a search module wishlist. The only thing I don't see mentioned much is filtering results but their own website has filters on it so I imagine their module supports it too. I'd be curious to test it and see how good the search results are, and I might just fork over the 130 Euro to do that. Very tempting, and a great find! Thanks for sharing it.

Link to comment
Share on other sites

For the quality of the code anf its functionality, it's been worth every penny since I purchased it two years ago. The developer is actually helping me with our custom ElasticSearch implementation, because he does know what he's doing :).

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...