Let's talk about Search!

[dynambee]

@mdekker said in Let's talk about Search!:

What's so great about it? Now I want to know!

Is it just because of the weights you can add? The custom search and index settings?

It pretty much covers all the bases:

• can manually choose which fields get indexed and which do not.
• set weight for each field type separately
• select the search operator (AND, OR, etc)
• can automatically disable elasticsearch during indexing processes. If the server is already overloaded then ES queries may be very slow during an index. If this setting is turned on then the site falls back to standard PS search during indexes to avoid problems.
• can force exact word match only (i wouldn't want this but i can see how some sites might)
• can set min and max word lengths
• easy to edit the word blocklist
• supports clustering for very large sites that have enough traffic that they might need it
• has logging support
• can choose when to index -- autoindex at time of adding an item, batch based scheduled index, or manual index

As an added bonus it's been under active development for 2 years so I would guess most of the bigger issues have been worked out.

[dynambee]

@moy2010 said in Let's talk about Search!:

For the quality of the code anf its functionality, it's been worth every penny since I purchased it two years ago. The developer is actually helping me with our custom ElasticSearch implementation, because he does know what he's doing :).

I'd be very interested in hearing more about your current ES setup, how many products you have in your store, and what sort of custom ES setup you are working towards now.

[moy2010]

@dynambee

I have ~5000 products, and my statement was very clear:

"I need something as good as ElasticSearch connector for prestashop, but that covers faceting search and a custom autocomplete search".

We have been working on this for about a year, and it's almost ready at this precise moment :). Just some minor tuning and it'll be ready to hit production by mid august!

[Havouza]

There are many subjects in this General discussion part that trigger and old layman's curiosity, so also this discussion. I found a site said to rank the ten best Elasticsearch hosts. When reading on the Elastic search homepage and in this thread that a minimum of 8 Gb ram is needed. But these hosts offer plans from 0.5 Gb ram and 5 Gb storage. Another 1 Gb ram and 20 Gb storage. 8 Gb ram is about 160 € per month. So how to calculate what you need. Some of the hosts say that RAM should range from a 1:20 dataset size for low search volume to 1:1 for high search volume. I assume that a shop can be in the low range.

Now the layman shows. What is a dataset in this context?

[Traumflug]

Dataset is obviously the data which is subject for searching. As a coarse estimate one can use the size of the uncompressed database backup file.

[Havouza]

So my ~5000 products 26 Mb db sextoy site will not need much RAM...

[dynambee]

I've edited my post above about the 8GB as it seems to have been adding to the confusion. Perhaps @mdekker could edit his as well. With more testing it's clear that 8GB is not necessary for a single web store of reasonable size. Except for large stores with heavy traffic it seems very possible to run ES on the same server as MySQL and the webserver are running.

ES, btw, is not CPU intensive for normal use. It just needs memory. As long as the server has enough memory to hold the dataset without causing problems for other services on the same server then there really should be no problem with a single server ThirtyBees install.

I don't have any live sites I can test this on right now but I will certainly add information as I learn more.

[moy2010]

@Havouza said in Let's talk about Search!:

So how to calculate what you need.

The only way to calculate how much RAM you need is by measuring it.

I'm not planning to sound sarcastic, so please don't read it that way. This sentence is repeated in ES forums like a mantra, and I fell into thinking that they were being sarcastic, too.

What the members in the ES community mean is that, due to its nature, ES can be used on a huge plethora of scenarios that range from analytics and logging to searching, among many others. This case is worth mentioning:

https://www.noschoolviolence.org/

They use ES to identify behavioral patterns in students, so they can intervene early.

This is one of the reasons for why there's no right answer to the question: "How many resources will ElasticSearch use?" You have to measure it in your very own ecosystem, every single time.

Now, back to our eCommerce interests, I'm currently using PrestaWach's ElasticSearch Connector, and my RAM consumption for ES is ~130MB for autocomplete and full-text search in production with ~5000 products.

I'm yet to measure the resources consumption for ES being used in many other scenarios, but so far so good :).

[Havouza]

Its a pity that the connector according to the developers not will work with TB and they are not interested to make it work. But I doubt they have even tested

[okom3pom]

On my website, 4k products, 27k combinations

Brad module with synonym, elasticsearch use 800 MB

    if ($isFuzzySeearchEnabled) {
        $searchQuery['query']['bool']['should'][] = [
            'multi_match' => [
                'fields' => ['name_lang_'.$idLang.'^8','manufacturer_name^5','default_category_name_lang_'.$idLang.'^5','ref_combi^5','attribute_keywords_lang_'.$idLang.'^3'],
                'type' => 'most_fields',
                                    'query' => $searchQueryString,
                                    'fuzziness' => 'AUTO',
                                    'prefix_length' => 2,
                                    'max_expansions' => 50,
                            'minimum_should_match' => "60%"
                //'operator' => 'AND'

            ],
        ];
    }

[dynambee]

Thank you @okom3pom for that information. Is this with PS or on 30bz? I found when I had the BRAD module installed on 30bz some category pages seemed to not load correctly. When I disabled & removed it things were okay. I didn't spend time checking out why that was though.

I'm also curious if either @mdekker or @lesley is able to comment on how things are progressing with 30bz search and what sort of direction they have decided to go. In particular, Algolia, Elasticsearch, both, neither, or something else...?

[okom3pom]

I use brad with PS

Currently I don't use filters. Problem's of categories can come from filters ?

U can test : esprit-equitation[dot]com

Ex synonyme : coffre = boite / chabraque = tapis

[okom3pom]

I also use a trick to highlight items from my brand

        // Boost DMH Products           
        $words_dmh = ['amortisseur','boots','etrier','licol','gogue','masque','transport'];         
        if( $body['manufacturer_name'] == 'DMH Equitation' ) {                  
            foreach( $words_dmh as $word ) {                                    
                $search = Tools::replaceAccentedChars(strtolower($name));
                $pos = strpos($name, $word);                    
                if ($pos !== false) {                   
                    $body['manufacturer_name']  = 'DMH Equitation '.$word ;                     
                }               
            }           
        }

In search the articles of my own brand are first

[dynambee]

You've really got BRAD working nicely on your shop, the search results are super fast and I still love the BRAD drop-down list design with the mini thumbnails in it.

I wasn't using any filters with BRAD either, I just installed it on 30bz in the default configuration. It seemed to work but it also seemed to have some compatibility issues, and the search results weren't optimized in the default installation.

I may still monkey around with BRAD a bit and see if I can figure out what's going on but for now I have some other stuff I have to take care of, unfortunately.

[okom3pom]

I migrate my test shop to 30bz

Brad work fine for the moment i have only 300 indexed products

chafoins[dot]com/presta2/

Login : oups Password : okom3pom

[dynambee]

@mdekker said in Let's talk about Search!:

I’m also curious if either @mdekker or @lesley is able to comment on how things are progressing with 30bz search and what sort of direction they have decided to go.

No efforts have been made. We're still talking with several parties before we can continue with the modules.

Thanks for the status update. I think I might try buying that 130 Euro ES module from the PS store and see how that goes with 30bz. They said it wouldn't work but I think it should, perhaps with some minor changes. I'll certainly let you know how it goes.

[Havouza]

When I taled to them I got the feeling that is was more about PS devotion than that they really knew that it would not work

[Havouza]

Yea I understood that too

[okom3pom]

ElasticSearchConnector and Brad does not work as same

Brad indexes all product informations.

Search query -> ElasticSearch give result -> php decode result -> Print

ElasticSearchConnector retrieves the id_product via elasticsearch and uses an override of the class Search to give the results

Search query -> ElasticSearch give result -> php decode result -> SQL query id_product IN (xx,yy,zz ) -> Print

[dynambee]

Is one of these methods better than the other?

[okom3pom]

For me Brad is faster on my website.

If i have the time i test elasticsearchconnector on 30Bz

[dynambee]

Disappointing that the expensive ES module uses the less efficient way.

I might buy it anyway and then as a paying customer try to encourage them to make it more efficient... I guess it depends on how much work it would take to rewrite that part of it, and how much of a knock-on affect there would be to other parts of the module.

[dynambee]

@mdekker said in Let's talk about Search!:

To be honest, the list you made above looks like just ElasticSearch settings. So if BRAD would have had these additional settings, it would have been suitable for you as well, right?

Pretty much, yes. When I had Brad installed I had some problems with category pages that went away when I uninstalled it. I'm not sure why that was though and it seemed to work for a while before I noticed the problems.

Ideally I'd also like to figure out how to use the filters that Brad seems to offer, and of course would have to figure out how to get the brad search box to actually replace the standard 30bz search box rather than it creating a separate search box.

Right now I'm fixing some other problems with our in-house automation but I plan to get back to the web-focused work soon and may try Brad again, perhaps on a fresh 30bz install.

[dynambee]

Yes. I will probably keep using it when I launch too, at least at first.

[dynambee]

So if I switched to something like Transformer or Warehouse Brad might actually work better?