dynambee

After getting some rest and a bit of time away from the computer I have done some more testing and figured some more things out. It's working MUCH better now. Details in my edited post above.

Searching for duel hardcore does bring up lines, but the top results are for non-hardcore Duel lines, followed by some hardcore lines.

I set up an 8GB VPS in the same datacenter as my web/db server, installed elasticsearch 5.4 on it, and set up the firewall etc so it would all work. Then I uninstalled BRAD from my 30bz shop, followed the directions to use composer to change the BRAD module install files for ES 5+, and then reinstalled BRAD using the newly created zip file. Everything went fine, and I was able to connect to the the other ES service and reindex the 1095 items on my test site. EDIT: Okay, I have figured a couple of things out after some more testing. Being well rested seems to help one think clearly. Whoda thought? The problem with searching for hardcore is that it actually was mentioned in the description of the items being returned. 100% my fault, should've looked at the item description in more detail. Turning off the fuzzy search option in the BRAD module dramatically improved the search results. In fact it works very, very well now. I'm not sure why fuzzy search causes so many problems on my site, maybe it is something that can be improved or tweaked or maybe it doesn't work well in ES yet. It might be because my items include words that would be otherwise uncommon in English since all products are sourced from Japan. It seams that BRAD (or ES?) puts equal weight on text found in item titles as it does on text found in item descriptions. Personally I would sooner the weight be put on the item titles and in other structured data such as manufacturer, category, attributes, and features. Being able to turn off description indexing altogether would be nice. Alternatively (and maybe even better!) would be to give the customer the option to search descriptions or not and let the shop owner decide if this option should default to "on" or "off". Perhaps these last two ideas aren't possible with ES, I'm not sure how it works internally. I turned "fuzzy search" off on my old 2GB dev box as well and it still had problems, but maybe I was too focused on why hardcore was causing so many troubles and didn't try enough other terms. Now I'm stoked to get 10000+ items onto a site and see how ES compares to the standard PS search! If the chart at the top of the http://bradsearch.io/ page is to believed then ~10,000 products is where the difference in speed becomes very visible. ~~Unfortunately when testing nothing has changed.~~ ~~If you visit https://www.inxonline.com you can try the search yourself, in the smaller search box with the black search button. (The top search is still using the standard 30bz search and it works fine.)~~ ~~Try searching for:~~ ~~1. hardcore~~ ~~This should bring up a list of fishing lines. They are the only items in the product database that contain the word hardcore. However what comes back is some fishing reels.~~ ~~2. g-soul x8 line~~ ~~This should also bring up some fishing line but instead returns Casio watches. I can kind of understand the similarities between "g-shock" and "g-soul" but with the extra terms it seems like it should work?~~ ~~3. casio g-shock~~ ~~This does return some Casio watches, but most of them aren't G-Shocks.~~ ~~I'm sure there are other examples as well but these are the first ones I tried.~~ ~~I guess there is either a problem with the way the data is being sent for index or with how the searches are being sent.~~ ~~Thoughts?~~

Another thought: Currently for carriers we can set maximum dimensions and a maximum weight. However it would be fantastic to also be able to set a maximum PRICE. In the USA, for example, First-Class Mail International "Small Packet" items can only be valued up to US$400. Items more expensive than $400 must be sent using a different service. Japan has a similar limit, and I suspect many (perhaps all) other countries do, too.

This post reminded me of a big frustration I have had with the PS shipping system. By extension this also means the 30bz shipping system as it is so far unchanged. The problem is that if you have two or more items in the cart that do not share any common carriers the checkout page presents odd combinations of shipping options and gives no indication of what they are. It just gives one or more prices. In some cases the cart will just say "no carrier available" and the customer can't check out at all! A lesser problem (probably not a problem at all in many cases) is that if only one carrier is shared between all items in the cart then that will be the only shipping option presented to the customer. Not being able to effectively combine carriers is a problem when a shop wants to offer free shipping on some items but billed shipping on other items, or if they have some items that require special shipping (oversize etc). As an example a shop may have "Express Mail Free" and "Express Mail Billed" carriers, along with "Registered Airmail Billed", "Economy Airmail", and "DHL Oversize". If we are doing automatic combinations my suggestion would be something like this: Assign a "Speed Grade" (SG) to each carrier. The ability to do this already exists in PS / 30bz. Using my examples from above: DHL Oversize: SG 9 Express Mail Free: SG 7 Express Mail Billed: SG 7 Registered Airmail Billed: SG 5 Economy Airmail Billed: SG 3 Have a new page in 30bz where names (and maybe descriptions) can be assigned to different Speed Grades. Each Speed Grade gets one name, and one description. During the checkout process, carriers get grouped by Speed Grade and the names/descriptions assigned in #2 are shown to give information to the customer about what the shipping options are. Example 1 Item 1: Carrier "Express Mail Free" (SG7) Item 2: Carrier "Express Mail Billed" (SG7) These two different carriers share the same Speed Grade and would therefore be automatically combined and presented as a single shipping option using the name assigned to that Speed Grade. The total cost of the two carriers would be added together to get the shipping cost. Example 2 Item 1: Carrier "DHL Oversize" (SG9) Item 2: Carrier "Express Mail Billed" (SG7) / Carrier "Registered Airmail Billed" (SG5) Item 3: Carrier "Express Mail Free" (SG7) Once we get into more complicated combinations it becomes more difficult to effectively combine shipping options. My suggestion would be to build a list of all the available speed grades for items in the cart. Above would be SG5, SG7, SG9. Then build shipping options based on each Speed Grade. If an item does not have a carrier for a given Speed Grade then take the next HIGHER available speed grade. If there is no HIGHER speed grade then take the next LOWER speed grade. So for the above sample cart we would have shipping options: DHL + Express Mail (SG9 + SG7) DHL + Express Mail + Registered Mail (SG9 + SG7 + SG5) Prices would be calculated on a per-carrier basis and then added together for each option shown to the customer. Example 3 Item 1: DHL Oversize (SG9) Item 2: Express Mail Free (SG7) Item 3: Express Mail Billed (SG7) Item 4: Express Mail Billed (SG7) / Registered Airmail Billed (SG5) / Economy Airmail Billed (SG3) Item 5: Express Mail Billed (SG7) / Registered Airmail Billed (SG5) / Economy Airmail Billed (SG3) Combinations would be: DHL + Express Mail (SG9 + SG7) DHL + Express Mail + Registered Airmail (SG9 + SG7 + SG5) DHL + Express Mail + Economy Airmail (SG9 + SG7 + SG3) Sometimes a given combination won't make sense. For example in Example 3 it's very possible that the cheapest option will be the fastest option, DHL + Express Mail due to the way the carriers are assigned. However this system would still be, IMO, a big improvement over the existing system that can't handle combining carriers at all. Thoughts? Ideas? (Edit: 1,2,4... I need to learn to count apparently!)

Osaka is a city with a lot of character. The people are much more outgoing than most Japanese, and more willing to be individuals. It's the most "asia-like" city in Japan really. It's the center of food and comedy for Japan, and the Kansai Region (Osaka, Kobe, Kyoto, Nara and some other surrounding places) pretty much was Japan until the government was moved to Tokyo in 1868. It's an interesting place to live and there's always lots of mental stimulation.

Domino's closes here at midnight, 1am in some places. McD's tends to be 24 hours, and of course there are places like Yoshinoya that are open 24 hours. Lots of options open but at this time of night none of them deliver.

Hope it's good. :) 4am here in Osaka right now and no pizza available. Maybe time for some ramen...

Done!

What's a pizza cost where you are?

I'll PayPal you the funds for it. Seriously.

I would be more than happy to test! I could send some pizza or something too, if you like. I'm utterly amazed with 30bz and really want to support the project.

@mdekker said in Let's talk about Search!: Would you guys like to help with coding? I cannot express how much I wish I could code in PHP so that I could help with this. I'm a business automation guy and most of my work today is in C#, VBA, and VBdotNET. In a past miserable life I did some SAP work in ABAP and many, many years ago I did work on mainframes and minicomputers. (IBM and DEC VAX, mostly COBOL but a little PL/I). I can basically read PHP but I wouldn't trust myself to actually write any code. :( Edit: System wanted to turn vb . net into a link so had to edit.

I think the theme needs to be modified to include whatever block it is that contains the filters. Each theme would be different I guess which is probably why the module doesn't make any changes itself.

I'm using the standard 30bz template and i don't think the filters show up on the search result page. I may not have done something properly though, I'm still a search n00b.

It's a pretty cool module, I was legitimately excited when I came across it. I really hope I can get it working, and figure out how to make the filters work (and figure out how to put them into the template too!)

It worked really well for me too, until I tried to index ~1100 items with a 2GB server. I really think it is probably the underpowered server causing the problems. Did you do anything in particular to set it up or just let it rip?

I think for basic "type things in and get results" search that would be fine. Ideally eventually it would be great to have a way to use more things like bradsearch does where results can be filtered based on price, color, size, etc, whatever is useful for the site. The biggest issue is that everything still needs to be sent to elasticsearch to be indexed. It works similar to how Algolia works where all data is sent to ES via JSON to be indexed. Queries are also sent via JSON, and results come back as JSON. Then the results need to be pulled out of the JSON and displayed to the user. Reindexes can be scheduled as cron jobs. I'm going to try contacting Invertus to see if they have any idea why things didn't work with my development system. I just had a look at the brad page and my post was tagged as spam by Disqus, not sure why! I clicked on the "this is not spam!" button and hopefully they clear it and post it soon.

They seem to work on bradsearch in bursts. A few months ago they did a fairly major upgrade and it now support ES 5 as well as ES 2.4. I suspect the problems I am having with brad are of my own making, mostly a very under-spec'd server. Less than 8GB of memory is not recommended on a dedicated ES VPS, my dev VPS has only 2GB and I'm running apache, nginx, redis, mysql, etc. Adding ES to that mix seems to be asking for trouble! That said, the documentation for bradsearch is pretty thin and I really don't understand wtf I am doing yet. Always happy to learn and search is something that has intrigued me for a long time but I'm not sure how long it will take me to get things straight.

Yes, you could probably have a shop anywhere in western Europe use that server without performance problems. I suspect it would happily support quite a few shops, but I don't have enough experience to say how many or how many records. I'm still in the very early days of learning about search. I had hoped that bradsearch would be a mostly turnkey solution that could be used with 30bz but I was seeing some very odd results. I would search for a term and completely unrelated items would be returned, items that I know don't contain that term anywhere at all. Elasticsearch is used by some very large websites (Facebook, Wikipedia, eBay, etc) so I know it's a capable search platform. There are a bunch of potential reasons it isn't working: Something is wrong with the way bradsearch is configured. A problem with how Elasticsearch 2.4.5 is configured on my Cloudways server. For some reason my server is corrupting the indexes (lack of memory would be my guess) My sad 2GB dev box is causing it to freak out (very possible IMO) There is an incompatibility between brad and 30bz (I doubt this as the db design is very similar to PS 1.6.1.x) I've screwed something else up along the way I suspect it's a memory issue. I might set up a couple of 16GB VULTR boxes for a month in Tokyo and put ES on both of them in a two-node cluster config and see how that goes. My ultimate goal is to have one ES setup per region that I have websites and to serve 10~20 sites from each ES box. I'm hoping this can be done with a single 16GB ES server but that may be wishful thinking. I'm a long, long way from that though!

You can put your Elasticsearch server anywhere you want, but the further it is from your website the more latency there is going to be for those search results. You can also share one Elasticsearch server between as many databases as you like, as long as the server has the hardware to run the number of searches that are being sent its way. For best results you would want to have the Elasticsearch server in the same datacenter as your web & database servers but having it somewhere nearby (same basic geographic region) really shouldn't be a hugely noticeable issue. This is also how Algolia works. You can also install Elasticsearch on the same server as your web and/or database server. If you have enough cores and enough memory it should be fine but two smaller servers is probably going to be better and may even be cheaper. Finally, about hardware, Elasticsearch performs at its best with 32GB of memory. More than 32GB can be problematic. 16GB seems to be okay for less busy installations and people do run it with 8GB. However servers with very small amounts of memory (<8GB) seem to have problems with running out of memory and/or crashing. Elasticsearch can be run in a cluster if more than 32GB is needed but somehow I don't think any of us are going to have a need for that. Edit: 32GB is the amount ES ideally wants to use. The best overall total for the machine seems to be 64GB, ~30GB for ES and the remainder for the OS & a very big disk cache. It also seems to be very important to make sure the correct I/O scheduler settings are used for SSD vs HDD. Lots of good information here, directly from Elastic.

Can anyone of you experts explain what 10000 records mean? I make no claims of being an expert, but I had the same question and managed to dig up some information. On Algolia a basic record is one item, and you can add as many attributes to that item as you like (Source) as long as the total size doesn't surpass 10KB of minified JSON (Source). However it's not quite as easy as that. Each index you create multiplies the number of records. So if you have 10,000 products that you want customers to be able to sort by price ascending and price descending then that is two indexes which means 10,000 products + 10,000 for price asc + 10,000 for price desc, a total of 30,000 records. So each new index you add (in each direction) adds another entire set of records. (Source). This pre-sorting is one reason why Algolia is so fast, but it does create a lot of records in a hurry. 100000 operations? Each time a search or sort is performed it is an operation. Also when you are adding or updating records it causes indexing, and each indexing event is an operation. There will be one operation per item you add or update. (Source). If you have 10,000 to 20,000 items and don't get too crazy with sort options then the $49/month plan should be sufficient, unless you have a very high traffic site. Multiply those numbers by 10 for the $249/month plan. You can also share one Algolia account between multiple websites. So if you have 5 websites each with 20,000 records they should all fit into one $249/month plan.

I uploaded about 1000 products to my dev site and then gave brad/elasticsearch a more serious trial. Unfortunately the results were not great, but I'm not sure exactly why. I've contacted the company that made the free module but as it's a free module I definitely don't expect a speedy reply or a lot of free support from them. They do use the module for their own customers though so it would be in their own best interests to fix any problems it may have. My dev server has only 2GB of memory which really isn't enough for a proper elasticsearch system. I may try boosting it up to 8GB or 16GB and see if that works, or I may try putting Elasticsearch onto a different server and seeing how that works. In better news, the 30bz search doesn't seem any different in speed with 1000 products than it does with 10 products. My next step will be to push towards 10,000 products and see how search works then. I also look forward to the Algolia module and seeing how it works and how much it would cost on a monthly basis. Even if it is more per month (within reason of course) it could be worth it just to have a turnkey & supported system.

I'm Canadian but have lived in Osaka for about 25 years. We sell a variety of things. On the "new product" side we sell Casio/Seiko/Citizen watches, Shimano/Daiwa/Miya reels/rods/other tackle, lines & lures from many Japanese companies, some music-related items, some AV stuff, some computer parts & accessories, some camera equipment. On the "used product" side we sell used photo gear (mostly lenses) as well as used handbags, LV and Chanel in particular. In the past year we have not focused much on the used business but do plan to get back to it later this year. Rakuten is a Japanese company and one of the world's largest eCommerce platforms. They also own a bank and some other businesses such as a point system used by many companies, credit cards, etc. The site layout is very Japanese which means it will make your eyes bleed to look at it for too long. Here is a item page from a Rakuten seller that I just pulled at random now. I've never looked at this item before now but it's an example of a typical product page on Rakuten. The majority of information on the page has nothing to do with the product being sold and you actually have to dig around on many pages to find product information. This is quite common on Japanese websites.

The biggest eCommerce sites in Japan are Amazon.co.jp, Rakuten, and Yahoo Auctions. Customers will expect natural Japanese descriptions and communications, and most won't be too interested in buying items shipped from overseas. Most Japanese don't speak any language other than Japanese so dealing with anything that isn't sold, supported, and described entirely in Japanese is not of much interest.