Indiegogo ElasticSearch project

[dynambee]

@Traumflug said in Indiegogo ElasticSearch project:

As far as I understand matters, thirty bees isn’t about demo’ing ElasticSearch

Why do you think this would be about demoing Elasticsearch? If you have a suggestion for a better as-yet-unmentioned high speed search engine I would be very happy to hear about it. Here is a list of the basics that the search engine needs to provide:

• High performance full-text search

• High performance faceted search (this allows for high speed result filtering)

• Should support synonyms, typos/misspellings, autocomplete, and search suggestions
• Flexible indexing with index weighting
• Scaleable from small shops with a few thousand products to very large shops with 100000, 1mil, or more products.
• Free (and Open Source so it will stay free.)
• Relatively light on resources so smaller shops can run it on their web server.
• Easy to configure
• Should require little or no store-owner management (like Apache or Redis, once installed they mostly "just work")
• Should provide a standardized API that the module can use to easily execute searches -- nothing based on complex SQL queries.
• Under active development with a strong community already in place so the future is relatively bright

To my knowledge there is only one currently available solution that meets the above requirements and that is Elasticsearch.

High speed search is a complex topic and I'm the first to admit that there are many things that I don't know or understand. However high speed search has fascinated me for years and I have done quite a bit of reading & research. If there is 30bz-suitable OSS search engine out there that I don't know about I would like to learn.

Right now this sheet reads much like “we barely know what we want, but we want everything”. A nice recipe for disappointment.

The current spec sheet is based on features currently provided by the best ES modules. There is no single module for PS that provides all of the mentioned features but everything mentioned is possible to do with ES. There are more features not mentioned that ES can also provide. None of this is wishful thinking or created from a lack of understanding of what is possible or what ES is able to do.

Discussing and refining a spec sheet not only expresses user/merchant/donator expectations, it also helps aligning these expectations with the developer’s ideas and help him laying out a plan on how to write the code.

Discussion and improvement is the goal. Writing like you have a complete understanding of the topic when you actually haven't done even basic research is not helpful in moving forward.

[dynambee]

I look forward to the production of one or more of the potential modules and will certainly contribute to all of them. Also happy to beta-test, of course.

[wakabayashi]

It seems that this project is the favourite of most merchants (https://thirtybees.com/blog/crowdfunding-modules/). @lesley when will the official founding campaing launch?

[dynambee]

I've also been following the poll closely. Voting seems to have slowed down quite a bit and the overall balance seems to have stabilized with ES having around 50% of the votes. I'm surprised the Google Marketplace module didn't do better than it has. I hope we can fund that module at some point as well, I think it's a very important set of functions for many sites.

I do have two questions, and one module feature request.

---

Questions:

1. What license will this module be released under? My preference, for what it's worth, would be one of the GNU General Public License versions. GPLv2 seems to be the most popular even today but either v2 or v3 would be fine. These are the licenses with the most assurance that the code will always be open source free software.

2. Who will own the copyright for the module? I think this is especially important if the planned license is to be anything other than GNU GPL but regardless this should be clarified before the project is started.

---

Feature request:

The module should respect the 30bz Back Office "Visibility" setting. This setting currently has four options:

1. Everywhere
2. Catalog Only
3. Search Only
4. Nowhere

Items set to "Everywhere" and "Search Only" should be indexed for search. Items set to "Catalog Only" or "Nowhere" should not be indexed for search or should be removed from the index if they are already there.

These settings are saved in the tb_product table columns indexed and visibility.

Any time indexed is set to 0 the item should not show up in search. If it has already been indexed it should be removed from the index. If it hasn't yet been indexed then it shouldn't be indexed.

Any time indexed is set to 1 the item should show up in search. If it has't been indexed yet then it should be added to the index queue or added the next time a cron job runs, depending on how the shop owner has the module indexing options set.

[dynambee]

I have suggestions in response to my own questions above:

1. The code should be released under GPLv2 or GPLv3

2. The copyright should be jointly held by Thirty Bees LLC and the top 10 contributors to the project. Contributors would have the option to not accept copyright if they wish, in which case they would not receive the copyright and their name or company name would not appear in the copyright notice. \ Alternatively rather than the top 10, all contributors over a certain dollar amount who wish to receive copyright. The minimum amount should be low enough that multiple people will contribute the required amount but not so low that a single project ends up with 175 different copyright holders. 100 Euro sounds like a good number to me.

The reason for #2 is that the owner of the copyright can choose to release the code under additional licenses if they want to. So if the code is under GPLv2 and the copyright owner decides to also release the same code as a proprietary product and sell it on the PS store, they could do this. There is nothing wrong with dual licensing and MySQL is a great example of a company that did this well. However for a community funded project being released as Free Software there should be some guarantee that the software will always remain Free Software.

Assigning the copyright to multiple people makes it very likely that the software will forever remain under the GPL and GPL alone. The reason for this is that in order to change the license every single copyright holder must agree to the change. If even one copyright holder can not be found or does not agree to the change then no change can legally happen.

[dynambee]

I have another idea for the license & copyright that might be interesting, though unusual:

• Thirty Bees Inc owns the copyright and the license is not open source.

• Anyone who contributes at least 5% of the crowdsource goal gets a perpetual license (including upgrades) for the project's module to use on one website.

• Anyone who contributes at least 10% of the crowdsource goal gets a perpetual license (including upgrades) for the project's module to use on an unlimited number of their own sites, but not to resell or include with projects that are sold.

• If a project is extraordinarily expensive for some reason then these amounts could be lowered or even changed to a fixed amount instead of a percentage. The goal is to encourage people to donate by giving them a benefit that isn't available to non-donators and if the amount required is too high then people won't be able to afford to donate or it won't be financially justifiable.

• When the 30bz Addon Store goes live (hopefully soon! :)) the module is sold in the store for, say, 100 Euro. Some modules will be more expensive of course, some may be cheaper.

• 50% of the sale price goes to Thirty Bees Inc. The remaining 50% is used to fund things like:

  • New modules that follow the same path (fund up to 50% of the cost so people can contribute as well)
  • New features for existing modules developed under this program.
  • Upgrading modules developed under this program to maintain 30bz compatibility as the core develops and changes.
  • Possibly used for other things such as to develop high quality themes to include with the 30bz core. .

• Usage of the funds would be voted on by the community. In the first round of votes all potential options are shown. If no single option gets more than 50% of votes then the two most popular options should have a runoff vote to make sure the winning choice has the support of at least 50% of voters.

• If it ends up that there is too much money in the pot with no good way to spend it then the selling price of the already-created modules should be lowered. Modules should not become completely free however, but the price could get quite low if necessary. If more funds are needed later then the prices can be increased again.

• I think there should be an agreement that if 30bz is sold to a competitor or shuts down that the modules in their form at that time should be released as open source. Likewise if the crowdfund project is ever abandoned completely then the modules should be released as open source.

There are, of course, some potential problems with this model. One is that the community would be trusting the company to follow the rules. I'm a trusting guy and see no reason not to trust @lesley and @mdekker but I can also understand that some people may not be as trusting. I don't really see a way to do this easily without trust though.

The second big problem is taxes. If the money is not spent within the tax year then it's going to be seen as profit for the company and the company will be taxed. ~~The only way to avoid this problem would be for a separate legal entity to be created as a non-profit to hold these funds. This would have overhead as well, which would have to be covered by the income from the sale of the modules created. I don't think this has to be dealt with right away, we'd actually have to see income & profit from sales before having to worry about this aspect. Still, it is something to consider and be aware of.~~ Edit: Thinking about this more it seems unlikely that a non-profit could hold funds that would be later used to generate income for a profit making business. I suspect the IRS would view this as tax fraud, but I'm not a tax accountant nor am I tax lawyer. If this is going to be a problem then tax will have to be paid on the balance and the balance would be lowered. Best to keep projects rolling and thus avoid losing funds to the gov't. More modules for the platform! :)

Likewise there are some big benefits to this model. For now 30bz and PS are very similar. If the modules our community funds are released as open source they could be relatively easily adapted to use on PS and then put on the PS store as a free module. This is the way open source works of course and it's not generally a bad thing. However from my perspective one of the big reasons for doing this crowdsourcing at all is to support 30bz and encourage it's growth as a platform. It would be quite disappointing (for me, anyway) to see our crowdsourced modules sold on the PS store. This is one big reason to not use an open source license.

Another big benefit is that the sale of these modules would support Thirty Bees Inc. As we all know, if 30bz doesn't make money then it will collapse. That would be really bad for all of us so anything we can do to help avoid it happening would be great.

Finally I see another big benefit to doing things this way and that is to encourage contributions to the crowdfund project. On the blog post discussion thread a user pointed out that without benefits to the contributors it will be hard to attract people to contribute to the project. Maybe people will contribute to the 1st project, but what about the 2nd? The 10th? If not contributing gives the same benefits as contributing far fewer people will contribute. This solves that problem very nicely.

[lesley]

Selling it would be against what we have said we are going to do as a company, I don't think that is something we are going to do. The module can be designed in such a way that it will be very difficult to ever make work in PrestaShop and that is how it will be designed.

A simple break down of how we are going to use the funds is as follows, 80% of the goal for funding the module will go to the developer. The remaining 12% will be set aside for tax liability. Indiegogo takes 8%, so that is how the 20% is rounded out.

[dynambee]

Selling it would be against what we have said we are going to do as a company, I don’t think that is something we are going to do.

I can see how it could create a potential conflict.

However I believe what you have said is that 30bz won't sell their own in-house modules. As the development for these modules is being paid for by the community, if the community wants to use the modules to support 30bz shouldn't that be acceptable?

Of course I make no claims to speak for the entire community (or anyone besides myself!) Many more people need to chime in with their thoughts on this.

[lesley]

I think that is fine, we just would not be able to do them through us. The whole point in the beginning was to make an open source module. Having paid or closed source modules is a conflict of interest with our companies principals.

[dynambee]

Perhaps once the community is more established we could consider doing some crowdfunded modules that would support 30bz. I don't think there is anyone in the community right now who is not part of Thirty Bees Inc who would be trusted by enough of the community to manage such a project just yet. Perhaps that will change over time.

[lesley]

I think that would be a viable solution at a point in the future.

One thing that we are trying to guard against with not selling modules that we create is the idea that we leave features out, so we can charge for them. Basically like a freemium type software. That is something we are fighting against. I have seen other packages do that and it is not something I want us to get tangled in. If it means we make less money and grow slower, so be it.

[vincentdenkspel]

I don't know what your business model is, but i assume that a large user bases is better for you business model. Hight quality open source modules will make TB more attractive for a larger user base. Lets say the current voter on the blogpost will pay € 75 each than all the modules could easily be developed. This would make TB more attractive and the user base will grow. Next year or so the user base would be much bigger. For the sake of argument that 300 people would be willing to pay € 75 next year. For this €22.500 other great modules could be developed. Next year maybe 600 people are willing to contriute €75 for crowd funding projects. And so on and so on.

Someone mentioned that crowdfunding would not be direct beneficial to him because other people who are not contributing also will get the game. First of all he will not get these great modules for €75. Secondly lets say 2% of the current forum members contribute in one way or another to the project (coding, bug fixing, documentation etc. etc). It is likely that the number of forum members will grow and as such also the number of contributors. More contributors will make TB as a project better in less time. So this is also beneficial to you as crowd funder.

I think crowd funding can beneficial to TB. The problem is getting people convinced

[Havouza]

My view for what it is worth. First as I said before, there is a huge difference between a vote and a contribution. If you made a poll on would you contribute to a crowdfunded module far less than the 85 current others had voted yes. And a poll for how much you would contribute would not say 75 $, but far less. I am not at all negative to crowdfunding, but I am realist- Lets take the ES module. Looking at PS there is one free and one paid module. In my book that means that the demand is low, because if it was high, many more developers would have one for sale So how many of the 43 voters for that module could think of paying 75$ for it? The first project will perhaps give an answer, we will have to see

[vincentdenkspel]

@Havouza "First as I said before, there is a huge difference between a vote and a contribution" you're absolutely right. That's why it is important to emphasise the benifits of contributing to the crowd funding project compared to buying this functionality from third party developers.

"Looking at PS there is one free and one paid module. In my book that means that the demand is low, because if it was high, many more developers would have one for sale" Here i disagree with you. You only focus on the elastic search modules. As discussed before you will also be able to do advanced filtering on a site. Your analysis is thus not complete. You also have take into account the advaced filter modules. There are various filters modules but let only look at the once that are developed by Presto Changeo and PrestaModule. The one of PrestaModule has been sold more the 5000 times on the PS addon store. That means at least €1.000.000 in sales !! This module is also sold on there own store. The advance filter module is only sold on there on store. I don't now the number of sales. But i think it is fair to conclude that you can't say that there isn't a demand for such a module.

The problem is the focus on the search capabilities of the module. I think also the filter options much be brought into the lime line.

"but I am realist-" This suggest that i'm not. You're entitled to your option but i don't see why my view is less realistic than yours. Especially since your substantiation was incomplete.

All depends how the crowd funding project comunicated: "help us build a elastic search module" or '"help us build the best elastic search module. And the best part is: you will not only get the best Elastic Search module, you will also get the best filter module. So two for the price of one"

[Havouza]

Well we have to see what happens. But the marketing is a little wrong. You will get the best Elasticsearch module. The filter part is also ES, just that this part has not been used in the existing modules for PS. But exist in modules for other platforms

[vincentdenkspel]

"Well we have to see what happens." yes we have

"just that this part has not been used in the existing modules for PS" That's the problem. The more important to the emphasise all the functionality they get in one module instead of two. You can't assume that people know that. I did not know that until Michael told me.

[vincentdenkspel]

I would like to emphasise the importance of highlighting all the capabilities and not only the search features by using a metaphor.

Currently it's a good weather for cruising in a cabrio. Unfortunately i don't have one. So lets say I would like to buy one. There are various manufactures that build cabrio's that i like. However what i do not know is that there is a small manufacturer that happen to build trucks. What is special about this truck is that is a transformer-truck. It can be transformed into a cabrio if you want to. So in the weekend I can cruise in my carbio and during the week I can bring my sold games to my customers. Unfortunately the manufactured did not communicate this possibility and I end up buying a normal cabrio. Would I have know the other features than I would have bought the truck. So the manufactures lost a sale by not communicating the other features.

[dynambee]

Just for the record, Brad does offer faceted search (aka filtering of search results). The downsides to Brad are that the license is non-standard and that the back-office part does not offer detailed configuration of the module.

The Polish-created Elasticsearch Connector module for PS has a finer level of control in the back office but doesn't offer faceted search and uses a slower way to search ES and then display results. It also does not have an option for displaying mini-thumbnails in the drop-down search results.

There is a $250 ES module for Magento ($500 if you are using Magento Enterprise) that has quite a good front-office appearance. I haven't looked at their back office in detail and I don't know how the internals of their querying works. Their demo site is very underpowered and/or very heavily loaded (probably underpowered and shared by many demo sites for their modules, thus being also overloaded) and the entire demo site runs slowly, at least from Japan.

[lesley]

What do you mean by a non standard license? The license they offer allows it to be re-licensed.

[dynambee]

~~It doesn't allow the software to be relicensed, the current license and the notice must be included. So this code can't be used in a GPL product, for example. It might be usable with, say, a BSD-style license but that type of license doesn't protect the code from being used in a closed-source commercial product.~~

Edit: Incorrect information, corrected in messages below.

[lesley]

From my understanding it does allow for re-licensing. As it says, "Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), * to deal in the Software without restriction, * including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, * and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, * subject to the following conditions:"

If it could not be re-licensed it would not be without restriction. We do have to include the original license, but we have had to do that with all the other files that have a copyright and license on them. But if you notice, it expressly gives the right to sublicense.

[dynambee]

Combining code from BSD-licensed projects (and Brad's license is a permissive license like BSD, Apache, MIT, etc) with GPL-licensed "copyleft" code is messy but not as impossible as I misremembered it to be (my bad). There is a very detailed writeup available on this very practice, including information on mixing code from different licenses inside the same file.

[lesley]

Honestly we can use a similar type license or just stick with the license in it and add our own copyright as well. Its not anything huge on our end, the same basic point we have in mind would apply to either license.

[dynambee]

It depends on the goals of the license. Permissive licenses like what Brad uses (and more standardized permissive licenses like BSD etc) allow their code to be used in commercial products and don't have a requirement to provide source code. Microsoft uses some BSD code in Windows, for example, and this is perfectly fine.

Producing a full search module for 30bz and licensing it with a permissive license would allow anyone to modify the code and then sell the resulting product on the PS store, pocketing the profits. I would have a hard time contributing financially to a 30bz module that could be reused in such a way. I want the module to benefit 30bz, not potentially be modified and sold by someone for PS.

[lesley]

Unless I am wrong we can license new files in any way of our choosing, having an end module that has multiple license types. Likely a lot of new files are going to need to be created for the front views, new filter classes, ect. We can severely cripple any commercial use that way where anyone would just as well fork the Brad module as to try to use ours for commercial purposes.