Newsletter update Oktober 2021! Version 1.3.0 is here & new backer perks!

[Smile]

A little later than expected but finally it is here. Thirty bees version 1.3. This is a major release and unless security updates are needed we expect this version to be there for the long term. Next major version 1.4 will be released in approximately 6 months.

What is added in functionality in thirty bees 1.3?

Please check carefully if you need to change something like translate or updating mail templates.

We have added:

• merge accounts of users (old + new email can be merged) - Passwords will not be sent by email but can be changed by clicking the recovery link in the email (update template if you customised it)
• coreupdater - new version that tracks errors so we can follow-up on upgrade issues
• work queue - mechanism to execute work jobs asynchronously in order to speed up request processing
• theme extensions - ability to extend themes, which allows us to introduce new front office features like new password reset
• work on php8 compatibility - module installation overhaul (1.6 still supported)
• product page tabs permissions - one can give for example a translator only access to certain parts
• Also note that version 1.3 is the last version that supports php 5.6. If your store still runs on this old php version, please update soon. If you fail to do so, you won’t be able to update to the next major releases.

We have to introduce new perks for super awesome backers and higher!

• Super Awesome Backer: We aim to view your post within a few days and will give you an answer and direction to look in to the problem so you can try solving it yourself. (Mention @datakick in your post)
• Bronze Sponsor We aim to view your post within a 3 days, and provide a concrete solution. (Mention @datakick in your post)
• Silver Sponsor We aim to view your post within a 48 hours, and provide a solution that you can directly implement. (Send a forum message to @datakick)
• Gold Sponsor We aim to view your post within a 24 hours, and provide a solution that you can directly implement. (Send a forum message to @datakick)

How to use this service? Well you have to sign up as backer and after the first month we will start fulfill the backers advantages. If you then post a topic just send the topic link in a privet message to @datakick he will then take care of you! If you think we can do this even better, just let us know your ideas and we will try to enhance the backers program. You can post your topic in a regular forum or in the supporters forum.

And as always: Don’t forget to join as a backer! Small or big we enjoy the beer we can buy from it. You can become a supporter for as little as $5 per month. And when you tip in more we can make nice enhancements to thirty bees.

[musicmaster]

What is the work queue? And to what extent will it be something programmers need to take into account when they develop something for TB?

[Mark]

What exactly is tb tracking? Is it just errors, where is the data that's being sent to the API?

Or is it non error tracking as well?

What exact data are you tracking please, where is data stored that we send and how is that going to help a) tb b) the merchant

I thought there was going to be a permission system on what data would be sent to tb @datakick?

[datakick]

@musicmaster / @Mark thanks for the questions. I don't have time to answer right now, but I'll make sure to reserver some time for detailed explanation of both features later, sometimes during the weekend. 

[the.rampage.rado]

@datakick, when speaking for newsletters I think you should check the URLs of the emails that the forum sends when there's an answer to some topic.

Every time the link is broken like this one:

Quote

 

jmeca has posted a comment on a question, Upgrade to 1.3.0 not working
 

Posted in Upgrade to 1.3.0 not working   Hi, so as there are problems with the coreupdater in what other ways can we upgrade to the next versions?

 

This happens only on mobile for some reason. In incognito, loged in or in Firefox it shows the proper content.

And the button "Go to the post" links to:     Does mobile chrome somehow truncate the URLs in this case?

 

Edited October 22, 2021 by the.rampage.rado

[Mark]

On 10/23/2021 at 2:18 AM, datakick said:

@musicmaster / @Mark thanks for the questions. I don't have time to answer right now, but I'll make sure to reserver some time for detailed explanation of both features later, sometimes during the weekend. 

Look forward to it @datakick

Edited October 29, 2021 by Mark

[zen]

great post that i am following

[Mark]

On 10/23/2021 at 2:18 AM, datakick said:

@musicmaster / @Mark thanks for the questions. I don't have time to answer right now, but I'll make sure to reserver some time for detailed explanation of both features later, sometimes during the weekend. 

Apppreciate it @datakick

[datakick]

Sorry about the delay, I totally forgot about this one 🙂

On 10/22/2021 at 12:01 PM, Mark said:

What exactly is tb tracking? Is it just errors, where is the data that's being sent to the API?

Or is it non error tracking as well?

What exact data are you tracking please, where is data stored that we send and how is that going to help a) tb b) the merchant

I thought there was going to be a permission system on what data would be sent to tb @datakick?

This new tracking feature in 1.3.0 is basically a mechanism to send anonymous usage data to thirty bees api server. We made very sure that no personal or even domain information are being shared with using this mechanism, so it's aligned with GDPR requirements.

At the moment we track only the very basic info about the installation: php version, installed php extensions, database version, and basic server settings (max exec time, memory limit, etc). In the future, this mechanism will be extended to track other informations like feature usage. 

Why do we need this information? There are couple of reasons, of course.

First of all, we are investing a lot of effort and money into this thirty bees enterprise. And we need to have some way to actually measure the adoption of the software. When we were considering purchasing thirty bees we discovered that nobody actually knew how many stores are using this software in the wild. That's not good really. Without this information, it's quite hard to prepare business plan, monetization strategy, or whatnot. Merchants are using google analytics, matomo, or other tracking software to gather this information about their visitors. We are using this new feature to do the same.

This first reason behind this feature is not something that merchants benefits from, thats for us. But there are other reasons that have (or will have) direct impact on merchants. With the collected statistics we can much better plan the development.

For example, we are considering dropping support for some php versions - one of the reasons being that it makes it very hard to update dependencies / third party libraries might not support older version of php. But we don't want to just drop the support out of the blue, and leave merchants in the bind. Before we do such drastic thing, we want to be sure that the impact will be minimal. 

Another example: feature usage, that we currently do not track, but we want to - if we knew how many merchants are actually using some feature (ASM, multistore, theme, module), we can much better plan our efforts. If we knew that only 5% of merchants are using ASM, then probably it's not a good idea to spend a lot of effort in fixing the bugs, but maybe it's time to consider dropping this feature from the core (moving to module) instead.

Php extensions are another interesting info to have, as it can answer the questions about potential features that might depend on some extension. 

Regarding consents. As I already wrote, the collected data are anonymised, there's no law or regulation that requires for us to obtain a consent. Regardless, we want to have the merchants the ability to decide if they want to help us collect this info or not. Unfortunately there wasn't enough time to implement all aspects of this consent mechanism. Currently, there is only a database table that governs consents. If you do not want to allow sending the info to api server, you can update table `tb_tracking_consent` and nothing will be sent to us. Of course, changing the consent in DB is not very user friendly 🙂 In the next major release there will be UI for this. There were other, more imporant tasks, to implement for 1.3.0. These consents will gain more importance once there are more 'personal' type of data to be collected.

On 10/20/2021 at 7:52 PM, musicmaster said:

What is the work queue? And to what extent will it be something programmers need to take into account when they develop something for TB?

Work queue is a mechanism to run tasks outside of the main execution thread 🙂 

Yeah, I know. That didn't help much. 

During execution of php script there might be some tasks that do not need to be executed immediately. For example, when you change order status, an email might be send. Sending email can actually take a lot of time, and that impacts the page response time. Employees have to wait couple of seconds just to see the result of this simple operation. Also, what happens if the smtp server is currently down? Yeah, the process will fail, back office page will display error message, and employee do not know what happened.

With work queue we do not send emails. Instead, we just describe the 'Send email' operation as a task, and submit it to work queue executor, and immediately continue. Work queue executor will ensure that the task is executed as soon as possible. The task description can be serialized and stored somewhere. This allows us to actually run work queue executors on different servers, or in different process on the same server. It also allows us to try to recover from errors -- for example, if sending email fails, we can retry it later after some delay. 

There are a lot of operations that can be converted to work queue tasks. Clearing cache, generating images, webservice calls, cleanup tasks, and more. 

By converting these operations to work queue tasks, we can also trigger them differently. We can have a cli tool to actually run these tasks -- so you can ssh to your server and run command to clear cache, or regenerate the images. These tasks can also be scheduled using build-in scheduler. 

In 1.3.0, we only introduced the basic mechanisms for work queue, mostly because of scheduler that we needed. There is still a lot of work on this functionality. For example, we need to actually implement different types of work queue executors. Currently, we only have 'Immediate executor', that runs submitted tasks immediately in the context of the current thread. But in the next release, we will introduce new type of executors. There will be

• immediate executor -- currently available, executed tasks immediately in the context of the thread. Does not support retrying. There is not really a benefit, apart from error handling.
• deferred executor -- the tasks will be executed in the context of the php thread, but only **after** the response is send to the user (not all servers supports this feature, though). This will increase page response time.
• standalone executor -- standalone php application that can run on the same server or on other server as well. Will receive the work queue tasks via some queue (redis, kafka, memcache...). This is solution for serious merchants that want to run high performant stores

I'm happy to answer any questions.

[Briljander]

Cool, I am not sure how I will benefit from it except from the email tasks being sent later but I guess it's a little bit over my head in form of technical information. I do still think it sounds really great and looks advanced and awesome.

About the tracking, fact is always better than guessing!

[Mark]

16 hours ago, datakick said:

Sorry about the delay, I totally forgot about this one 🙂

This new tracking feature in 1.3.0 is basically a mechanism to send anonymous usage data to thirty bees api server. We made very sure that no personal or even domain information are being shared with using this mechanism, so it's aligned with GDPR requirements.

At the moment we track only the very basic info about the installation: php version, installed php extensions, database version, and basic server settings (max exec time, memory limit, etc). In the future, this mechanism will be extended to track other informations like feature usage. 

Why do we need this information? There are couple of reasons, of course.

First of all, we are investing a lot of effort and money into this thirty bees enterprise. And we need to have some way to actually measure the adoption of the software. When we were considering purchasing thirty bees we discovered that nobody actually knew how many stores are using this software in the wild. That's not good really. Without this information, it's quite hard to prepare business plan, monetization strategy, or whatnot. Merchants are using google analytics, matomo, or other tracking software to gather this information about their visitors. We are using this new feature to do the same.

This first reason behind this feature is not something that merchants benefits from, thats for us. But there are other reasons that have (or will have) direct impact on merchants. With the collected statistics we can much better plan the development.

For example, we are considering dropping support for some php versions - one of the reasons being that it makes it very hard to update dependencies / third party libraries might not support older version of php. But we don't want to just drop the support out of the blue, and leave merchants in the bind. Before we do such drastic thing, we want to be sure that the impact will be minimal. 

Another example: feature usage, that we currently do not track, but we want to - if we knew how many merchants are actually using some feature (ASM, multistore, theme, module), we can much better plan our efforts. If we knew that only 5% of merchants are using ASM, then probably it's not a good idea to spend a lot of effort in fixing the bugs, but maybe it's time to consider dropping this feature from the core (moving to module) instead.

Php extensions are another interesting info to have, as it can answer the questions about potential features that might depend on some extension. 

Regarding consents. As I already wrote, the collected data are anonymised, there's no law or regulation that requires for us to obtain a consent. Regardless, we want to have the merchants the ability to decide if they want to help us collect this info or not. Unfortunately there wasn't enough time to implement all aspects of this consent mechanism. Currently, there is only a database table that governs consents. If you do not want to allow sending the info to api server, you can update table `tb_tracking_consent` and nothing will be sent to us. Of course, changing the consent in DB is not very user friendly 🙂 In the next major release there will be UI for this. There were other, more imporant tasks, to implement for 1.3.0. These consents will gain more importance once there are more 'personal' type of data to be collected.

Work queue is a mechanism to run tasks outside of the main execution thread 🙂 

Yeah, I know. That didn't help much. 

During execution of php script there might be some tasks that do not need to be executed immediately. For example, when you change order status, an email might be send. Sending email can actually take a lot of time, and that impacts the page response time. Employees have to wait couple of seconds just to see the result of this simple operation. Also, what happens if the smtp server is currently down? Yeah, the process will fail, back office page will display error message, and employee do not know what happened.

With work queue we do not send emails. Instead, we just describe the 'Send email' operation as a task, and submit it to work queue executor, and immediately continue. Work queue executor will ensure that the task is executed as soon as possible. The task description can be serialized and stored somewhere. This allows us to actually run work queue executors on different servers, or in different process on the same server. It also allows us to try to recover from errors -- for example, if sending email fails, we can retry it later after some delay. 

There are a lot of operations that can be converted to work queue tasks. Clearing cache, generating images, webservice calls, cleanup tasks, and more. 

By converting these operations to work queue tasks, we can also trigger them differently. We can have a cli tool to actually run these tasks -- so you can ssh to your server and run command to clear cache, or regenerate the images. These tasks can also be scheduled using build-in scheduler. 

In 1.3.0, we only introduced the basic mechanisms for work queue, mostly because of scheduler that we needed. There is still a lot of work on this functionality. For example, we need to actually implement different types of work queue executors. Currently, we only have 'Immediate executor', that runs submitted tasks immediately in the context of the current thread. But in the next release, we will introduce new type of executors. There will be

• immediate executor -- currently available, executed tasks immediately in the context of the thread. Does not support retrying. There is not really a benefit, apart from error handling.
• deferred executor -- the tasks will be executed in the context of the php thread, but only **after** the response is send to the user (not all servers supports this feature, though). This will increase page response time.
• standalone executor -- standalone php application that can run on the same server or on other server as well. Will receive the work queue tasks via some queue (redis, kafka, memcache...). This is solution for serious merchants that want to run high performant stores

I'm happy to answer any questions.

Great answer thanks @datakick and all important groundwork for the future of TB. Sounds like, for non anonymised and specific data, there will be a consent process put in place, which is where I was coming from. I did mention and suggest that TB did this myself probably a year or so ago, so its good to see it begin to happen...  !  It would be great for us to see exactly what data was actually sent to TB at some point.

 

Edited November 7, 2021 by Mark

[the.rampage.rado]

11 hours ago, Briljander said:

Cool, I am not sure how I will benefit from it except from the email tasks being sent later but I guess it's a little bit over my head in form of technical information. I do still think it sounds really great and looks advanced and awesome.

About the tracking, fact is always better than guessing!

On top of my head here's the most pressing issue with TB that can be fixed with this solution - order finalizing. When customer clicks on "Confirm my order" it usually takes up to 5-10 seconds to refresh the page. In the meantime emails are sent, order is made, etc. This mechanism could potentially reduce the time in this vital step and improve UX.

 

I had numerous questions about this issue from inpatient clients clicking numerous time or even 'having issues on the last step'....

[wakabayashi]

@datakick Thx for all the explanations. What I haven't got yet: what is the work queue technically? Is it actually a well organized/structured cronjob? Cause what you describe, are we doing over cronjobs. Instead of doing a task, we just save it to a "cronjob table" and every minute the cronjob is checking, if he has some tasks to do ☺️ That's why I wonder, if it's actually pretty much the same or if it's not cronjob related!?

[datakick]

On 11/10/2021 at 11:40 AM, wakabayashi said:

@datakick Thx for all the explanations. What I haven't got yet: what is the work queue technically? Is it actually a well organized/structured cronjob? Cause what you describe, are we doing over cronjobs. Instead of doing a task, we just save it to a "cronjob table" and every minute the cronjob is checking, if he has some tasks to do ☺️ That's why I wonder, if it's actually pretty much the same or if it's not cronjob related!?

Yes, cron is one possible implementation for Work Queue Executor. Not the only one, though 🙂

There should be some implementations that does not need any additional configuration, implementation that works right out of the box after the installation. This will be true for Immediate and Deferred executors. Then we can have advanced executors (standalone, cron-based,...), but for them to work properly additional infrastructure will need to be implemented. For cron-based implementation there will have to be cron endpoint. For standalone, there will need to be redis + requirement for VPS to run dedicated php cli application... 

[musicmaster]

According the announcement Thirty Bees 1.3 should work with PHP 5.6 and higher. However, when I try to install it under 5.6 I get an error that PHP 7.2 is needed.

[datakick]

14 minutes ago, musicmaster said:

According the announcement Thirty Bees 1.3 should work with PHP 5.6 and higher. However, when I try to install it under 5.6 I get an error that PHP 7.2 is needed.

 

https://thirtybees.com/download-specific/

[musicmaster]

5 minutes ago, datakick said:

https://thirtybees.com/download-specific/

I have no idea how I could arrive on that page. On the first post of this forum thread it says 5.6. When I use the menu to find the download page I get to https://thirtybees.com/download/ that doesn't mention PHP versions at all. It has a tab release notes for each version but when you click that button you come on a Github page without any information.

[datakick]

3 minutes ago, musicmaster said:

I have no idea how I could arrive on that page. On the first post of this forum thread it says 5.6. When I use the menu to find the download page I get to https://thirtybees.com/download/ that doesn't mention PHP versions at all. It has a tab release notes for each version but when you click that button you come on a Github page without any information.

Yes, it's mentioned on the primary CTA on download page.

Default download package is indeed for php7.2, since it's very unlikely that any new installations would be done on php 5.6

Support for legacy php versions are mostly for existing stores, and they do not need to download packages. But in rare case anybody needs it, it is there.

 

[musicmaster]

1 minute ago, datakick said:

Yes, it's mentioned on the primary CTA on download page.

Default download package is indeed for php7.2, since it's very unlikely that any new installations would be done on php 5.6

Support for legacy php versions are mostly for existing stores, and they do not need to download packages. But in rare case anybody needs it, it is there.

 

Only visible when you move your mouse over it. That is why I missed it. Maybe such a design is not such a good idea...

Actually it is important to be clear about the requirements of 1.3. I am migrating a shop and was planning to immediately upgrade it to 1.3. That failed for other reasons.

BTW after the migration I get on some pages a duplicate key error in the ps_referrer_cache table. Any idea what that might be?

[datakick]

5 minutes ago, musicmaster said:

BTW after the migration I get on some pages a duplicate key error in the ps_referrer_cache table. Any idea what that might be?

Not sure. Maybe you have truncated ps_referer / ps_connection / ps_connection_source tables, but forgot to truncate ps_referrer_cache as well? Go to Stats > Referrers, and click on Refresh Index and then Refresh Cache. That might sync those tables together.

[musicmaster]

That sounds quite likely.

However, the Stats > Referrers page gives an error:

[datakick]

45 minutes ago, musicmaster said:

That sounds quite likely.

However, the Stats > Referrers page gives an error:

 

This is error in 1.0.8, fixed in 1.1.0 by https://github.com/thirtybees/thirtybees/commit/840a9d6b0e54a5e8fb46cbf52242348f5651339b