Jump to content
thirty bees forum

GitUpdater preview


Recommended Posts

The flawed update mechanism

Perhaps you remember the 1.0.8 update a couple of weeks ago. While it certainly solved bugs and misalignments for many, it also resulted in quite a number of reports about messed up shops after the update. That's not tolerable, of course.

Thinking about this problem I compared to what we developers do to switch between versions, and why we pretty much never mess up a shop installation by doing so, despite switching back and forth all the time. Developers do a git checkout 1.0.7 to switch to version 1.0.7, git checkout 1.0.5 to switch to version 1.0.5, git checkout 1.0.x to switch to the latest development, and so on. And there's the fundamental distinction: doing such a checkout doesn't apply a patchset or a set of changed files, like the current updater module does. When checking out a version, Git compares files currently on disk with kind of a master set of files (stored in the .git directory), then changes files on disk to make them match this master set.

The current update module, which is a refined version of Prestashops 1-click updater, has a fundamental flaw: it doesn't check whether files on disk are right, but blindly applies a set of changed files. Who creates this set of changed files? Right: thirty bees developers, which have never seen your shop installation. This set of changed files gets created not on knowledge, but on assumptions. The assumption that your shop installation matches a clean shop installation. Which is often not true, as we unfortunately experience over and over again.

Meet GitUpdater

Having the fundamental flaw detected, a new updater module started to evolve. Prepared sets of file changes are gone. Instead, the module compares the shop installation to a clean master installation, which gets served by api.thirtybees.com. This always works, and after an update, a shop will always be in a clean, sane state. A recipe for reliability.

Advantages

  • This works from any release to any release. Select the wanted version, click Compare and see which files changed since then.
  • Which means, one can downgrade or rollback just a easily as update.
  • Another use case is to compare the current installation to the master installation. Updating from e.g. 1.0.8 to 1.0.8 is perfectly possible, to restore a broken shop.
  • Before too long, this will also support updating to development branches. E.g. to test bug fixes provided on Github. Or even single commits.
  • Manual changes get detected. During comparison, the module detects files which don't match the original version, so the merchant can deal with his manual edits.
  • Smooth operations. All operations taking place got split into small chunks, so one can virtually never run into maximum processing time limits.

The preview

Here it is:

gitupdater-2019-01-05.zip

Install it by unpacking in the modules folder or using Add a new module (top right corner) in back office -> Modules & Services -> Modules & Services.

Having the module installed, a new menu item, Updater, appears in menu Preferences. That's where this module gets controlled.

Yet To Do

  • Actual updates. Currently the module does comparisons only, not actually change files. Which is why playing with this module preview is safe.
  • File filtering. Some files should get ignored. For example, an update shouldn't reinstall the install/ folder despite this folder being present in a clean installation. It also shouldn't remove product images or translations, despite them being not present in releases.
  • User interface. Currently, this module shows a list of all changed/created/removed files. This can be a huge list with up to 10,000 lines. Ideas to display this list in a less scary way are welcome.

And for those not installing the module immediately, here's a screenshot:

0_1546800980968_Screenshot GitUpdater 2019-01-06.png

  • Like 3
  • Thanks 1
Link to comment
Share on other sites

I just realized I have a huge problem after updating to 1.0.8 after the last update with the old updater. My entire logo/header/menu is gone on specifically mobile. Works perfectly on desktop. If I re-update with this module would this possibly fix this, or do you have any idea what may have caused this?

Link to comment
Share on other sites

@traumflug said in GitUpdater preview:

thirty bees doesn't provide the logo of your shop, only you have this artwork. Which means, you can fix this only by re-uploading the logo yourself.

That wasn't the problem. My developer brother got it working for me.

He said this:

‘The core API changed it’s API likely this module isn’t specifying that it doesn’t want blah escaped And they required later that modules do that They’ll likely be able to tell you deeper info about what to change that you should relay to me and we can get it done we can also just look in the module and figure it out but I’m lazy and let’s be honest it’s really easy to just update global.css’'

Module is the theme configurator in warehouse theme.

Link to comment
Share on other sites

This is a feature I always whished for. Especially the downgrade part. I would like a checkbox so I would be able to uncheck a specific file I don't want to update. (all files checked by default) I don't know how database changes are handled though. This is something I always wanted for tb. To be stable and add features missing in 1.6

Keep up the great work.

Link to comment
Share on other sites

@nickon said in GitUpdater preview:

This is a feature I always whished for. Especially the downgrade part. I would like a checkbox so I would be able to uncheck a specific file I don't want to update. (all files checked by default) I don't know how database changes are handled though. This is something I always wanted for tb. To be stable and add features missing in 1.6

Keep up the great work.

This is an intersting idea. But I guess, it can also make troubles. Imagine a new function is added to a file, which you don't update. Once this new function is called, your shop will throw an error.

But of course I see your point. Especially for theme files, it could be helpful.

Link to comment
Share on other sites

@Traumflug how do you plan to tackle database migration? Say 1.0.9 will introduce new db column for table A, and foreign key to this column from table B.

  1. how will be these db changes created when migrated from 1.0.8 to 1.0.9?
  2. what happen when you downgrade from 1.0.9 to 1.0.8
  3. what about order of modification (for example we need to drop constraint before we drop column itself)?
Link to comment
Share on other sites

@wakabayashi said in GitUpdater preview:

@nickon said in GitUpdater preview:

This is a feature I always whished for. Especially the downgrade part. I would like a checkbox so I would be able to uncheck a specific file I don't want to update. (all files checked by default) I don't know how database changes are handled though. This is something I always wanted for tb. To be stable and add features missing in 1.6

Keep up the great work.

This is an intersting idea. But I guess, it can also make troubles. Imagine a new function is added to a file, which you don't update. Once this new function is called, your shop will throw an error.

But of course I see your point. Especially for theme files, it could be helpful.

When I read nickon's suggestion I thought "great" but than I got wakabayashi's point of causing eventually some troubles, but maybe there could be a function included that the manually changes files are kept with a prefix, e.g. M, into the same folder. So one can easily check single files if the work or what exactly is different and include the same changes again if needed. (Not everybody keeps a good record of changes done)

Link to comment
Share on other sites

I would like a checkbox so I would be able to uncheck a specific file I don’t want to update.

Such a feature was considered. And found to be a pretty sure way to make merchants shoot into their own foot. The way I recognize merchants, they won't compare their patched file with the new file and then merge their changes over. They'd just click "don't change", which is a recipe for disaster. One can't update just parts of a software and expect the result to work.

People editing their files manually also have to deal with this manually.

  • Like 1
Link to comment
Share on other sites

Thats why I suggested all the files would be checked by default. Youi could have a checkbox stating "Select files to skip while updating" with a big fat red warning that I can break my shop, and then let the user uncheck the files. Or based on the md5sum let me only skip files that are already manually edited. If I have edited a file manually and do an upgrade (even now) the chance to break the shop is high. There are ways to put safeguards to prevent the user of messing up. And then you have the user that can always mess things up:-)

But for me that is not the basic issue. The issue is the database. I guess downgrading eg from 1.0.8 to 1.0.7 could be doable but what if I want to downgrade to 1.0.1. ? That is the real danger not the (edited) files

Link to comment
Share on other sites

how do you plan to tackle database migration?

Classes should learn to maintain their database table them selfs. Like it was sketched here: https://github.com/thirtybees/thirtybees/commit/f7a55b7d772f58a0019f938bbfdf73528374087e PHP gets updated, then each class's installCheck() gets called, which detects and removes distinctions between the necessary table and the one currently in the database. Sounds complicated, but is probably pretty easy once class ObjectModel has learned to do this for the general case.

In case this approach turns out to be too unreliable to implement, the fallback strategy would be to run upgrade scripts like the current updater does. Update from 1.0.8 to 1.0.9 would run all 1.0.9 upgrade scripts. A downgrade in the opposite direction would run the same set of 1.0.9 scripts. Which means, these upgrade scripts would have to learn to work in both directions if necessary. AFAIK, we didn't have a case where a database downgrade is crucial for a software downgrade yet.

A nice thing about the update approach following the Git approach is, the updater has access to all files he feels a need for. Files of the old version as well as files for the new version, installed files and not installed ones.

Updates/downgrades are planned to run independently from core software. The module prepares the update by providing all the changed files somewhere, then calls a temporary PHP script (obfuscated name!) to move them into place atomically. And yes, in midst of this copy/move operation, core software has to be assumed to be broken. That's the same for any in-place updater.

Link to comment
Share on other sites

Is it possible to try to borrow something like SUN/Oracle's Solaris snapshots mechanism? They used it for upgrade process: during upgrading OS a snapshot for the new version created, so after upgrade complete one have a choise to boot into new OS, or into old one in case of bad upgrade. A snapshot itself contains only blocks that has been changed since it creation so it doesn't take much disk space.

Link to comment
Share on other sites

@DaoKakao Possible, I would think yes. Actually working is another story. Let me just preface this with this is solely my opinion. But Sun has a ton of control over the environments their software runs on. They have full time system administrators and teams of system administrators running their software against machines they have full control over. This is what would make it difficult and unreliable for us. We have hosts that throttle disk i/o, processing i/o, and file information. So it would be hard to make something that could reliably work.

Link to comment
Share on other sites

I mentioned sun/solaris just because had some experience with it and was happy of such easy and simple approach. Well, please, don't take it as a critics/stupid advice, et c. Anyway, all TB devteam deserves nothing but great respect for your efforts.

Link to comment
Share on other sites

Oh, I didn't. Its just building out an infrastructure like that and making it work across platforms is difficult, that is why no one else has done it. I would love to be able to include 99% of the ideas you all come up with, they are generally great ideas. We just have to be honest with our limitations.

  • Like 1
Link to comment
Share on other sites

Installed the module and did some testing - what an innovative, great idea! Congrats! Here's some feedback: I noticed that the module also checks the theme folders (obsolete files) - maybe you could remove the check for custom themes altogether? Maybe add a "select all" box in the GUI for obsolete files? I do understand that it compares the filelist to a vanilla install of TB - but no-one would run their shop as a plain vanilla anyway, so maybe add some options to exclude certain directories? Just an idea....

Link to comment
Share on other sites

I finally get some time to test this, and must say I really like it so far.

I've noticed that the module reports some files as obsolete, while they are not. Notably these

config/settings.inc.php --- generated by system itself, it's not part of git. Crucial file, must never be removed config/settings.old.php --- contains saved version of settinsgs.inc.php whenever it gets regenerated config/defines_custom.inc.php --- this file exists so merchants can safely upgrade to newer version of thirtybees and keep their settings (dev mode, etc) mails/... --- not all mail templates are part of git, this dir should be ignored override/... --- this should be definitely ignored translations/... --- this should be definitely ignored themes/... --- themes should not be touched

  • Like 1
Link to comment
Share on other sites

What would be the thought on having check boxes next to the files? Where you can choose to leave some files untouched?

Like @wakabayashi described above, leaving code files untouched easily leads to a broken shop. All an updater could do is to keep a backup of modified files, so merchants can review them not only before, but also after an update.

That said, thanks a lot for all your testing help. I'm currently reworking the filter list. Finding out which files to update and which ones not is pretty tricky. With the previous updater this was done by the update actions building script (on the thirty bees server), and this wasn't exactly bullet proof either.

So far the list is (PHP regular expressions, files coming with the distribution don't get removed by this filter): php const INSTALLATION_PATH_FILTER = [ '#^cache/#', '#^img/#', '#^upload/#', '#^download/#', '#^translations/#', '#^mails/#', '#^override/#', '#^themes/(?!community-theme-default/)#', '#^themes/community-theme-default/cache/#', '#^themes/community-theme-default/lang/#', '#^themes/community-theme-default/mails/#', '#^config/settings.inc.php$#', '#^config/settings.old.php$#', '#^config/defines_custom.inc.php$#', ];

Which means: - Ignore cached files, except those coming with the distribution. - Same for overrides, uploaded files, downloaded files, translations and mail templates. - Ignore all themes except the default one. - Ignore settings, of course.

An updater has no business with any of these. Translations/mail templates have their own update mechanism, just like modules.

Not yet done are files which shouldn't come with the release, because they get overwritten by setting up the shop. Favicons, for example. Some category images (they should get copied from the install/ folder during installation instead, but that's another issue).

Likely I can provide another preview later today.

Link to comment
Share on other sites

Here's this next preview:

gitupdater-2019-01-11.zip

  • Reviewed file filtering in detail. Including all the review report observations above, the stuff shown in the previous post, also another filter to keep favicons & friends unchanged.
  • Processing log and file lists collapse now, giving a fairly short panel with the Update button visible immediately. An attempt to make this two-click-updater fearless to use, slapping long lists into a merchant's face has certainly some repellent taste.
  • File lists can get expanded by a click, of course.
  • File lists now show the number of files affected.

Next step will be to implement actual updates.

Link to comment
Share on other sites

  • 2 weeks later...

Alright, time for another preview.

Advances since the last preview:

  • Updates now actually work! So far tested with a clean installation, one can switch between v1.0.7 and v1.0.8 back and forth as often as one desires.
  • Applies the same database upgrades as upgrading to v1.0.8 with the old updater did.

Which means, all the essential stuff is implemented.

Still missing:

  • Some nice exit. Currently, one is kind of stuck after an update and has to select something in the menu to move on (and see the new version in the upper left corner).
  • Actual removal of selected obsolete files. So far, these checkboxes are just decoration.
  • Machinery to update the underlying clean master installation automatically. To allow merchants to update their shop a few minutes after a Git commit was pushed to the Gitub repository.

To demonstrate the modules' capabilities, I've uploaded current branch 1.0.x manually. Happy updating!

gitupdater-2018-01-21.zip

  • Like 4
Link to comment
Share on other sites

Don't be surprised if it looks like nothing happens. With the comparison done, updates are quite speedy:

1989371039_Screenshot_2019-01-21UpdaterRepRapDIY.png.c4659869a84c0e7c622d5e541d2f5775.png

That's 0.2+0.6+0.1+0.1 = 1.0 seconds for the whole download and update process.

Shop downtime is even less, it's just the time between Created update script and Update script was executed. Apparently less than 0.1 seconds.

  • Like 5
Link to comment
Share on other sites

Seems to work rather nicely. There are a couple things we might want to tweak depending on community input 

 

1. It looks like the module is hard coded to use ssl. So we need to either add something to the htaccess to allow cross origin from non ssl to ssl to people, or just detect and use what is currently being used. On the first try it got blocked because of a cross origin request to the domain I was using, but I was logged into the back office with the non ssl url.

 

2. I think there should be an option to exclude the theme. 

 

3. I think we need to have something in place to set aside the stats modules. Like if someone goes from 1.0.3 to 1.0.7, the shop will just not work because the old stats modules are there. 

  • Like 1
Link to comment
Share on other sites

Works very nice and fast. Congrats on this release! 

I tested it a bit, and here are my notes:

1. my locally modified files are correctly listed in Files to get changed, but are not marked as M (this worked when I updated from 1.0.8 to 1.0.x, but when not when downgrading back to 1.0.8)

2. I could not find if my modified files are saved anywhere before they get overwritten by updater module

3. I strongly recommend to not touch community theme at all, as everyone makes changes to it. Unfortunately we don't have ps17 template inheritance functionality yet

 

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...