Safe maintenance strategy, shop cloning


  • Global Moderator

    By profession I’m more a server admin and embedded developer, so please bear with my ignorance. Nevertheless I currently have a PS 1.7 shop almost ready for production.

    What I miss in pretty much any shop software is a safe way to handle maintenance. A way to deal with upgrades, patches, code improvements, without interrupting services to the customer. It looks much like every merchant just tweaks the running shop and hopes the best s/he makes no mistake. No strategy other than quickly reverting wrongdoings or, if things go really wrong, take the shop down to recover from a backup. 30B’s stated goal is even to allow code editing from backoffice, surgery on the open, beating heart!

    As server admin I’ve learned that one shouldn’t do such things unless one doesn’t care about customer experience or enjoys to live on the dangerous side. The safe way is to do maintenance offline. On the public web server in a separate installation or on a local installation on the developer’s machine. Then to test, test, test. Only after tests were satisfying, development code is synchronized with the public server. This is safe and comes with a minimum of downtime, which means a minimum of customer disruption.

    However, and this is the reason of this post, this requires a way to clone or synchronize a shop. Rsync shop-here to shop-there with the confidence that the copy works the same way as the origin.

    AFAIK, PrestaShop doesn’t allow such a strategy. Server name is stored in the database, so one has to tweak that manually after an upload. Even installation location on the server is stored somewhere, one can’t move a shop from one folder to another without breaking it. Wether a module is present or not isn’t determined by looking at files, but by looking at some DB entry, making things inconsistent when one just removes or adds module files.

    Are there ThirtyBees plans to improve this situation? For PS I made a couple of patches which resolve most issues for monoshops (multishops are more complex). Are such improvements welcome? What do others think?


  • administrators

    Could you show us the patches required to improve it?


  • Global Moderator

    Of course. Note that these patches are done for PS 1.7. They should be sufficient for discussing the strategy, I’ll happily port them to TB, then.

    Patch 1 is about the shop domain. PS uses the PS_SHOP_DOMAIN configuration variable for this. This variable is used in dozens of places, so I decided to override the value read from the database, instead. Nice side effect: previous behavior is kept, new behavior is applied if that field is set to ‘automatic’ in back office, only.

    0_1488899287833_0001-CO-Allow-automatic-PS_SHOP_DOMAIN.patch

    Patch 2 does the same for installation location.

    0_1488899624077_0001-CO-Allow-automatic-for-Base-URI-too.patch

    Patch 3 also tweaks a configuration variable, this time PS_SHOP_ENABLE. Instead of storing that value in the database, it’s stored by presence of a file. This way one can easily set a store into maintenance mode by uploading a file by FTP (name matters), then do the synchronization, then remove the file. Scripts are very good at doing such sequences : - )

    0_1488899940454_0001-CO-store-PS_SHOP_ENABLED-not-in-the-database-but-by-.patch

    While patch 2 appears to work so far I could think of a better strategy in this case. Code location has no interference with multishop capabilities, so the IMHO better way would be to get rid of that database entry entirely in favor of always automatic detection. Not done this way so far, because I have to keep code changes small for not running into conflicts on the next PS upgrade (or migration to TB).

    P.S.: glad to see one can upload patches in this forum!


  • administrators

    Thanks a lot for sharing the code. I would love to implement it as soon as possible. Currently trying to figure out if this can be packaged with the next PATCH version or MINOR version. It seems to touch private code only (stuff we don’t want to expose to modules and themes), so it might land in 1.0.1

    I expect to be able to give you a more detailed report next week. The result will likely be something hybrid, I think, which gives priority to the files on the machine and if available prevents the database from being read for multistore info.

    You can improve the performance a little bit by using php files with just one big array in it. This way the machine can cache the file (OPCache) and decrease I/O.


  • Global Moderator

    Glad to see you like it.

    TBH, the more I think about the issue, the more I think the right way would be to get rid of all this code in classes/shop/Shop.php and classes/shop/ShopURL.php which tries to figure the “right” URL and also tries to redirect to that.

    A monoshop shouldn’t care about the domain it’s hosted on. If a merchant wants to redirect something, e.g. http:// -> https:// or typo-domain -> real domain, .htaccess is the right place to do so.

    A multishop should just read the domain, not try to “fix” it. If a domain happens to be not one of the multishop domains, that’s a configuration error, nothing which should be validated at runtime on every single page request. Removing all that code would also remove a number of database queries, as far as I can see.

    I think I’ll play a bit with this idea.


  • Global Moderator

    Let me put a status update here.

    I managed to get a development shop running from a clone of the Git repo. There I applied this ‘automatic’ branch and added a couple of additional things:

    • An attempt to not read shop domains from the database in classes/shop/Shop.php at all. Which means to remove all this redirection code there, speeding up page loads.

    • Doing redirection via .htaccess. Writing .htaccess in backoffice adds a few lines to do this redirection.

    All three parts work nicely for monoshops. When trying to review the first commit for multishops I enabled multishops. After doing so, frontoffice was broken (infinite redirects), so this first commit apparently needs rework.

    Then I tried to add a second shop. This messed up the database somehow. Whatever I try to access, backoffice or frontoffice, results in “Internal Server Error”. Disabling multishop via phpMyAdmin didn’t improve this. Next step here is to drop that shop for starting over.

    Code is here: https://github.com/Traumflug/ThirtyBees/commits/ignoredomain

    P.S.: Generally, all this redirection stuff is very complex. I found at least 8 code locations dealing with it. Looks like there are at least four levels: domain “fixing” in Shop.php, something similar in FrontController.php, some widespread mechanism to redirect depending on content and an approach distinct from all this for switching between http and https. This led me to the conclusion that not storing the domain somehow would be a way to intrusive change. New plan after fixing the commits described above is to just move the configuration table into an ordinary file, making it accessible for upload scripts.



  • Very nice work y’all, this will be nice to have.


  • administrators

    That is certainly some nice work. I’m currently on vacation but once I’m back I’ll have a look at it. I, too, have a feeling this can be simplified a lot. Current system is way too complex and you can just see that by how it’s used by both PS employees and module devs.


  • Global Moderator

    Next update. Progress is slow, but I learn a lot along the way :-)

    Today I tried to put a shim between ShopUrlCore and ObjectModel. The idea is to override all database accesses with accesses to the array in question. This shim is class ObjectFileModel. Code is on branch ‘objectfilemodel’: https://github.com/Traumflug/ThirtyBees/tree/objectfilemodel

    This mostly works, but one way only. The array gets updated after e.g. adding a new URL to a shop. But the page load following that falls back to reading the DB. Because page rendering uses AdminController, which accesses the database directly, ignoring ObjectModel.

    Looks like I have to try to put a shim between AdminShopUrlController and AdminController tomorrow, too. Another idea is to connect AdminShopUrlController to ShopUrlCore directly, but then I’d loose all the validation, field forming and relational (multishop, multilanguage) code.


  • Global Moderator

    Saturday night update.

    The shim between ShopUrlCore and ObjectModel is now well established as ‘ObjectFileModel’. Things start to work, I think editing multishop URLs (backoffice -> Advanced parameters -> Multishop) is complete already.

    This shim between AdminShopUrlController and AdminController was tried, but dropped. Instead I taught ObjectFileModel to not only write, but also read entries. Typical for the Model-View-Controller concept, but PrestaShop apparently prefers to read from the DB directly (somewhat understandable, one can do some data shaping, like sorting, inside DB queries).

    Code is on branch ‘objectfilemodel’: https://github.com/Traumflug/ThirtyBees/tree/objectfilemodel , rebased to latest ‘master’. Branch length currently 12 commits.

    Things get clearer, commits smaller, which is usually a good sign. Currently hunting down all usages of the ‘shop_url’ table and replacing that with the file based storage. :-)


  • Global Moderator

    Saturday night update.

    Reviewed each commit again and I think the existing ones are usable now. As I implemented synchronization between file based and database based storage (will go away later), every single commit keeps a fully working front- and backoffice.

    Still working on eliminating shop_url database table usage, step by step. Code is freshly pushed on https://github.com/Traumflug/ThirtyBees/tree/objectfilemodel , 19 commits by now. Guess is, another 20 are required to complete this task.


  • administrators

    That’ s some good progress.

    Btw, when I looked into it, I found this library: https://phinx.org/
    Do you think we could use it for this purpose as well?


  • Global Moderator

    Phinx looks like a good option if the task were to get rid of the database entirely. For handling just one ‘table’ it’s overfeatured. Also, the “engine” is already completed. It’s just an ordinary PHP array, and writing that is as simple as

    file_put_contents($this->def['path'],
                "<?php\n\n".
                'global $shopUrlConfig;'."\n\n".
                '$shopUrlConfig = '.var_export($shopUrlConfig, true).';'."\n");
    

    Reading it is as simple as … well, it’s already there, in a global variable. For better abstraction there’s ShopUrl::get()

    Most of the work is to replace queries also accessing other tables or doing some data shaping. Queries with WHERE, JOIN LEFT, CONCAT or ORDER inside (pretty much any query). This has to be split into a DB access for only the other table(s), then mixing and reshaping both results to get what’s required. Phinx won’t allow to query both, file based and DB based storage, with a single query, as far as I can see, so no advantage for this part.


  • Global Moderator

    Good news: I removed the shop_url table here and the shop continues to work :-)

    I hope I catched everything. Stuff like this:

            foreach (Shop::getAssoTables() as $tableName => $row) {
                // ...
                    $res = Db::getInstance()->getRow('SELECT * FROM `'._DB_PREFIX_.$tableName.'` WHERE `'.$id.'` = '.(int) $oldId);
    

    is hard to catch, because not grep-able. For now I simply hope this is never called with $tableName = 'shop_url'.

    Next step is to adjust the installer. Then to get rid of PS_SHOP_DOMAIN, PS_SHOP_DOMAIN_SSL, etc.


  • Global Moderator

    The part replacing the shop_url DB table by a file is done, see https://github.com/thirtybees/ThirtyBees/pull/162

    To achieve fully scriptable shop synchronization, configuration variables PS_SHOP_DOMAIN, PS_SHOP_DOMAIN_SSL and a few others have to go away, too. That’s a separate task. Code for backwards compatibility will be provided, of course, so a Configuration::get('PS_SHOP_DOMAIN'); will still succeed, even if there’s no longer such a database entry.



  • @Traumflug I follow with interest your topic although I rather read Chinese :) I don’t know why but I feel excited about your achievement :) Congrats!



  • @Traumflug i’m wondering, is not having hardcoded “shop_url” really helps you a lot? because for me it’s bs, can you give me real life example of your workflow, for the store upgrade? what is easier before and after your changes?

    i’m trying to understand your point, but for me hardcoded shop_url, is one of the smallest problems, in terms of PrestaShop sync between dev and production version


  • Global Moderator

    “bs”, like “beautiful surprise”?

    A real life example?

    cd /var/www/html/
    cp -rp my-shop-test my-shop-live
    

    After that, ‘my-shop-live’ won’t work without digging into backoffice and fixing things there. Which in turn disables ‘my-shop-test’.

    Another example:

    The latter won’t work either, it’ll redirect to localhost.

    Even if it’s a small problem, it should be fixed. However, I’m not aware of any other problems preventing shop synchronization.

    And to some extent I wonder why you lament here and on Github instead of showing us on how to synchronize a TB/PS-shop. If the strategy described in the opening post here is achievable so easily, post the solution!



  • I don’t claim to understand what he actually doing but the way I get it is: I could make a copy of my live site on my local machine and vice versa at any time. I believe the hardcoded shop URL is barrier for ignorant merchants like me. You could search prestashop forum and you will find countless topics concerning that. But I am not a developer and that’s the way I understand it!



  • @MockoB said in Safe maintenance strategy, shop cloning:

    I don’t claim to understand what he actually doing but the way I get it is: I could make a copy of my live site on my local machine and vice versa at any time. I believe the hardcoded shop URL is barrier for ignorant merchants like me. You could search prestashop forum and you will find countless topics concerning that. But I am not a developer and that’s the way I understand it!

    yeah, i think that is some example which i’m willing to understand, but the word “synchronize” is completely out of topic here, because “shop_url” is the easiest thing to do, while working on developer, and production store, at the same time, really

    @Traumflug said in Safe maintenance strategy, shop cloning:

    “bs”, like “beautiful surprise”?

    A real life example?

    cd /var/www/html/
    cp -rp my-shop-test my-shop-live
    

    After that, ‘my-shop-live’ won’t work without digging into backoffice and fixing things there. Which in turn disables ‘my-shop-test’.

    Another example:

    The latter won’t work either, it’ll redirect to localhost.

    Even if it’s a small problem, it should be fixed. However, I’m not aware of any other problems preventing shop synchronization.

    And to some extent I wonder why you lament here and on Github instead of showing us on how to synchronize a TB/PS-shop. If the strategy described in the opening post here is achievable so easily, post the solution!

    Well, lament? Maybe because it’s wasting time and energy, both you, and Michael’s who needs to check this feature. Ok, you can copy production site to localhost, and what later? Hm? What about differences between two databases? Or, you want to work with remote production database, on localhost? I guess not.

    I wish TB all the best, and when i see wasting time on such a feature… it’s not looking good :)


Log in to reply
 

Looks like your connection to thirty bees forum was lost, please wait while we try to reconnect.