Jump to content
thirty bees forum

braffas

Members
  • Posts

    30
  • Joined

  • Last visited

  • Days Won

    1

Everything posted by braffas

  1. Yes, an update to 1.5.1 has just been moved to the top of my to-do list. I am already in the process of testing it. 🙂
  2. It was a significant security breach. Thank you for directing my attention to it. The breach was initiated through the vulnerability detailed here: https://build.prestashop-project.org/news/2022/major-security-vulnerability-on-prestashop-websites/ It turns out that our old blog module, Prestablog, contained the vulnerability. The attack used a payload with an injection (hex value) sent to a text processing function (getByRewrite) that generates SEO URLs for blog categories. Fortunately, the script triggered multiple warnings so the POST data was captured by the collectlogs module, which provided the critical lead I needed. All database queries in the module have now been escaped and thoroughly tested for SQL injection vulnerabilities. Everything is secure now. Phew!
  3. Thank you so much! I'm looking into it further. The possibility of a security breach hadn't occurred to me. I am examining the log files and related data immediately. I will get back to you as soon as I have determined the cause.
  4. Hello! I've encountered a bizarre issue that's driving me nuts. For some unknown reason, the smarty cache (PS_SMARTY_CACHE in configuration) becomes active at random times throughout the day. I've searched the source files, both in core and modules, to locate where this might be happening (PS_SMARTY_CACHE and SMARTY_CACHE) and I simply can't find it. I searched for Configuration::updateValues/set etc. changes and direct changes to the configuration table in the database. Furthermore, I'm the only one with rights to access the AdminPerformanceController, so it's not a manual change. Right now, I have set up a CRON job to check the value and email me when the value changes. The last time it happened was Saturday evening at 21:05 This time doesn't match any CRON job that I have running, and no one is active in the store at this time (other than customers). The problem is our server suffers a slow death when the file cache starts to build up large due to our catalog of over 4 million products. The issue started happening a couple of months ago. I'm running TB 1.4 (this issue did not start after an update). For now, I've set up the CRON job to disables the cache whenever it becomes active … That works … but would REALLY appreciate it if anyone has any input about why/where this is happening?! 😄 Best regards Kasper
  5. I know about the different DB methods I use execute, executeS, getValue etc. (and I fixed those places where I had used the wrong method by mistake, nice addition to the core btw). The ErrorHandling sadly does not seem to be the problem. I added the limit from your fix, but the slowness persists. It starts after around 10k cycles on the heavy combined updates/inserts (products, features, manufactures, suppliers etc.). It may be some override or something. I will let you know when I find the cause. Right now, I'm going to run the script on small portions of the data every minute or so.
  6. Ah. That makes sense. It definitely involves a lot of cycles. I go over every line of the datafeeds, analyse the data and compare it with existing data before updating / inserting / skipping. The import script was all build with speed and low memory usage in mind, and it was really fast, and had a relatively low memory usage. Db::executeS() seems unaffected. Generation of our own data feeds seems unaffected. I did install the collectlogs module but it did not seem to make a huge difference. I will uninstall it again and check. Which array in core should i be looking for (error collection)? Best regards Kasper
  7. Hello What changes in 1.4 could cause high load and memory usage after updating from 1.3 -> 1.4. I run a lot of custom import scripts with 1m - 5m products. I mostly use the Db::execute() method for my updates and inserts. The general store speed seems about the same, but these CLI scripts are VERY slow now, and takes a lot of resources. I have made no changes to the import scripts or server config. Php is running 90 - 100% and memory goes up to 30GB+ (if I let it) There are no errors in the logs ... Best regards Kasper
  8. I noticed the same problem some time ago. For me it only happens with percentage quantity discounts from catalog price rules. For some reason the solution is to set the 'tax include' when creating the percentage price rule. (there was a log error hinting something about tax).
  9. In modules\ganalytics\views\templates\hook\analyticsjs.tpl function uuidv4() { return ([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g, c => (c ^ crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4).toString(16) ) } This function throws a syntax error in IE 11 which in my case blocked customers from checking out on IE 11 (and probably lower versions also). TBH that function looks scary as hell, i have no idea how to fix it.
  10. Hello It would be nice if it was possible to exclude some URL patterns from the full page cache. I have had a few modules that refused to work even with dynamic hooks, so i had to disable them in the controller->run method Like this: if (stripos($_SERVER['REQUEST_URI'], 'gift-cards/gift-card') !== false) { unset($cacheableControllers[array_search('product', $cacheableControllers)]); } The comment on the method says "this method should not be overridden!" so right now I'm stuck with editing the core.
  11. Categories are not removed from the cache (Redis etc) when invalidated, because the cache keys are deleted before they are collected. See fix below. /** * Invalidate an entity from the cache * * @param string $entityType * @param int|null $idEntity * * @since 1.0.0 */ public static function invalidateEntity($entityType, $idEntity = null) { $keysToInvalidate = []; if ($entityType === 'product') { // Refresh the homepage $keysToInvalidate = array_merge( $keysToInvalidate, static::getKeysToInvalidate('index') ); Db::getInstance()->delete( 'page_cache', '`entity_type` = \'index\'' ); if ($idEntity) { // Invalidate product's categories only $product = new Product((int) $idEntity); if ($product) { $categories = $product->getCategories(); foreach ($categories as $idCategory) { //Edit by Kasper: Moved to the top in the loop to collect the keys before they are deleted. $keysToInvalidate = array_merge( $keysToInvalidate, static::getKeysToInvalidate('category', $idCategory) ); //Edit by Kasper: Moved to the bottom of the loop so we can delete the keys after they are collected. Db::getInstance()->delete( 'page_cache', '`entity_type` = \'category\' AND `id_entity` = '.(int) $idCategory ); } } } else { // Invalidate all parent categories Db::getInstance()->delete( 'page_cache', '`entity_type` = \'category\'' ); $keysToInvalidate = array_merge( $keysToInvalidate, static::getKeysToInvalidate('category') ); } } $keysToInvalidate = array_merge( $keysToInvalidate, static::getKeysToInvalidate($entityType, $idEntity) ); Db::getInstance()->delete( 'page_cache', '`entity_type` = \''.pSQL($entityType).'\''.($idEntity ? ' AND `id_entity` = '.(int) $idEntity : '') ); $cache = Cache::getInstance(); foreach ($keysToInvalidate as $item) { $cache->delete($item); } }
  12. Everything seems to work as expected when the blockuserinfo->displayNav is marked as dynamic. So it's not really a bug, but it a very important thing to change before you activate fullpage cache. Can anyone give a short explanation about how the dynamic hooks are handled, or is it documented somewhere?
  13. @wakabayashi Yes, that seems to work. But not caching a stock module like blockuserinfo, should probably be a default feature. Also when i look at the cached data inside redis, the page with username is still cached, that freaked me out a bit. When i get back to work, i have to check how exactly the dynamic module hooks are handled before i enable full page caching again. I don't want customer names in google search results etc :D
  14. Hello. I just noticed that a users name was cached in the nav section. That's pretty bad. I tested it by logging in and refreshing the cache, and yep my name was cached in the nav section when i was logged out (also tested in other browser). ~~It only happens on the index page.~~ it happens on all cached pages I will investigate further. But any idea how this can happen? It seems to help when i enable the 'user info block' in the cache settings.
  15. renderForm returns 'true' on the last line. This will display '1' in the admin template (/index.php?controller=AdminOrders&addorder).
  16. Nope, Braffas is also my GitHub name I will check GitHub first next time. Great that it's getting fixed :)
  17. I don't know if this is the right place to post this, but here goes. In the method below Customer->customerExists public static function customerExists($email, $returnId = false, $ignoreGuest = true) { if (!Validate::isEmail($email)) { if (defined('_PS_MODE_DEV_') && _PS_MODE_DEV_) { die(Tools::displayError('Invalid email')); } return false; } $sql = new DbQuery(); $sql->select('`id_customer`'); $sql->from(bqSQL(static::$definition['table'])); $sql->where('`email` = \''.pSQL($email).'\' '.Shop::addSqlRestriction(Shop::SHARE_CUSTOMER)); $sql->where('`is_guest` = 0'); $result = Db::getInstance(_PS_USE_SQL_SLAVE_)->getValue($sql); return ($returnId ? (int) $result : (bool) $result); } The $ignoreGuest param is not used by the DB query anymore. This may give some annoying problems (no errors) with modules that use the method.
  18. braffas

    Thank you!

    @mdekker It's mainly the database lookup (this call getManufacturers) i fixed it by adding cache to the lookup in these 2 methods (below) I delete the all cache every morning after our automatic updates. But one could delete the rewrites from cache on the manufacturer / supplier update/add hooks Another solution (probably better for compatibility) would adding the rewrite directly to the the manufacturer/supplier tables. (maybe in a future 30bz update? :)) Original versions (in Dispatcher.php) protected function supplierID($rewrite) { // Rewrite cannot be empty if (empty($rewrite)) { return 0; } $context = Context::getContext(); $suppliers = Supplier::getSuppliers(false, $context->language->id, true); foreach ($suppliers as $supplier) { if (Tools::link_rewrite($supplier['name']) === $rewrite) { return (int) $supplier['id_supplier']; } } return 0; } protected function manufacturerID($rewrite) { // Rewrite cannot be empty if (empty($rewrite)) { return 0; } $context = Context::getContext(); $manufacturers = Manufacturer::getManufacturers(false, $context->language->id, true); foreach ($manufacturers as $manufacturer) { if (Tools::link_rewrite($manufacturer['name']) === $rewrite) { return (int) $manufacturer['id_manufacturer']; } } return 0; } My overrides with cache. Speed increase ~3 sec to ~0.90 protected function manufacturerID($rewrite) { if (empty($rewrite)) { return 0; } $context = Context::getContext(); if (!$manufacturer_rewrites = Cache::getInstance()->get('manufacturer_rewrites')) { $manufacturers = Manufacturer::getManufacturers(false, $context->language->id, true); foreach ($manufacturers as $manufacturer) { $manufacturer_rewrites[Tools::link_rewrite($manufacturer['name'])] = $manufacturer['id_manufacturer']; } Cache::getInstance()->set("manufacturer_rewrites", $manufacturer_rewrites, 86400); } $id_manufacturer = (int)$manufacturer_rewrites[$rewrite]; return $id_manufacturer; } protected function supplierID($rewrite) { if (empty($rewrite)) { return 0; } $context = Context::getContext(); if (!$supplier_rewrites = Cache::getInstance()->get('supplier_rewrites')) { $suppliers = Supplier::getSuppliers(false, $context->language->id, true); foreach ($suppliers as $supplier) { $supplier_rewrites[Tools::link_rewrite($supplier['name'])] = $supplier['id_supplier']; } Cache::getInstance()->set("supplier_rewrites", $supplier_rewrites, 86400); } $id_supplier = (int)$supplier_rewrites[$rewrite]; return $id_supplier; }
  19. braffas

    Thank you!

    Production server is a dedicated box with 64gb / E5-1650 Running: Debian 8 / Apache 2.4 / PHP 5.6 (FPM) / Mariadb 10.1 / Redis 3.2.9 The shop also runs fine with the same setup on a 2gb Vagrant box (what i use for development). I had to do overrides to some core classes and tweak some modules to make it work with 200k products. Mostly limits to ajax calls / pagination (back office) One thing i had to change after migrating to 30bz was the way that supplier and manufacturer rewrites are handled. That slowed down the site quite a bit (on those pages). i will post the change i made when i get back to work, since it may be useful for someone else with 40k+ manufactures / suppliers.
  20. braffas

    Thank you!

    Yep. That's the one :)
  21. braffas

    Thank you!

    I just wanted to say thank you to the 30bz team. I was thrilled when i first learned about the project 2 weeks ago, and now our shop has finally migrated to 30bz :) The migration gave me no problems with the modules i wrote myself, and only minor problems with some of the modules i purchased. We ship about 250 orders each days, and we got around 200k products in the database, and i must say, 30bz runs like a dream! So thank you so much for turning this project in the right direction. Keep up the good work!
  22. Are you 100% sure that there is no overrides? I previously used Page Cache from Jpresta (the class is called PageCache) it's included in the dispatcher override and it's still included when you chose to disable all overrides in the TB config AND it will also remain included when you uninstall the module. The module seems to have a hard time cleaning up. So check your overrides maybe there is something left behind. Just search your override folder for "/pagecache.php'"
×
×
  • Create New...