Usage of consistent webP Format

[wakabayashi]

I am working on my new theme and webP support is a big deal of it. Since we have now over 30k products data size is kind of an issue.

Right now, when you enable webp. The original image is saved as jpg. I did change this behaviour to webp. So when I upload I only have webp files. But this leads to issues in BO -> jpg is hardcoded there.

Why does actually the theme need to support webp? Is it not kind of the server, that needs to support it? Cause my idea was now: give the merchant a simple select, which formats he wants to use and then this is used for all uploaded images in FO and BO.

What do you think?

[wakabayashi]

22 minutes ago, e-com said:

This is how the scaling of thumbnails has always worked both in PS 1.6 and in TB.
If source image is smaller than the generated thumbnail, a background is added around it. For JPG white, for PNG and WebP transparent.
And this is certainly correct action. By resizing images up (enlarging them) we will have ugly blurry images.

Hm but the image is scaled down in browser afterwards. I agree that scaling up (blurry) and then scale them down afterwards is very stupid. But not doing it means, that the actual object of the image becomes smaller. IMO this is worse, than the other...

Here you see what I mean: https://thirtybees.genzo.ch/de/coffee-and-tea/Coffee/coffee-beans. On not so large displays, it's just having a lot of white ^^

Well ok, at least I know now, that I haven't broke it 😃

Edited February 16, 2023 by wakabayashi

[e-com]

CSS class img-responsive will always scale our image to size of container. But it will always only make image smaller. It also does not perform image enlargement.

[datakick]

9 hours ago, wakabayashi said:

And this is certainly correct action. By resizing images up (enlarging them) we will have ugly blurry images.

Well, for 2x images, I think we should not do that. 

If the source file is not big enough to generate 2x image, then don't do that in the first place. 

[wakabayashi]

4 hours ago, datakick said:

Well, for 2x images, I think we should not do that. 

If the source file is not big enough to generate 2x image, then don't do that in the first place. 

I am not totally sure, if I understand you right. My idea was something like

$widthOfRetinaImage = min(2*$imageType['width'], $widthOfSourceFile);
$widthOfRetinaImage = max($imageType['width'], $widthOfRetinaImage);

So that the width is always between $imageType['width'] and 2*$imageType['width'], but no blurry happens.

Was that your idea too?

Edited February 17, 2023 by wakabayashi

[datakick]

I meant this;

if ($widhtOfSourceFile >= 2*imageTypeWidth) {
   // generate retina format
} else {
  // do nothing. retina version for this particular image will not exists
}

Are there any benefits if the file between $imageType['width'] and 2*$imageType['width']? For example, if source file has width 1005, and imagetype width is set to 1000, should we create image-2x with image width 1005 instead of expected width 2000? Even if the image size is 1500, will there be any retina benefits?

[wakabayashi]

This was my first idea. I thought, that size 1500 would be a benefit, but yeah this is probably a detail. My main point was, to prevent the scaling down of the actual object on an image. Both solutions wil work for this purpose 🙂

Edited February 17, 2023 by wakabayashi

[wakabayashi]

Did anyone ever use scenes sucessfully? I mean years back... This seems to be horribly broken since years. But actually I would like this feature 😕

Anyone (@e-com) interested in bringing this back? Help needed... 🤕 I can not even find any saving process of the table tb_scene_products!?

Edit: I just made some progess on this 🤓

Edited April 2, 2023 by wakabayashi

[wakabayashi]

How do you like me now? 🤣

Ok, there are still a lot of issues left in scenes. But the video shows that this hidden feature is not completly death 🤭

[e-com]

11 hours ago, wakabayashi said:

Did anyone ever use scenes sucessfully? I mean years back... This seems to be horribly broken since years. But actually I would like this feature 😕

Anyone (@e-com) interested in bringing this back? Help needed... 🤕 I can not even find any saving process of the table tb_scene_products!?

Edit: I just made some progess on this 🤓

It seems to me that these photo formats: scene, m_scene and Scene class are archaisms still left over from PS 1.4 where there was an AdminScenes controller for mapping product photos.
In PS 1.5 and PS 1.6 there was no longer this controller.
And in PS 8 there is no longer this Scene class.

 

[e-com]

Sorry, my mistake. But AdminScenesController is and in PS 1.6 and in TB, only it is not installed in menu.

[datakick]

24 minutes ago, e-com said:

Sorry, my mistake. But AdminScenesController is and in PS 1.6 and in TB, only it is not installed in menu.

Yes, it was deprecated in 1.6 or in 1.5, I'm not sure.

The functionality was kept in the core, but enabled only for uses that had some scenes active, with the intention to remove it completely.

I agree with this, actually -- there is no reason why this should be part of the core. If it's useful for somebody, we could extract the functionality to some module.

But it should be removed from core.

[wakabayashi]

13 minutes ago, datakick said:

I agree with this, actually -- there is no reason why this should be part of the core. If it's useful for somebody, we could extract the functionality to some module.

But it should be removed from core.

Yes, it's no core functionality. I agree. But as a module it's very interesting IMO. On all social media plattforms it's becoming "normal" that you tag products on posts/images/videos.

Maybe I can do that one day, but won't promise anything as I have already a lot of "projects" open 😉

In general I am making good progress on the image rewrite. I am now testing things with Beesblogmodule. With my rewrite almost all custom image stuff can be dropped there. Instead you just define this in BeesBlogPost::$definition:

'images' => [
    'beesblogposts' => [
        'inputName' => 'post_image',
        'path' => _PS_IMG_DIR_.'beesblog/posts/',
        'imageTypes' => [
            ['name' => 'beesblog_post', 'width' => 800, 'height' => 500]
        ]
    ]
]

It's going to be a lot of fun (almost no work) as a dev to use images 😎 Merchants will have consistent webp support 😏

[wakabayashi]

I plan to finish this project soon. I have done so much now and need to stop, otherwise it will never get to an end. Here are some highlights:

• Consistent webp support in core (upload product images in webp is now possible)
• Modules can simply use core image system instead of implementing themself (beesblog was used as a test and simply supports now webp too). Just use AdminController and objectModels and image implementation takes minutes instead of days…
• Getting an image link was never so easy: all entity types are working with Link::getGenericImageLink(). This function also returns the source image, if you let $imageType empty.
• All image entities support now usage of $link_rewrite
• New we don't try to guess if 'home', 'home_default' and 'Niara_default' are all the same image type. We add all image type names to the db. First this may shock, how many different image types are used. But we introduce an alias system, that let you merge them. So a merchant has full control.
• Category image and category image thumbnail are now handled separately, so no more bugs on this. This is true for all other object model classes. They can add as many image inputs as they want.
• Image quality can possibly raise. I noticed some dubious places, were source file were possibly reduced in quality. So that meant, that image quality was actually lower than expected. I personally plan to use webp and 95 quality. I guess that raises quality and could even reduce size compared to old jpg system 😉
• Employee image is now showed instead of icon-user.
• Default images were fixed (this was a complete mess). Now you can simply upload it under localization -> language and you are fine.
• Generation of retina images was improved a bit (no more blurry image generation). But I am still not totally convinced about this feature. The implementation doesn't check anywhere, if a retina display requesting an image or just a normal display.

In general a huge amount of hardcoded stuff was removed and replaced by clean methods/constants. I would expect, that we can add new image extensions like "AVIF" quite quickly now. But I haven't tested this yet. Unfortunately plesk doesn't support AVIF yet and I don't want to install this manually on my server. 

So who is excited and ready to help testing this? To make clear, why this isn't a simple PR:

Edited April 6, 2023 by wakabayashi

[30knees]

16 hours ago, wakabayashi said:

 

So who is excited and ready to help testing this? To make clear, why this isn't a simple PR:

If guidance is available how best to test so it's useful for you I'll help! 

Edited April 7, 2023 by 30knees

[wakabayashi]

On 4/7/2023 at 9:11 AM, 30knees said:

If guidance is available how best to test so it's useful for you I'll help! 

Thanks a lot. I believe best would be if @datakick can make a dev branch for it. So that everyone with a teststore can use core updater and play around. Unfortunately I haven't reached him last weeks.

If have used last days to test things on my own teststore. Things are looking good. I only found some details to fix. I could reduce size of img folder quite a lot (more than 30%). 

[movieseals]

Available to test too.  I have developped my own workaround for webp but something native to the software would be a huge step forward.

 

[wakabayashi]

I have been in touch with @datakick the last days. We discussed a lot of things about this change. He improved the core updater and established a new dev branch there. He also pointed out a lot of issues. I am first going to fix/improve these points.

When I am done, you will then be able to use core updater. 😎

[wakabayashi]

As some may have seen in another thread. This project is somehow stuck. As I invested weeks of work and it's fundamental functionality for the future of my shop, I won't drop it. But if it doesn't make it to the core, I have to develope on a highly customized version of tb (70 files are changed). Even with git rebase this is kind of a pain and since I am going to use it in all my modules. All my work, won't be compatible with core anymore. It will mean the end of genzo_krona and my contribution for core stuff.

The only solution to this would be, if the core is changed in some way:

1. My alpha version is further integrated as intially planned in collaboration of community, datakick & me. 
2. Tb implements the same/similar functionality themselfes

Both seem very unrealistic to me now. The first one is basically where we are, but it doesn't go forward at all. The second one: why would you do that, if I have already done 80% of the job?

Yesterday I had suddenly a third idea. Maybe it could be implemented, if I (& maybe others) pay for the hours, that Datakick need to invest to make it more robust. But nor do I know if it's realistic, nor do I know if it's in my (companies) interest. Probably depending on the amount. 🤑 Maybe that's more about @Smile than @datakick!? Cause I guess as smile is paying, he decides on what is worked on (kind of understandable). And probably webp support is not on high priority for smile!? 

Otherwise it is, what it is 🙈 Live goes on a for all sides 🤗

[30knees]

Thanks for the update, @wakabayashi.

Maybe it's time for a general update on the status and future of the software, @Smile?

[datakick]

I apologise for the late reply, I was on a vacation last two weeks and I'm slowly burning through my unread mails and threads now 🙂

Regarding the @wakabayashi image rewrite -- this is scheduled to go to bleeding edge after the release of 1.5 (which will be very soon). We talked about pros and cons and risks about putting this into 1.5, and decided it's too risky to do so.

While this is very nice functionality, it also brings some backwards compatibility problems and issues.

The main (but not only) issue is that friendly urls of product images changed. We need a backwards compatibility solution for this issue, one that would work on both nginx and apache. We have to be sure that if store update to new version their images will still be displayed. That's not the case with current code, if deployed on nginx server. I've tested it on my demo server and it broke very fast.

I understand that it can be very disheartening that it takes so long for us to merge this. I know @wakabayashi put a lot of effort into that. But we can't just merge is as it is and hope for best. It probably works nicely for his store, but we need to be sure it will work for everyone else as well. We need to be sure there are no visible bugs. We have to do core review to check for security and other issues. And this all takes a lot time -- if you look at this change, you will find that it touches 72 files, a lot of addition and deletion.

I've already spend around two days of my time on code reviews, on fixing various small issues. There are still some issues that we need to work on before we merge it.

But it will be merged. Just not into 1.5

 

 

[wakabayashi]

@datakick I am working again a bit on this as my shop relaunch comes closer. 

Today I checked, how rewrite rules should look like and generated them quickly. This works fine for me:

rewrite ^/categories/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/c/$1-$2.webp break;
rewrite ^/categoriesthumb/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/c/thumb/$1-$2.webp break;
rewrite ^/genzoportallink/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/genzo_portal/links/$1-$2.webp break;
rewrite ^/manufacturers/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/m/$1-$2.webp break;
rewrite ^/scenes/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/scenes/$1-$2.webp break;
rewrite ^/scenesthumb/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/scenes/thumbs/$1-$2.webp break;
rewrite ^/stores/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/st/$1-$2.webp break;
rewrite ^/suppliers/([0-9]+)-([^\s/]+)/([^\s/]+)\.webp$ /img/su/$1-$2.webp break;
rewrite ^/products/([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$1$2$3.webp break;
rewrite ^/products/([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$1$2$3$4.webp break;
rewrite ^/products/([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$3/$1$2$3$4$5.webp break;
rewrite ^/products/([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$3/$4/$1$2$3$4$5$6.webp break;
rewrite ^/products/([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$3/$4/$5/$1$2$3$4$5$6$7.webp break;
rewrite ^/products/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$3/$4/$5/$6/$1$2$3$4$5$6$7$8.webp break;
rewrite ^/products/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$3/$4/$5/$6/$7/$1$2$3$4$5$6$7$8$9.webp break;
rewrite ^/products/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.webp$ /img/p/$1/$2/$3/$4/$5/$6/$7/$8/$1$2$3$4$5$6$7$8$9$10.webp break;

You see genzoportallink is just a module from myself, that uses the ObjectModel image definition.

Still some stuff left, that need to be fixed:

• The multistore thing with different themes and with different sizes for same imageType
• The gathering and include of objectModelFiles that use $definition['images'].

Probably other stuff as well, but on my shops things look quite good 😳 😏

[wakabayashi]

I went live my branch of consistent webp images. In general it works well. But I just found another big issue, but I want someone to confirm. I need someone, that uses webp and google search console. Please open an url in google search console and check the html of the crawled page. Is there webp or jpg used?

@datakick Do you think we need to keep this browserSupport? https://github.com/thirtybees/thirtybees/blob/e6ca6b43d5235c80c1ebeff087ece88fd4541c83/classes/ImageManager.php#L881

In alot of usages it causes troubles. You have to adjust all ajax calls for webp support and it seems that crawlers might fall to it.

[datakick]

1 minute ago, wakabayashi said:

I went live my branch of consistent webp images. In general it works well. But I just found another big issue, but I want someone to confirm. I need someone, that uses webp and google search console. Please open an url in google search console and check the html of the crawled page. Is there webp or jpg used?

@datakick Do you think we need to keep this browserSupport? https://github.com/thirtybees/thirtybees/blob/e6ca6b43d5235c80c1ebeff087ece88fd4541c83/classes/ImageManager.php#L881

In alot of usages it causes troubles. You have to adjust all ajax calls for webp support and it seems that crawlers might fall to it.

We can safely drop this, as browser support for webp is very good now: https://caniuse.com/?search=webp 

 

[netamismb]

in Search console I found: <img class="img-responsive center-block"
                     itemprop="image"
                     src="https://www.rock4u.eu/1134-Niara_large/camiseta-mujer-iron-maiden-logotipo.jpg"

outside of search console <img class="img-responsive center-block" itemprop="image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=" srcset="https://www.rock4u.eu/1134-Niara_large/camiseta-mujer-iron-maiden-logotipo.webp" title="Camiseta mujer IRON MAIDEN - Logotipo" alt="Camiseta mujer IRON MAIDEN - Logotipo" width="458" height="458">

But this page is indexed in search console.

[wakabayashi]

4 minutes ago, netamismb said:

in Search console I found: <img class="img-responsive center-block"
                     itemprop="image"
                     src="https://www.rock4u.eu/1134-Niara_large/camiseta-mujer-iron-maiden-logotipo.jpg"

outside of search console <img class="img-responsive center-block" itemprop="image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII=" srcset="https://www.rock4u.eu/1134-Niara_large/camiseta-mujer-iron-maiden-logotipo.webp" title="Camiseta mujer IRON MAIDEN - Logotipo" alt="Camiseta mujer IRON MAIDEN - Logotipo" width="458" height="458">

But this page is indexed in search console.

This is a little confusing to me. Once there is jpg and once webp 🥵 But ok, as we can drop it anyway, we won't have this issue in future 👍