Blog posts can not contain math symbols

[Dolfijn]

Hi,

I'm using the TB blog module and I happen to have a store with rather specialized (Radio frequency) hardware products. So, my blog posts are technical too. And sometimes there is some amount of math involved (more often then not). But symbols like Γ (gamma), ϕ (phie) and  π (pie) are all replaced as a ? (question mark ).

The result is that all post are rendered useless and are very annoying for the reader to read.

How can I solve this?

Regards,
Peter

[Dolfijn]

I solved it!

Problem was indeed the individual bees tables that had the latin1 char set. I solved it by exporting these tables to a sql file. There I changed the encoding to utf8 and then dropped, create, insert the new tables. And now I can use the special symbols too.

Thanks for all suggestions, especially those from musicmaster!

By the way, if I had payed attention to the TB Server Requirements, I would have configured MySql with full UTF-8 support. And this problem would never have occured in the first place 🙂

Edited June 1, 2019 by Dolfijn

[datakick]

I've just tested this, and it works correctly:

This is probably related to your theme / css / fonts you are using

[Dolfijn]

Ok, thank you. Gives me a direction. I'm using the default theme and also the default font. May I asked how you selected / created the symbols in the test post. I copied them from Wikipedia. Maybe that introduced the problem?

[datakick]

I just copied and pasted text you entered here

[Dolfijn]

I tried the same (copying the symbols from the forum into a new emtpy post). And what I get is:

"But symbols like ? (gamma), ? (phie) and  ? (pie) are all replaced as a ? (question mark )."

It happens immediately after saving the post. So, when I re-edit the post I just saved, the symbols are already replaced by question marks (so i'm still in the back office). Can that still be related to the theme / css and / or font then?

I also found out that the question marks exist in the database text field too..

Edited May 31, 2019 by Dolfijn

[musicmaster]

41 minutes ago, Dolfijn said:

I tried the same (copying the symbols from the forum into a new emtpy post). And what I get is:

"But symbols like ? (gamma), ? (phie) and  ? (pie) are all replaced as a ? (question mark )."

It happens immediately after saving the post. So, when I re-edit the post I just saved, the symbols are already replaced by question marks (so i'm still in the back office). Can that still be related to the theme / css and / or font then?

I also found out that the question marks exist in the database text field too..

What char set are you using in your TB database? 

[Dolfijn]

The database uses : collation utf8mb4_unicode_ci

[Traumflug]

Maybe a matter of HTMLPurifier. See back office -> Preferences -> General, 4th switch.

[Dolfijn]

I switched the HTMLPurifier off but the problem remains.

[Dolfijn]

I also looked up the server configuration:

• Server: Localhost via UNIX socket
• Server type: MySQL
• Server connection: SSL is not being used
• Server version: 5.6.42 - MySQL Community Server (GPL)
• Protocol version: 10
• User:
• Server charset: UTF-8 Unicode (utf8)

From what I now understand, the problem could be in the charset UTF-8 (uses only three bytes). It should be utfmb4 (uses 4 bytes).

Can that be right?

Edited June 1, 2019 by Dolfijn

[musicmaster]

Try copying text from a source that is guaranteed utf-8 and not latin, like this: https://www.bbc.com/russian/vert-fut-48422051

What happens when you try such texts in product names?

[Dolfijn]

That gives :

?? ???? ?????, ????? ????????, ? ????? ??????. ? ????? ???????????? ?????? ? ???? ???????, ? ?? ??????, ??????, ??????…

From what I understand, the mysql database is using UTF-8 instead of utf8mb4. I think I will give changing the sql db charset a try..

[musicmaster]

The difference between UTF-8 and utf8mb4 is marginal. All shops migrated from Prestashop use utf-8

[musicmaster]

Did you try using it in product names too?

[Dolfijn]

Sorry, I forgot. I pasted the symbols in the title of a test product. And then all the symbols are displayed correctly!! Both in backend and front-end. What does this mean?

[musicmaster]

A database as a whole has a char set. But individual tables in it can have different char sets. Can you check what char sets the blog tables use?

[Dolfijn]

I checked.

But first I have to mention that I made an error looking up the default collation. Is is not utf8mb4_unicode_ci  as I reported but latin1_swedish_ci. Also, i found out that each individuel table uses Colloation utf8_general_ci. Except for all the bees blog tabels. They are all in latin1_swedish_ci.

I guess that is due to the fact that the MySQL version I use has a default 'latin1' config. And all the other tables are imported from PS and in the sql export file, all tables are declared with " ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;"

So, I now expect that if a change the charset of the bees blog tables to utf8 the problem will be solved. Agree?

By the way, should I change the default collation into utf8mb4_unicode_ci too (and the default char set to utf8) at the database level?

[musicmaster]

8 hours ago, Dolfijn said:

So, I now expect that if a change the charset of the bees blog tables to utf8 the problem will be solved. Agree?

By the way, should I change the default collation into utf8mb4_unicode_ci too (and the default char set to utf8) at the database level?

Two times: yes. The first  thing is easy to test.