Updating failed. Error message: Could not update post in the database when UTF-8 symbol in paragraph

[oldrup]

Hey

Gutenberg fails saving the post, if it contains the utf-8 symbol Lightning Mood:🗲
https://www.utf8icons.com/character/128498/lightning-mood

See attached screenshot.

Pasting in utf-8 symbols/icons works in any other editor on the planet ✔ ✌

WordPress system health is 100%

Is anyone able to reproduce this issue?

[youknowriad]

I'm not able to reproduce this issue. Is it possible that you have somekind of proxy/firewall that prevents this on the network level. Consider checking the response of the API call on the network tab of your browser's development tools.

[oldrup]

Thanks for trying this out..

Tested on a different site, same WP version, same theme, no issue.

Testet again on original site with different utf-8 icons:

All icons with a 5 digit unicode hex code (like this one https://www.utf8icons.com/character/128497/lightning-mood-bubble) FAILS

All icons with a 4 digit hex code (like this one https://www.utf8icons.com/character/9889/high-voltage-sign) WORKS

I obviously didn't test all unicode symbols, but enough to notice the pattern.

On the site where it fails, the API call network tab returns this error when the post update fails:

api-fetch.min.js?ver=3.6.4:1 POST https://oldrup.dk/wp-json/wp/v2/posts/2024?_locale=user 500

I tested in both Chrome and Firefox, same results

I disabled all plugins, same results.

I'm not using a proxy. I was using Ghostery ad blocker, but disabled that too.

So in conclusion, it does not appear to be a widespread problem, but the weird difference about being able to save posts with 4 hex digits unicodes, and not 5 hex digits unicodes, does smell of some software input validation, idk...

If I can do anything else to diagnose and squash any possible bug, please let me know.
Bjarne

[lgsankhe]

Me too facing the same problem.. Trying to add UTF symbol - 😍

When I try to update post with 😍, its showing error Updating failed. Could not update post in the database

[koolhand]

Same. But.... a clue...

When using the classic editor plugin, switching to HTML mode and giving it the same Gutenberg-flavoured HTML generated, and specifying the &#; Unicode code, I can save without error.

<!-- wp:paragraph {"className":"arrow"} -->
<p class="arrow">&#129130;</p>
<!-- /wp:paragraph -->

Whereas even if I use the code editor, not the visual editor, in Gutenberg, I get the "Could not update post in the database" error. It's like Gutenberg is unescaping Unicode, not the other way around.

[youknowriad]

I don't think it's related to Gutenberg but more to the REST API somehow.

@oldrup do you think you have a specific CDN used on the failing websites? any plugins that are different (security plugins maybe)?

One way to try to get more details is to check the network response on your browser's console or your webserver logs.

[oldrup]

@youknowriad Tried again with WordPress 5.5.1, all plugins disabled. Same issue on my host.
There should not be a CDN involved, not that I know of.
I looked into the browsers console, and it does return an error when attempting to save with the UTF-8 icon - no idea what to make of it, really :/ :

`XHRPOSThttps://oldrup.dk/wp-json/wp/v2/posts/1979?_locale=user
[HTTP/2 500 Internal Server Error 1204ms]

POST
https://oldrup.dk/wp-json/wp/v2/posts/1979?_locale=user
Status500
Internal Server Error
VersionHTTP/2
Transferred1.08 kB (99 B size)

access-control-allow-credentials
	true
access-control-allow-headers
	Authorization, X-WP-Nonce, Content-Disposition, Content-MD5, Content-Type
access-control-allow-methods
	OPTIONS, GET, POST, PUT, PATCH, DELETE
access-control-allow-origin
	https://oldrup.dk
access-control-expose-headers
	X-WP-Total, X-WP-TotalPages, Link
allow
	GET, POST, PUT, PATCH, DELETE
alt-svc
	quic=":443"; ma=2592000; v="43,46", h3-Q043=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-25=":443"; ma=2592000, h3-27=":443"; ma=2592000
cache-control
	no-cache, must-revalidate, max-age=0
content-type
	application/json; charset=UTF-8
date
	Tue, 15 Sep 2020 17:16:11 GMT
expires
	Wed, 11 Jan 1984 05:00:00 GMT
link
	<https://oldrup.dk/wp-json/>; rel="https://api.w.org/"
server
	LiteSpeed
vary
	Origin,User-Agent
x-content-type-options
	nosniff
X-Firefox-Spdy
	h2
x-litespeed-cache-control
	no-cache
x-robots-tag
	noindex
x-wp-nonce
	1c65bd7c86
	
Accept
	application/json, */*;q=0.1
Accept-Encoding
	gzip, deflate, br
Accept-Language
	en-GB,da;q=0.7,en;q=0.3
Cache-Control
	no-cache
Connection
	keep-alive
Content-Length
	9447
Content-Type
	application/json
Cookie
	ls_smartpush=fff; _lscache_vary=78b58861fda01604b033a3f4e5845eaf; wordpress_logged_in_78ae7e08c3223b3f3aae6911ba99bc41=bjarne%7C1600623815%7CPDjtJpzMBa2mHYyVCyGGUoktbPh5UVOCCr8uxCZQlkN%7C1420fde9519d0097bc6432ebf1ac292fe709b6215b0a4f44f11118b609b33aa5; wp-settings-1=libraryContent%3Dbrowse%26mfold%3Do; wp-settings-time-1=1599414215; wfwaf-authcookie-b33243b1e2cdb940fc54ee045600ba84=1%7Cadministrator%7C31617478cc4d29a7972d09e8d524b30d7913b12df644121ee99b92ca2924ebb3
DNT
	1
Host
	oldrup.dk
Origin
	https://oldrup.dk
Pragma
	no-cache
Referer
	https://oldrup.dk/wp-admin/post.php?post=1979&action=edit
User-Agent
	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0
X-HTTP-Method-Override
	PUT
X-WP-Nonce
	1c65bd7c86`

[youknowriad]

I'm really sorry @oldrup :( but I'm not sure we can do anything to help here to be honest.

I'm going to close the issue for now as it seems specific to some installs but please reopen if you do find anything that can allow us to reproduce here.

[eliasgomezsainz]

Same here. Switching to Classic Editor allowed me to save.

[stormwild]

This helped for me: https://stackoverflow.com/questions/63100373/wordpress-unable-to-save-or-update-posts-with-emoji

Added to the functions.php of the currently active theme using the Theme File Editor.

add_filter( 'wp_insert_post_data', function( $data, $postarr ) {
if ( ! empty( $data['post_content'] ) ) {
    $data['post_content'] = wp_encode_emoji( $data['post_content'] );
}
return $data;
}, 99, 2 );

[planetahuevo]

@youknowriad
This is happening to me too on a 6.3 brand new site. Trying to copy these emojis:

<!--StartFragment--><google-sheets-html-origin><style type="text/css"><!--td {border: 1px solid #cccccc;}br {mso-data-placement:same-cell;}--></style>

🏡 | 🏘️ | 🏙️  
-- | -- | --


<!--EndFragment-->

[youknowriad]

I wonder if it has something to do with the server. Would be cool to find the exact failure. Personally I'm still not able to reproduce.

[planetahuevo]

I was the server. The database was set as utf8mb3 instead of utf8mb4.
For anyone having this problems, ask their hosting to fix this.

[koolhand]

Nice find!

That might relate to my experience above - entering the &#; Unicode code in the HTML worked, but raw emoji didn't.

Could Gutenburg be submitting the raw emoji and not encoding as an HTML entity? So showing source and putting in HTML entity ☝ always succeeds because it submits it that way. But using the WYSIWYG editor submits the emoji ☝ as is, doesn't encode as the HTML entity code, and so only succeeds when the DB charset natively supports emoji characters?

[planetahuevo]

Whatever Gutenberg is doing, with a proper database it should work, and when is not, is giving us information about it.
I do not think GB should change anything here.
But yes, the problem is that the unicode code is not 4 bytes, so it works with UTF8mb3, but the emoji is not so it requires the mb4.
https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html
This happens too in other areas of WP, post names, etc, and with the classic editor. That is why I found it so easy, because it happened to me in the past.

[grantbarrett]

I'm having the same issue. Using flag emojis causes a page update to fail. Removing them lets it succeed.

[grantbarrett]

Here are two other places where this problem has been reported: https://wordpress.org/support/topic/problem-when-saving-with-emoji/#post-17117624 and https://wordpress.org/support/topic/emojis-not-working-in-wordpress-posts/

[grantbarrett]

PS: Following these instructions solved the issue for me: https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8

I didn't have sufficient privileges to just run a SQL command in phpMyAdmin, so I did a modified version of what is given there.

I logged into my server via the shell, then generated a list of tables with this:

mysql -e "show full tables where Table_Type = 'BASE TABLE'"

Then I converted that output to a .sql file (which I named "change_charset.sql") where each table gets a line like this:

ALTER TABLE ReplaceWithTableName CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

At the very top of that file put this line:

ALTER DATABASE DATABASENAME CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Then I uploaded that change_charset.sql file to my server and figured out its path, then executed this command:

mysql -u awww -pPASSWORD -h HOST -P PORTNUMBER DATABASENAME < /sites/user/change_charset.sql

If you are going to follow my steps, replace PASSWORD, HOST, PORTNUMBER (if needed; if not needed, remove "-P PORTNUMBER"), and DATABASENAME in the above MySQL command with the equivalents for your own server. You can find these in your wp-config.php file in your root HTTP/web directory.

[kbarkhurst]

A combination of two updates fixed this for me: an update to the wp_config.php file and an update to the function.php (which I implemented using the Snippets plugin) rather than updating directly.

An example of an emoji that was causing a problem is the mailbox, copied and pasted from here directly into the WP editor: https://emojipedia.org/closed-mailbox-with-raised-flag
📫

WP_Config.php Update:

/** Database Charset to use in creating database tables. */
define( 'DB_CHARSET', 'utf8mb4' );

/** The Database Collate type. */
define('DB_COLLATE', 'utf8mb4_unicode_ci');

The Function.php update that worked for me is provided by @stormwild above. It uses the wp_encode_emoji function that helps store emojis in your WordPress database using the UTF8 Character set. It can also be referenced at https://developer.wordpress.org/reference/functions/wp_encode_emoji/

<?php
add_filter( 'wp_insert_post_data', function( $data, $postarr ) {
if ( ! empty( $data['post_content'] ) ) {
    $data['post_content'] = wp_encode_emoji( $data['post_content'] );
}
return $data;
}, 99, 2 );

Hope this helps someone else, too.

[CarusoApps]

More info here

https://www.reddit.com/r/Wordpress/comments/1as7s8v/updating_failed_could_not_insert_post_into_the/

and here

https://core.trac.wordpress.org/ticket/60362

[Mousteay]

A combination of two updates fixed this for me: an update to the wp_config.php file and an update to the function.php (which I implemented using the Snippets plugin) rather than updating directly.

An example of an emoji that was causing a problem is the mailbox, copied and pasted from here directly into the WP editor: https://emojipedia.org/closed-mailbox-with-raised-flag 📫

WP_Config.php Update:

/** Database Charset to use in creating database tables. */
define( 'DB_CHARSET', 'utf8mb4' );

/** The Database Collate type. */
define('DB_COLLATE', 'utf8mb4_unicode_ci');

The Function.php update that worked for me is provided by @stormwild above. It uses the wp_encode_emoji function that helps store emojis in your WordPress database using the UTF8 Character set. It can also be referenced at https://developer.wordpress.org/reference/functions/wp_encode_emoji/

<?php
add_filter( 'wp_insert_post_data', function( $data, $postarr ) {
if ( ! empty( $data['post_content'] ) ) {
    $data['post_content'] = wp_encode_emoji( $data['post_content'] );
}
return $data;
}, 99, 2 );

Hope this helps someone else, too.

This helps me out... Thanks

[Nalci]

Thank you so much @kbarkhurst ! It solved the problem I had in my database since I set up my site. It apparently saved emojis as they are and now the emojis getting saved as HTML entity.

I only had to hit save on every page and post where I'm using emojis. And don't forget to delete the old revisions with the plain emoji/s.

I got the following in my database when I searched for something:
Warning: #1977 Cannot convert 'utf8mb4' character 0xF09F988A to 'utf8mb3'
Warning: #1977 Cannot convert 'utf8mb4' character 0xF09F91BE to 'utf8mb3'

It's so weird because all tables are in utf8mb4.

[Yadley]

A combination of two updates fixed this for me: an update to the wp_config.php file and an update to the function.php (which I implemented using the Snippets plugin) rather than updating directly.
An example of an emoji that was causing a problem is the mailbox, copied and pasted from here directly into the WP editor: https://emojipedia.org/closed-mailbox-with-raised-flag 📫
WP_Config.php Update:

/** Database Charset to use in creating database tables. */
define( 'DB_CHARSET', 'utf8mb4' );

/** The Database Collate type. */
define('DB_COLLATE', 'utf8mb4_unicode_ci');

The Function.php update that worked for me is provided by @stormwild above. It uses the wp_encode_emoji function that helps store emojis in your WordPress database using the UTF8 Character set. It can also be referenced at https://developer.wordpress.org/reference/functions/wp_encode_emoji/

<?php
add_filter( 'wp_insert_post_data', function( $data, $postarr ) {
if ( ! empty( $data['post_content'] ) ) {
    $data['post_content'] = wp_encode_emoji( $data['post_content'] );
}
return $data;
}, 99, 2 );

Hope this helps someone else, too.

This helps me out... Thanks

This has worked for me in 2024, thank you for sharing!

[Tomblarom]

YESS thank you! :) That fixed it.

[simran-gera]

Yes that worked out for me!!
Thanks
Added this at the end of the file functions.php.

add_filter( 'wp_insert_post_data', function( $data, $postarr ) {
if ( ! empty( $data['post_content'] ) ) {
    $data['post_content'] = wp_encode_emoji( $data['post_content'] );
}
return $data;
}, 99, 2 );