Unable to generate slug with Chinese characters

[billmn]

Bug Description

If you try to add Chinese characters in the Title field of an entry, Statamic doesn't generate the slug.

How to Reproduce

1. In the Title field of an entry, insert: 你好，世界
2. Press "Save"

Statamic version: 3.0.25

PHP version: 7.4.8

Install method (choose one):

• Fresh install from statamic/statamic

[duncanmcclean]

I think this issue is related to pid/speakingurl#61, an issue on speakingurl which is the package used by Statamic to create the slugs.

The update the maintainer suggests on there is for speakingurl to be swapped out in favour of Limax, another package.

[jackmcdade]

Doesn't look like we can use Limax as it's intended to be used server side (Node) and has a huge package load size if attempting to run client-side. I couldn't get it to compile.

If we want to support Chinese and other multibyte characters, we'll need to find a compact library that can handle it.

[sheldonkotyk]

Any news on this. Also affects taxonomy terms.

[jackmcdade]

What if we replaced https://github.com/pid/speakingurl with a javascript helper that hits a server-side endpoint so we can use the same logic on both client/server and not worry about js package loads. 🤔

@jasonvarga whatcha think?

[jesseleite]

Different issue, but slug related... #8429 cc/ @martyf

Maybe we can tackle both at once, if we're taking a pass on slug generation?

[jasonvarga]

What if we replaced pid/speakingurl with a javascript helper that hits a server-side endpoint so we can use the same logic on both client/server and not worry about js package loads.

That's what I was thinking too. At least for the more important language based ones like entry slugs. For stuff like field handles maybe it's less important.

[duncanmcclean]

Different issue, but slug related... #8429 cc/ @martyf

Maybe we can tackle both at once, if we're taking a pass on slug generation?

It doesn't look like I linked it to the PR but that edge case has since been fixed in the current slug implementation (see #8923). It'll also be fixed when we merge #9440 since Laravel's Str::slug helper covers it too 💪

[duncanmcclean]

This has been fixed in Statamic 5, which we're hoping to release in the next few days/weeks.