A path to ex_cldr 3.0 (Localize 1.0)

[kipcole9]

After considerable reflection I'm preparing to embark on the next generation of ex_cldr. And since it will have breaking changes it will be a 3.0 release. One of the objectives is to make the library easier to discover, understand and use for developers. I will likely also rename it to localize, version 1.0.

I expect that development will take most of 2025 to complete.

Localize 1.0 (ex_cldr 3.0) goals

Since the first commit in May 2016, ex_cldr has been installed nearly 5m times, suggesting it has good support from the Elixir community. At the same time OTP and Elixir have improved. A new type system is in active development. And some of the constraints of the existing family of nearly 30 ex_cldr libraries have continued to irritate.

The main goals are:

Rename to Localize

Make the libraries more discoverable and name them in a way that is more reflective of what the libraries do. CLDR is the core data and specification. The CLDR teams main library is called icu. The javascript implementation is intl. In my opinion, icu is just as opaque as CLDR. Internationalisation implies a distinction between "us" and "others" - reinforcing the common perception that localisation is something that can be deferred to "later". As Wikipedia defines:

Internationalization is the process of designing a software application so that it can be adapted to various languages and regions without engineering changes. Localization is the process of adapting internationalized software for a specific region or language by translating text and adding locale-specific components.

Therefore the implementation of localize (ex_cldr) is "Internationalisation". The use of the libraries is "Localization". My hope is that the libraries will become more mainstream, and the standard go-to for formatting numbers, dates, times, Phoenix routes and so on. Whether the application is targeting only one language or culture or many.

Remove the "backend" approach to hosting CLDR data

During the first implementation of ex_cldr, gettext was the architectural model. Hence the CLDR data is encapsulated in "backend modules" that are configured at compile time. There are benefits to his approach on the BEAM, but there are also limitations:

1. The locale configuration needs to be specified at compile time. This limits the ability to add new locales to a running system.
2. Slow compilation for configurations with many locales. The compilation speed issue is not the Elixir compiler, its in the later BEAM passes. More specifically, then the number of function heads for a given function increase, at a certain point the performance degrades exponentially.
3. It raises the barrier to adoption since a backend must be configured before any functionality can be used. That doesn't fit well with a REPL-oriented or Livebook-oriented approach to development.

Separate the core CLDR data ingestion and data loading

The current ex_cldr implementation incorporates the modules that important, transform and store the raw CLDR locale data. From a developers point-of-view, these modules are irrelevant at best and and overhead at worst. It also makes contirbuting to ex_cldr more difficult because of the combination of concerns. The updated implementation with clearly separate the data transform process and more strictly define the the transformed locale data struct.

Simplify library packaging and testing

The proliferation of libraries has historically intended to serve a few purposes:

1. Reduce the memory footprint by only encapsulating locale data specific to a given library. If you just want to format numbers, then only the number formats are baked in.
2. Keep the release process flexible. By separating functional requirements into separate libraries (like numbers, currencies, dates/times, units, ...) each of the libraries can be maintained and released separately as required. This was very useful in the first few years as new libraries have been developed. However the last couple of years have seem a higher level of stability as the libraries have matured and this proliferation now makes testing more complex that it should be.

This new development will re-package into a smaller number of libraries. One possible packaging might be:

• localize_transformer to be the CLDR data transformer
• Localize - combine current ex_cldr, ex_cldr_numbers, ex_cldr_currencies, ex_cldr_dates_times, ex_cldr_calendars, ex_cldr_units (formatting only), ex_cldr_territories and ex_cldr_languages into a single library. These are the core formatting libraries.
• There are a couple of Phoenix-related libraries: ex_cldr_html and ex_cldr_localized_routes that could be combined into localize_phoenix library.

Canonical error and exception handling

Today the libraries consistently return {:error, {ExceptionStruct, binary_message}} which date back to when I didn't correctly understand exception structs. In the new version the return should be {:error, ExceptionStruct} with correctly defined exceptions including a message formatter that can, itself, be translated.

Leverage the emerging type system

A simple majority of bug reports on ex_cldr libraries has been around dialyzer warnings. Today most of the common libraries are quite robust and will pass dialyzer with with flags :error_handling, :unknown, :underspecs, :extra_return, :missing_return but these tend to lend little overall developer benefit.

The aim of the new libraries will be to fully leverage the emerging type system and type notation. The most significant work will likely be to type the locale data itself given how complex that data structure is. However the benefit is likely to be more correct code faster for library development (I don't expect it will provide great benefit to library consumers).

Build a new translator library

With the approval of the UInicode Message Format 2.0 in November 2024, the time is right to implement it for Elixir. In addition, there is room for a more fully-featured translation ecosystem for Elixir that goes beyond gettext. Gettext's backend architecture, compile-time configuration and .pot serialisation has benefits - especially consistency with other language ecosystems. However gettext messages have important drawbacks with plurals and grammatical gender and in how translators can easy understand the intent and context of a message. Message Format 2.0 seeks to overcome these issues and the timing is write from an Elixir perspective.

One possibility will be to explore making message translation pluggable in Phoenix after the library is able to provide a solid foundation.

The broader goals are to be able to:

• add translations at runtime (human built and AI translated),
• collaborate on message translations and share them in the Elixir community
• be able to serialise the messages in different ways (cloud storage, database, text files, ...)
• be able to translate through a Phoenix Live Dashboard
• potentially introduce an external service to act as a community translation portal

Re-implement units of measure

The library I most enjoy from a functional perceptive is ex_cldr_units but it has some major flaws due to my misunderstanding of the spec (to be fair, the spec has improved a lot in its readability in the last few years too). Remediating it to be more extendable, correct and maintainable is dififcult.

I plan to reimplement it and split into two libraries. One will focus on parsing, conversion and localisation. The second will focus on unit math - basically an algebra engine that should be able to work with any struct that supports a protocol I will define. Similar to how the ratio operates today. Indeed it may be possible to leverage the ratio library directly given that @Qqwy is far smarter than I am.

Fill out the missing pieces

On this new foundation, there is still work to be done to implement:

• localized collation
• text transforms

These are very interesting topics in their own right and deserve the time to implement them. I'm hoping with the new implementation there could be volunteers to help.

Other topics

Feedback is welcome on other areas of improvement. Especially with the goal of being a flexible runtime platform with improved DX, packaging, testing and integration and CLDR coverage.

cc: @LostKobrakai @Schultzer

[LostKobrakai]

I like the direction of all of the things you mentioned. I'd be curious mostly about the backend less architecture, but also very interested in message format 2.0 out of personal curiousity.

[kipcole9]

"No backend" architecture

My working hypothesis is that storing locale data in :persistent_term - one term per locale - will be very similar performance to the backend module approach. But with better compilation, less code duplication, fewer modules, and a much easier to understand data model (which could be reused by other localisation libraries).

Of course this hypothesis has to be validated. And there is some very tricky work for those areas - like number formatting - where the data is compiled to code. For those few but important cases, generating modules at runtime will be required. I suspect that will be trickier than I currently think.

[LostKobrakai]

That sounds about what I had in mind thinking about that topic over the years as well. Figuring out there tradeoffs between compile time inconvenience to runtime inconvenience will be an interesting challange. Maybe loading stuff from persistent_term can become the new default, while still retaining a way to optionally optimize (some of) the codepaths at compile time.

[Qqwy]

Funnily enough, 'generating modules at runtime' is how persistent_term worked until it became built-in.

[kipcole9]

Message Format 2.0

This format is syntactically and semantically more complex that Message Format 1.0 (which is implemented in ex_cldr_messages. This is primarily in the interests of making it dramatically more understandable by translators, with some compromises (in my opinion) to make it easier to target the javascript world.

There is a lot to unpack here - and the actual formatting is the most straight forward. In addition I want to break away from a Gettext dependency and a .pot dependency to a more pluggable and dynamic runtime.

One of the other design decisions is how to manage "message keys". In Gettext, the embedded message is the key. For complex messages in Message Format 2.0 this is very likely not a sustainable or scalable approach.

Suffice to say, there is a lot of design work required first to fulfil the vision I have - including being a first class part of the Phoenix ecosystem.

[LostKobrakai]

Yeah, I've looked at message format a few times this year and it's a really interesting format, but especially the integration with date formatting and other externally provided functionalities makes it also not straight forward to implement. I can see problem with message keys, but not really sure if there's much of an option besides the "it keys itself". Gettext uses a similarity score to either retain an existing translation or not. Really I'd consider figuring out how similar the input is (or not) to whatever is used to translate the content, not a problem of the tool providing the message format integration.

I think this one would be best of starting with understanding, parsing and rendering the format and the layer on the parts making it practical to use onto that.

[kipcole9]

I use a similar approach in ex_cldr_messages to Gettext. There needs first to be a canonicalisation step so that whitespace doesn't impact the similarity score (too much) - gettext doesn't have a whitespace/multiline issue but Message Format sure does.

[kipcole9]

Which means, I suspect, that an Elixir formatter plugin and an appropriate Sigil would be useful.

[LostKobrakai]

Yes. Eventually I'd love to get to a sigil, which can efficiently compose with html as well and return a LV optimized static vs. dynamic split.

[Munksgaard]

Will this help with the issue of localizing strings containing HTML tags formatting?

@kipcole9 and I have talked about this a few times and there has been some discussion on the forums about this issue as well, but I've never seen a really great solution.

Essentially the problem (as I see it) is that of localizing something like the following:

You accept our <.link href="...">terms</.link> when you sign up.

Right now, this cannot be done without fooling around with raw or splitting up the string. Splitting up the string makes it very difficult for translators to figure out how the different strings belong together. On the other hand, using raw requires me to trust that translators can write correct HTML in their translations, including potentially long attribute lists for (e.g. when styling something using Tailwind).

[Schultzer]

Using a sigil that parses the string would eventually fix this, but the sigil would have to be aware of MSGF 2 and heex, which I don’t think would be an issue at all, but there would be some duplication.

Keep in mind that there could be unknowns around tying a locale sigil and heex together which makes it infeasible.

[Qqwy]

basically an algebra engine that should be able to work with any struct that supports a protocol I will define

Let me know if/when you reach that point! I'd love to collaborate. I still have very fond memories of the time we worked on the calendar and datetime support in Elixir core.

(And also, thanks for your flattering words! 😅)