Migration guide for changes in naming convention

[trin94]

Documentation description

We are experiencing difficulties migrating from dlt 1.3.0 to dlt 1.5.0 due to changes in the normalization behavior, specifically related to the contraction of underscores.

My current understanding:

We can upgrade from 1.3.0 to 1.5.0 by simply updating the version of the dependency. The _dlt_version table stores the schema used when the source was created and maintains the original normalization behavior despite changes in newer versions of dlt. However, new sources will adopt the updated normalization behavior.

If we want to switch to the new behavior immediately, we can set

[schema]
use_break_path_on_normalize=true

This setting will cause the system to ignore the normalization behavior stored in the schema of existing sources and immediately apply the new behavior.

Our Problem

We have multiple sources with nested records created using the previous version of dlt, resulting in table names like:

• aaa_bbbb_cccccccc__dddddddd_ddd

Using the updated behavior, this table name would change to:

• aaa_bbbb__cccccccc__dddddddd_ddd

Locally, we often start with a clean state and no existing tables, including during tests. This means new sources will be created using the updated behavior.

We are uncertain about how to proceed. Should we migrate all existing table names to the new pattern, or should we set use_break_path_on_normalize to true and merge new tables with the old ones if they exist? Any recommendations on how to resolve this issue would be greatly appreciated. :)

Are you a dlt user?

Yes, I run dlt in production.