Support for generatedAlwaysAs columns?

[theelderbeever]

This question applies to both the python library as well as the rust one but does delta-rs spport/utilize generatedAlwaysAs columns? With version deltalake 0.10.0 I can read the table but the real question is in whether partition pruning will behave as expected? With the schema example below there are three generated columns of which __year__ and __month__ are partition fields. Are these respected on write and on read?

Schema(
    [
        Field("timestamp", PrimitiveType("timestamp"), nullable=True),
        Field(
            "__year__",
            PrimitiveType("integer"),
            nullable=True,
            metadata={"delta.generationExpression": "YEAR(timestamp)"},
        ),
        Field(
            "__month__",
            PrimitiveType("integer"),
            nullable=True,
            metadata={"delta.generationExpression": "MONTH(timestamp)"},
        ),
        Field(
            "__day__",
            PrimitiveType("integer"),
            nullable=True,
            metadata={"delta.generationExpression": "DAY(timestamp)"},
        ),
    ]
)

[roeap]

Hi @theelderbeever,

right now delta-rs and with that the python binding do not support generated columns. Although i think this is just an issue on the write path.

We are actively working on extending protocol support and will be adding features for the rest of the year.

[ahirner]

Any updated plans?

Although i think this is just an issue on the write path

If columns are generated from truncated timestamps and used to partition a table, could predicates be generated on the read path as well?

Lets take the table outline here: #1495. With the datafusion integration, the file would get pruned:

select * from my_table where timestamp>'2024-10-10';

[ion-elgreco]

@ahirner this is supported since latest release (0.24)