From d94f159a5f0f4937c23a760e68247b30e0d673e3 Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Sun, 20 Oct 2024 15:15:18 -0700 Subject: [PATCH 01/15] Current work on variant updates. --- VariantEncoding.md | 90 ++++++++---- VariantShredding.md | 327 ++++++++++++++++++++++++++++---------------- 2 files changed, 275 insertions(+), 142 deletions(-) diff --git a/VariantEncoding.md b/VariantEncoding.md index c6d2d113..03bcd473 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -39,13 +39,31 @@ Another motivation for the representation is that (aside from metadata) each nes For example, in a Variant containing an Array of Variant values, the representation of an inner Variant value, when paired with the metadata of the full variant, is itself a valid Variant. This document describes the Variant Binary Encoding scheme. -[VariantShredding.md](VariantShredding.md) describes the details of the Variant shredding scheme. +The [Variant Shredding spec](VariantShredding.md) describes the details of shredding Variant values as typed Parquet columns. -# Variant in Parquet -A Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`. -Both fields `value` and `metadata` are of type `binary`, and cannot be `null`. +## Variant in Parquet -# Metadata encoding +A Variant value in Parquet is represented by a group with 2 fields, named `variant_value` and `metadata`. +The Variant group must be annotated with the `VARIANT` logical type. +Both fields `value` and `metadata` are of type `binary`. +The `metadata` field is required and must be a valid Variant metadata, as defined below. +The `variant_value` field is optional. +When present, the `variant_value` field must be a valid Variant value, as defined below. +The `variant_value` field may be null only when parts of the Variant value are shredded according to the Variant Shreedding spec. + +This is the expected representation in Parquet: + +``` +optional group variant_event (VARIANT) { + required binary metadata; + optional binary variant_value; +} +``` + +There are no restrictions on the repetition of Variant groups (required, optional, or repeated). +The Variant group name is the name of the Variant column. + +## Metadata encoding The encoded metadata always starts with a header byte. ``` @@ -95,7 +113,7 @@ The first `offset` value will always be `0`, and the last `offset` value will al The last part of the metadata is `bytes`, which stores all the string values in the dictionary. All string values must be UTF-8 encoded strings. -## Metadata encoding grammar +### Metadata encoding grammar The grammar for encoded metadata is as follows @@ -119,7 +137,7 @@ Notes: - If `sorted_strings` is set to 1, strings in the dictionary must be unique and sorted in lexicographic order. If the value is set to 0, readers may not make any assumptions about string order or uniqueness. -# Value encoding +## Value encoding The entire encoded Variant value includes the `value_metadata` byte, and then 0 or more bytes for the `val`. ``` @@ -132,16 +150,16 @@ value | value_header | basic_type | | | +-------------------------------------------------+ ``` -## Basic Type +### Basic Type The `basic_type` is 2-bit value that represents which basic type the Variant value is. The [basic types table](#encoding-types) shows what each value represents. -## Value Header +### Value Header The `value_header` is a 6-bit value that contains more information about the type, and the format depends on the `basic_type`. -### Value Header for Primitive type (`basic_type`=0) +#### Value Header for Primitive type (`basic_type`=0) When `basic_type` is `0`, `value_header` is a 6-bit `primitive_header`. The [primitive types table](#encoding-types) shows what each value represents. @@ -152,7 +170,7 @@ value_header | primitive_header | +-----------------------+ ``` -### Value Header for Short string (`basic_type`=1) +#### Value Header for Short string (`basic_type`=1) When `basic_type` is `1`, `value_header` is a 6-bit `short_string_header`. ``` @@ -163,7 +181,7 @@ value_header | short_string_header | ``` The `short_string_header` value is the length of the string. -### Value Header for Object (`basic_type`=2) +#### Value Header for Object (`basic_type`=2) When `basic_type` is `2`, `value_header` is made up of `field_offset_size_minus_one`, `field_id_size_minus_one`, and `is_large`. ``` @@ -181,7 +199,7 @@ The actual number of bytes is computed as `field_offset_size_minus_one + 1` and `is_large` is a 1-bit value that indicates how many bytes are used to encode the number of elements. If `is_large` is `0`, 1 byte is used, and if `is_large` is `1`, 4 bytes are used. -### Value Header for Array (`basic_type`=3) +#### Value Header for Array (`basic_type`=3) When `basic_type` is `3`, `value_header` is made up of `field_offset_size_minus_one`, and `is_large`. ``` @@ -198,21 +216,21 @@ The actual number of bytes is computed as `field_offset_size_minus_one + 1`. `is_large` is a 1-bit value that indicates how many bytes are used to encode the number of elements. If `is_large` is `0`, 1 byte is used, and if `is_large` is `1`, 4 bytes are used. -## Value Data +### Value Data The `value_data` encoding format depends on the type specified by `value_metadata`. For some types, the `value_data` will be 0-bytes. -### Value Data for Primitive type (`basic_type`=0) +#### Value Data for Primitive type (`basic_type`=0) When `basic_type` is `0`, `value_data` depends on the `primitive_header` value. The [primitive types table](#encoding-types) shows the encoding format for each primitive type. -### Value Data for Short string (`basic_type`=1) +#### Value Data for Short string (`basic_type`=1) When `basic_type` is `1`, `value_data` is the sequence of UTF-8 encoded bytes that represents the string. -### Value Data for Object (`basic_type`=2) +#### Value Data for Object (`basic_type`=2) When `basic_type` is `2`, `value_data` encodes an object. The encoding format is shown in the following diagram: @@ -282,7 +300,7 @@ The `field_id` list must be `[, , , , , ]`. The `value` list can be in any order. -### Value Data for Array (`basic_type`=3) +#### Value Data for Array (`basic_type`=3) When `basic_type` is `3`, `value_data` encodes an array. The encoding format is shown in the following diagram: ``` @@ -323,7 +341,7 @@ The `field_offset` list is followed by the `value` list. There are `num_elements` number of `value` entries and each `value` is an encoded Variant value. For the i-th array entry, the value is the Variant `value` starting from the i-th `field_offset` byte offset. -## Value encoding grammar +### Value encoding grammar The grammar for an encoded value is: @@ -364,7 +382,7 @@ It is semantically identical to the "string" primitive type. The Decimal type contains a scale, but no precision. The implied precision of a decimal value is `floor(log_10(val)) + 1`. -# Encoding types +## Encoding types | Basic Type | ID | Description | |--------------|-----|---------------------------------------------------| @@ -375,7 +393,7 @@ The Decimal type contains a scale, but no precision. The implied precision of a | Logical Type | Physical Type | Type ID | Equivalent Parquet Type | Binary format | |----------------------|-----------------------------|---------|-----------------------------|---------------------------------------------------------------------------------------------------------------------| -| NullType | null | `0` | any | none | +| NullType | null | `0` | UNKNOWN | none | | Boolean | boolean (True) | `1` | BOOLEAN | none | | Boolean | boolean (False) | `2` | BOOLEAN | none | | Exact Numeric | int8 | `3` | INT(8, signed) | 1 byte | @@ -404,12 +422,12 @@ The *Logical Type* column indicates logical equivalence of physically encoded ty For example, a user expression operating on a string value containing "hello" should behave the same, whether it is encoded with the short string optimization, or long string encoding. Similarly, user expressions operating on an *int8* value of 1 should behave the same as a decimal16 with scale 2 and unscaled value 100. -# String values must be UTF-8 encoded +## String values must be UTF-8 encoded All strings within the Variant binary format must be UTF-8 encoded. This includes the dictionary key string values, the "short string" values, and the "long string" values. -# Object field ID order and uniqueness +## Object field ID order and uniqueness For objects, field IDs and offsets must be listed in the order of the corresponding field names, sorted lexicographically. Note that the field values themselves are not required to follow this order. @@ -423,14 +441,36 @@ Field names are case-sensitive. Field names are required to be unique for each object. It is an error for an object to contain two fields with the same name, whether or not they have distinct dictionary IDs. -# Versions and extensions +## Versions and extensions An implementation is not expected to parse a Variant value whose metadata version is higher than the version supported by the implementation. However, new types may be added to the specification without incrementing the version ID. In such a situation, an implementation should be able to read the rest of the Variant value if desired. -# Shredding +## Shredding A single Variant object may have poor read performance when only a small subset of fields are needed. A better approach is to create separate columns for individual fields, referred to as shredding or subcolumnarization. [VariantShredding.md](VariantShredding.md) describes the Variant shredding specification in Parquet. + +## Conversion to JSON + +Values stored in the Variant encoding are a superset of JSON values. +For example, a Variant value can be a date that has no equivalent type in JSON. +To maximize compatibility with readers that can process JSON but not Variant, the following conversions should be used when producing JSON from a Variant: + +| Variant type | JSON type | Representation requirements | Example | +|---------------|-----------|----------------------------------------------------------|--------------------------------------| +| Null type | null | `null` | `null` | +| Boolean | boolean | `true` or `false` | `true` | +| Exact Numeric | number | Digits in fraction must match scale, no exponent | `34`, 34.00 | +| Float | number | Fraction must be present | `14.20` | +| Double | number | Fraction must be present | `1.0` | +| Date | string | ISO-8601 formatted date | `"2017-11-16"` | +| Timestamp | string | ISO-8601 formatted UTC timestamp including +00:00 offset | `"2017-11-16T22:31:08.000001+00:00"` | +| TimestampNTZ | string | ISO-8601 formatted UTC timestamp with no offset or zone | `"2017-11-16T22:31:08.000001"` | +| Binary | string | Base64 encoded binary | `"dmFyaWFudAo="` | +| String | string | | `"variant"` | +| Array | array | | `[34, "abc", "2017-11-16]` | +| Object | object | | `{"id": 34, "data": "abc"}` | + diff --git a/VariantShredding.md b/VariantShredding.md index 31e1f528..ecb3d146 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -23,7 +23,7 @@ > **This specification is still under active development, and has not been formally adopted.** The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. -Query engines encode each Variant value in a self-describing format, and store it as a group containing `value` and `metadata` binary fields in Parquet. +Query engines encode each Variant value in a self-describing format, and store it as a group containing required `variant_value` and `metadata` binary fields in Parquet. Since data is often partially homogenous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. We refer to this process as **shredding**. Each Parquet file remains fully self-describing, with no additional metadata required to read or fully reconstruct the Variant data from the file. @@ -33,154 +33,247 @@ This document focuses on the shredding semantics, Parquet representation, implic For now, it does not discuss which fields to shred, user-facing API changes, or any engine-specific considerations like how to use shredded columns. The approach builds upon the [Variant Binary Encoding](VariantEncoding.md), and leverages the existing Parquet specification. -At a high level, we replace the `value` field of the Variant Parquet group with one or more fields called `object`, `array`, `typed_value`, and `variant_value`. -These represent a fixed schema suitable for constructing the full Variant value for each row. - Shredding allows a query engine to reap the full benefits of Parquet's columnar representation, such as more compact data encoding, min/max statistics for data skipping, and I/O and CPU savings from pruning unnecessary fields not accessed by a query (including the non-shredded Variant binary data). Without shredding, any query that accesses a Variant column must fetch all bytes of the full binary buffer. -With shredding, we can get nearly equivalent performance as in a relational (scalar) data model. +With shredding, readers can get nearly equivalent performance as in a relational (scalar) data model. + +For example, `SELECT variant_get(variant_event, '$.event_ts', 'timestamp') FROM tbl` only needs to access `event_ts`, and the file scan could avoid fetching the rest of the Variant value if this field was shredded into a separate column in the Parquet schema. +Similarly, for the query `SELECT * FROM tbl WHERE variant_get(variant_event, '$.event_type', 'string') = 'signup'`, the scan could first decode the shredded `event_type` column, and only fetch/decode the full Variant event value for rows that pass the filter. -For example, `select variant_get(variant_col, ‘$.field1.inner_field2’, ‘string’) from tbl` only needs to access `inner_field2`, and the file scan could avoid fetching the rest of the Variant value if this field was shredded into a separate column in the Parquet schema. -Similarly, for the query `select * from tbl where variant_get(variant_col, ‘$.id’, ‘integer’) = 123`, the scan could first decode the shredded `id` column, and only fetch/decode the full Variant value for rows that pass the filter. +## Variant Metadata -# Parquet Example +Variant metadata is stored in the top-level Variant group in a binary `metadata` column regardless of whether the Variant value is shredded. +All `variant_value` columns within the Variant must use the same `metadata`. -Consider the following Parquet schema together with how Variant values might be mapped to it. -Notice that we represent each shredded field in `object` as a group of two fields, `typed_value` and `variant_value`. -We extract all homogenous data items of a certain path into `typed_value`, and set aside incompatible data items in `variant_value`. -Intuitively, incompatibilities within the same path may occur because we store the shredding schema per Parquet file, and each file can contain several row groups. -Selecting a type for each field that is acceptable for all rows would be impractical because it would require buffering the contents of an entire file before writing. +All fields for a variant, whether shredded or not, must be present in the metadata. -Typically, the expectation is that `variant_value` exists at every level as an option, along with one of `object`, `array` or `typed_value`. -If the actual Variant value contains a type that does not match the provided schema, it is stored in `variant_value`. -An `variant_value` may also be populated if an object can be partially represented: any fields that are present in the schema must be written to those fields, and any missing fields are written to `variant_value`. +## Value Shredding -The `metadata` column is unchanged from its unshredded representation, and may be referenced in `variant_value` fields in the shredded data. +Variant values are stored in Parquet fields named `variant_value`. +Each `variant_value` field may have an associated shredded field named `typed_value` that stores the value when it matches a specific type. +For example, a Variant field, `measurement` may be shredded as long values by adding `typed_value` with type `int64`: ``` -optional group variant_col { - required binary metadata; - optional binary variant_value; - optional group object { - optional group a { - optional binary variant_value; - optional int64 typed_value; - } - optional group b { - optional binary variant_value; - optional group object { - optional group c { - optional binary variant_value; - optional binary typed_value (STRING); +optional group measurement (VARIANT) { + required binary metadata; + optional binary variant_value; + optional int64 typed_value; +} +``` + +Both `variant_value` and `typed_value` are optional fields used together to encode a single value. +Values in the two fields must be interpreted according to the following table: + +| `variant_value` | `typed_value` | Meaning | +| null | null | The value is missing | +| non-null | null | The value is present and may be any type, including null | +| null | non-null | The value is present and the shredded type | +| non-null | non-null | The value is present and a partially shredded object | + +An object is _partially shredded_ when the `variant_value` is an object and the `typed_value` is a shredded object. + +If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `variant_value`. + +### Shredded Value Types + +Shredded values must use the following Parquet types: + +| Variant Type | Equivalent Parquet Type | +|-----------------------------|------------------------------| +| boolean | BOOLEAN | +| int8 | INT(8, signed=true) | +| int16 | INT(16, signed=true) | +| int32 | INT32 / INT(32, signed=true) | +| int64 | INT64 / INT(64, signed=true) | +| float | FLOAT | +| double | DOUBLE | +| decimal4 | DECIMAL(precision, scale) | +| decimal8 | DECIMAL(precision, scale) | +| decimal16 | DECIMAL(precision, scale) | +| date | DATE | +| timestamp | TIMESTAMP(true, MICROS) | +| timestamp without time zone | TIMESTAMP(false, MICROS) | +| binary | BINARY | +| string | STRING | +| array | LIST; see Arrays below | +| object | GROUP; see Objects below | + +#### Primitive Types + +Primitive values can be shredded using the equivalent Parquet primitive type from the table above for `typed_object`. + +Unless the value is shredded in an object field, `typed_value` or `variant_value` (but not both) must be non-null. + +#### Arrays + +Arrays can be shredded using a 3-level Parquet list for `typed_value`. + +If the value is not an array, `typed_value` must be null. +If the value is an array, `variant_value` must be null. + +The list `element` must be a required group that contains a `variant_type` (`binary`) and may contain a shredded `typed_value` field. + +For example, a `tags` Variant may be shredded as a list of strings using the following definition: +``` +optional group tags (VARIANT) { + required binary metadata; + optional binary variant_value; + optional group typed_value (LIST) { // must be optional to allow a null list + repeated group list { + required group element { + optional binary variant_value; + optional binary typed_value (STRING); + } } - } } - } } ``` -| Variant Value | Top-level variant_value | b.variant_value | a.typed_value | a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes | -|---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------| -| {a: 123, b: {c: “hello”}} | null | null | 123 | null | hello | null | All values shredded | -| {a: 1.23, b: {c: “123”}} | null | null | null | 1.23 | 123 | null | a is not an integer | -| {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c set to non-null to indicate VariantNull | -| {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to null, to indicate that c is missing | -| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field d is stored as variant_value | -| [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | null | null | null | null | null | Not an object | - -# Parquet Layout +All elements of an array must be non-null, since `array` elements cannote be missing. +Either `typed_value` or `variant_value` (but not both) must be non-null. -The `array` and `object` fields represent Variant array and object types, respectively. -Arrays must use the three-level list structure described in [LogicalTypes.md](LogicalTypes.md). +#### Objects (Option 1) -An `object` field must be a group. -Each field name of this inner group corresponds to the Variant value's object field name. -Each inner field's type is a recursively shredded variant value: that is, the fields of each object field must be one or more of `object`, `array`, `typed_value` or `variant_value`. +Fields of an object can be shredded using a Parquet group for `typed_value` that contains shredded fields. -Similarly the elements of an `array` must be a group containing one or more of `object`, `array`, `typed_value` or `variant_value`. +If the value is not an object, `typed_value` must be null. -Each leaf in the schema can store an arbitrary Variant value. -It contains an `variant_value` binary field and a `typed_value` field. -If non-null, `variant_value` represents the value stored as a Variant binary. -The `typed_value` field may be any type that has a corresponding Variant type. -For each value in the data, at most one of the `typed_value` and `variant_value` may be non-null. -A writer may omit either field, which is equivalent to all rows being null. + +If the value is a partially shredded object, the `variant_value` must not contain shredded fields. If such fields are present, the object is invalid and readers must either fail or use the values from the `variant_value`. -Dictionary IDs in a `variant_value` field refer to entries in the top-level `metadata` field. +Each shredded field is represented as a required group that contains a `variant_value` and a `typed_value` field. -For an `object`, a null field means that the field does not exist in the reconstructed Variant object. -All elements of an `array` must be non-null, since array elements cannote be missing. +For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: +``` +optional group event (VARIANT) { + required binary metadata; + optional binary variant_value; + optional group typed_value { + required group event_type { + optional binary variant_value; + optional binary typed_value (STRING); + } + required group event_ts { + optional binary variant_value; + optional int64 typed_value (TIMESTAMP(true, MICROS)); + } + } +} +``` -| typed_value | variant_value | Meaning | -|-------------|----------------|---------| -| null | null | Field is Variant Null (not missing) in the reconstructed Variant. | -| null | non-null | Field may be any type in the reconstructed Variant. | -| non-null | null | Field has this column’s type in the reconstructed Variant. | -| non-null | non-null | Invalid | +The group for each named field is required. +A field's `variant_value` and `typed_value` are set to null (missing) to indicate that the field does not exist in the variant. -The `typed_value` may be absent from the Parquet schema for any field, which is equivalent to its value being always null (in which case the shredded field is always stored as a Variant binary). -By the same token, `variant_value` may be absent, which is equivalent to their value being always null (in which case the field will always have the value Null or have the type of the `typed_value` column). +#### Objects (Option 2) -# Unshredded values +Fields of an object can be shredded using a Parquet group for `typed_value` that contains shredded fields. -If all values can be represented at a given level by whichever of `object`, `array`, or `typed_value` is present, `variant_value` is set to null. +If the value is not an object, `typed_value` must be null. -If a value cannot be represented by whichever of `object`, `array`, or `typed_value` is present in the schema, then it is stored in `variant_value`, and the other fields are set to null. -In the Parquet example above, if field `a` was an object or array, or a non-integer scalar, it would be stored in `variant_value`. +If the value is a partially shredded object, the `variant_value` contains non-shredded fields. If a field name is defined in the `variant_value` and the shredded field is non-null, the object is invalid and readers must either fail or return the value contained in `variant_value`. -If a value is an object, and the `object` field is present but does not contain all of the fields in the value, then any remaining fields are stored in an object in `variant_value`. -In the Parquet example above, if field `b` was an object of the form `{"c": 1, "d": 2}"`, then the object `{"d": 2}` would be stored in `variant_value`, and the `c` field would be shredded recursively under `object.c`. +Each shredded field of the object is shredded as an optional field using the equivalent Parquet type from the table above using the field name. -Note that an array is always fully shredded if there is an `array` field, so the above consideration for `object` is not relevant for arrays: only one of `array` or `variant_value` may be non-null at a given level. +For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: +``` +optional group event (VARIANT) { + required binary metadata; + optional binary variant_value; + optional group typed_value { + optional binary event_type (STRING); + optional int64 event_ts (TIMESTAMP(true, MICROS)); + } +} +``` -# Using variant_value vs. typed_value +A shredded field set to null to indicate that the field does not exist in the variant or is defined in `variant_value`. -In general, it is desirable to store values in the `typed_value` field rather than the `variant_value` whenever possible. -This will typically improve encoding efficiency, and allow the use of Parquet statistics to filter at the row group or page level. -In the best case, the `variant_value` fields are all null and the engine does not need to read them (or it can omit them from the schema on write entirely). -There are two main motivations for including the `variant_value` column: +## Nesting -1) In a case where there are rare type mismatches (for example, a numeric field with rare strings like “n/a”), we allow the field to be shredded, which could still be a significant performance benefit compared to fetching and decoding the full value/metadata binary. -2) Since there is a single schema per file, there would be no easy way to recover from a type mismatch encountered late in a file write. Parquet files can be large, and buffering all file data before starting to write could be expensive. Including a variant column for every field guarantees we can adhere to the requested shredding schema. +``` +optional group event (VARIANT) { + required binary metadata; + optional binary variant_value; + optional group typed_value { + required group event_type { + optional binary variant_value; + optional binary typed_value (STRING); + } + required group event_ts { + optional binary variant_value; + optional int64 typed_value (TIMESTAMP(true, MICROS)); + } + required group location { + optional binary variant_value; + optional group typed_value { + required group latitude { + optional binary variant_value; + optional double typed_value; + } + required group longitude { + optional binary variant_value; + optional double typed_value; + } + } + } + required group tags { + optional binary variant_value; + optional group typed_value (LIST) { + repeated group list { + required group element { + optional binary variant_value; + optional binary typed_value (STRING); + } + } + } + } + } +} +``` -# Top-level metadata + +``` +optional group event (VARIANT) { + required binary metadata; + optional binary variant_value; + optional group typed_value { + optional binary event_type (STRING); + optional int64 event_ts (TIMESTAMP(true, MICROS)); + optional group location { + optional double latitude; + optional double longitude; + } + optional group tags (LIST) { + repeated group list { + required group element { + optional binary variant_value; + optional binary typed_value (STRING); + } + } + } + } +} +``` -Any values stored in a shredded `variant_value` field may have dictionary IDs referring to the metadata. -There is one metadata value for the entire Variant record, and that is stored in the top-level `metadata` field. -This means any `variant_value` values in the shredded representation is only the "value" portion of the [Variant Binary Encoding](VariantEncoding.md). -The metadata is kept at the top-level, instead of shredding the metadata with the shredded variant values because: -* Simplified shredding scheme and specification. No need for additional struct-of-binary values, or custom concatenated binary scheme for `variant_value`. -* Simplified and good performance for write shredding. No need to rebuild the metadata, or re-encode IDs for `variant_value`. -* Simplified and good performance for Variant reconstruction. No need to re-encode IDs for `variant_value`. +| Variant Value | Top-level variant_value | b.variant_value | a.typed_value | a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes | +|---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------| +| {a: 123, b: {c: "hello"}} | null | null | 123 | null | hello | null | All values shredded | +| {a: 1.23, b: {c: "123"}} | null | null | null | 1.23 | 123 | null | a is not an integer | +| {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c set to non-null to indicate VariantNull | +| {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to null, to indicate that c is missing | +| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field d is stored as variant_value | +| [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | null | null | null | null | null | Not an object | -# Data Skipping +## Data Skipping Shredded columns are expected to store statistics in the same format as a normal Parquet column. In general, the engine can only skip a row group or page if all rows in the `variant_value` field are null, since it is possible for a `variant_get` expression to successfully cast a value from the `variant_value` to the target type. -For example, if `typed_value` is of type `int64`, then the string “123” might be contained in `variant_value`, which would not be reflected in statistics, but could be retained by a filter like `where variant_get(col, “$.field”, “long”) = 123`. +For example, if `typed_value` is of type `int64`, then the string "123" might be contained in `variant_value`, which would not be reflected in statistics, but could be retained by a filter like `where variant_get(col, "$.field", "long") = 123`. If `variant_value` is all-null, then the engine can prune pages or row groups based on `typed_value`. This specification is not strict about what values may be stored in `variant_value` rather than `typed_value`, so it is not safe to skip rows based on `typed_value` unless the corresponding `variant_value` column is all-null, or the engine has specific knowledge of the behavior of the writer that produced the shredded data. -# Shredding Semantics - -Reconstruction of Variant value from a shredded representation is not expected to produce a bit-for-bit identical binary to the original unshredded value. -For example, in a reconstructed Variant value, the order of object field values may be different from the original binary. -This is allowed since the [Variant Binary Encoding](VariantEncoding.md#object-field-id-order-and-uniqueness) does not require an ordering of the field values, but the field IDs will still be ordered lexicographically according to the corresponding field names. - -The physical representation of scalar values may also be different in the reconstructed Variant binary. -In particular, the [Variant Binary Encoding](VariantEncoding.md) considers all integer and decimal representations to represent a single logical type. -This flexibility enables shredding to be applicable in more scenarios, while maintaining all information and values losslessly. -As a result, it is valid to shred a decimal into a decimal column with a different scale, or to shred an integer as a decimal, as long as no numeric precision is lost. -For example, it would be valid to write the value 123 to a Decimal(9, 2) column, but the value 1.234 would need to be written to the `variant_value` column. -When reconstructing, it would be valid for a reader to reconstruct 123 as an integer, or as a Decimal(9, 2). -Engines should not depend on the physical type of a Variant value, only the logical type. - -On the other hand, shredding as a different logical type is not allowed. -For example, the integer value 123 could not be shredded to a string `typed_value` column as the string "123", since that would lose type information. -It would need to be written to the `variant_value` column. - -# Reconstructing a Variant +## Reconstructing a Variant It is possible to recover a full Variant value using a recursive algorithm, where the initial call is to `ConstructVariant` with the top-level fields, which are assumed to be null if they are not present in the schema. @@ -216,7 +309,7 @@ def ConstructArray(array): newVariantArray.append(ConstructVariant(array[i].variant_value, array[i].object, array[i].array, array[i].typed_value) ``` -# Nested Parquet Example +## Nested Parquet Example This section describes a more deeply nested example, using a top-level array as the shredding type. @@ -237,7 +330,7 @@ It contains an array of objects, containing an `a` field shredded as an array, a ``` -The corresponding Parquet schema with “a” and “b” as leaf types is: +The corresponding Parquet schema with "a" and "b" as leaf types is: ``` optional group variant_col { @@ -270,7 +363,7 @@ optional group variant_col { } ``` -In the above example schema, if “a” is an array containing a mix of integer and non-integer values, the engine will shred individual elements appropriately into either `typed_value` or `variant_value`. +In the above example schema, if "a" is an array containing a mix of integer and non-integer values, the engine will shred individual elements appropriately into either `typed_value` or `variant_value`. If the top-level Variant is not an array (for example, an object), the engine cannot shred the value and it will store it in the top-level `variant_value`. Similarly, if "a" is not an array, it will be stored in the `variant_value` under "a". @@ -281,13 +374,13 @@ Consider the following example: { "a": [1, 2, 3], "b": 100, - “c”: “unexpected” + "c": "unexpected" }, { "a": [4, 5, 6], "b": 200 }, - “not an object” + "not an object" ] ``` @@ -295,13 +388,13 @@ The second array element can be fully shredded, but the first and third cannot b ``` [ - { “c”: “unexpected” }, + { "c": "unexpected" }, NULL, - “not an object” + "not an object" ] ``` -# Backward and forward compatibility +## Backward and forward compatibility Shredding is an optional feature of Variant, and readers must continue to be able to read a group containing only a `value` and `metadata` field. From b91b9c43359a8b72b42c1fdb83b293c46de781d1 Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Tue, 22 Oct 2024 15:56:51 -0700 Subject: [PATCH 02/15] Remove Option 2, which cannot be used because stats aren't trusted. --- VariantEncoding.md | 10 +-- VariantShredding.md | 178 +++++++++++++++++--------------------------- 2 files changed, 72 insertions(+), 116 deletions(-) diff --git a/VariantEncoding.md b/VariantEncoding.md index 03bcd473..a40429f6 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -43,20 +43,20 @@ The [Variant Shredding spec](VariantShredding.md) describes the details of shred ## Variant in Parquet -A Variant value in Parquet is represented by a group with 2 fields, named `variant_value` and `metadata`. +A Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`. The Variant group must be annotated with the `VARIANT` logical type. Both fields `value` and `metadata` are of type `binary`. The `metadata` field is required and must be a valid Variant metadata, as defined below. -The `variant_value` field is optional. -When present, the `variant_value` field must be a valid Variant value, as defined below. -The `variant_value` field may be null only when parts of the Variant value are shredded according to the Variant Shreedding spec. +The `value` field is optional. +When present, the `value` field must be a valid Variant value, as defined below. +The `value` field may be null only when parts of the Variant value are shredded according to the Variant Shreedding spec. This is the expected representation in Parquet: ``` optional group variant_event (VARIANT) { required binary metadata; - optional binary variant_value; + optional binary value; } ``` diff --git a/VariantShredding.md b/VariantShredding.md index ecb3d146..34fdf9fc 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -23,7 +23,7 @@ > **This specification is still under active development, and has not been formally adopted.** The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. -Query engines encode each Variant value in a self-describing format, and store it as a group containing required `variant_value` and `metadata` binary fields in Parquet. +Query engines encode each Variant value in a self-describing format, and store it as a group containing required `value` and `metadata` binary fields in Parquet. Since data is often partially homogenous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. We refer to this process as **shredding**. Each Parquet file remains fully self-describing, with no additional metadata required to read or fully reconstruct the Variant data from the file. @@ -43,36 +43,36 @@ Similarly, for the query `SELECT * FROM tbl WHERE variant_get(variant_event, '$. ## Variant Metadata Variant metadata is stored in the top-level Variant group in a binary `metadata` column regardless of whether the Variant value is shredded. -All `variant_value` columns within the Variant must use the same `metadata`. +All `value` columns within the Variant must use the same `metadata`. All fields for a variant, whether shredded or not, must be present in the metadata. ## Value Shredding -Variant values are stored in Parquet fields named `variant_value`. -Each `variant_value` field may have an associated shredded field named `typed_value` that stores the value when it matches a specific type. +Variant values are stored in Parquet fields named `value`. +Each `value` field may have an associated shredded field named `typed_value` that stores the value when it matches a specific type. For example, a Variant field, `measurement` may be shredded as long values by adding `typed_value` with type `int64`: ``` optional group measurement (VARIANT) { required binary metadata; - optional binary variant_value; + optional binary value; optional int64 typed_value; } ``` -Both `variant_value` and `typed_value` are optional fields used together to encode a single value. +Both `value` and `typed_value` are optional fields used together to encode a single value. Values in the two fields must be interpreted according to the following table: -| `variant_value` | `typed_value` | Meaning | +| `value` | `typed_value` | Meaning | | null | null | The value is missing | | non-null | null | The value is present and may be any type, including null | | null | non-null | The value is present and the shredded type | | non-null | non-null | The value is present and a partially shredded object | -An object is _partially shredded_ when the `variant_value` is an object and the `typed_value` is a shredded object. +An object is _partially shredded_ when the `value` is an object and the `typed_value` is a shredded object. -If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `variant_value`. +If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `value`. ### Shredded Value Types @@ -102,14 +102,14 @@ Shredded values must use the following Parquet types: Primitive values can be shredded using the equivalent Parquet primitive type from the table above for `typed_object`. -Unless the value is shredded in an object field, `typed_value` or `variant_value` (but not both) must be non-null. +Unless the value is shredded in an object field, `typed_value` or `value` (but not both) must be non-null. #### Arrays Arrays can be shredded using a 3-level Parquet list for `typed_value`. If the value is not an array, `typed_value` must be null. -If the value is an array, `variant_value` must be null. +If the value is an array, `value` must be null. The list `element` must be a required group that contains a `variant_type` (`binary`) and may contain a shredded `typed_value` field. @@ -117,11 +117,11 @@ For example, a `tags` Variant may be shredded as a list of strings using the fol ``` optional group tags (VARIANT) { required binary metadata; - optional binary variant_value; - optional group typed_value (LIST) { // must be optional to allow a null list + optional binary value; + optional group typed_value (LIST) { # must be optional to allow a null list repeated group list { - required group element { - optional binary variant_value; + required group element { # shredded element + optional binary value; optional binary typed_value (STRING); } } @@ -130,98 +130,78 @@ optional group tags (VARIANT) { ``` All elements of an array must be non-null, since `array` elements cannote be missing. -Either `typed_value` or `variant_value` (but not both) must be non-null. +Either `typed_value` or `value` (but not both) must be non-null. -#### Objects (Option 1) +#### Objects Fields of an object can be shredded using a Parquet group for `typed_value` that contains shredded fields. If the value is not an object, `typed_value` must be null. - -If the value is a partially shredded object, the `variant_value` must not contain shredded fields. If such fields are present, the object is invalid and readers must either fail or use the values from the `variant_value`. +If the value is a partially shredded object, the `value` must not contain the shredded fields. +If shredded fields are present in the variant object, it is invalid and readers must either fail or use the values from the `value`. -Each shredded field is represented as a required group that contains a `variant_value` and a `typed_value` field. +Each shredded field in the `typed_value` group is represented as a required group that contains optional `value` and `typed_value` fields. +This layout enables readers to skip data based on the field statistics for `value` and `typed_value`. For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: ``` optional group event (VARIANT) { required binary metadata; - optional binary variant_value; - optional group typed_value { - required group event_type { - optional binary variant_value; + optional binary value; # a variant, expected to be an object + optional group typed_value { # shredded fields for the variant object + required group event_type { # shredded field for event_type + optional binary value; optional binary typed_value (STRING); } - required group event_ts { - optional binary variant_value; + required group event_ts { # shredded field for event_ts + optional binary value; optional int64 typed_value (TIMESTAMP(true, MICROS)); } } } ``` -The group for each named field is required. -A field's `variant_value` and `typed_value` are set to null (missing) to indicate that the field does not exist in the variant. - -#### Objects (Option 2) - -Fields of an object can be shredded using a Parquet group for `typed_value` that contains shredded fields. - -If the value is not an object, `typed_value` must be null. +The group for each named field must be required. -If the value is a partially shredded object, the `variant_value` contains non-shredded fields. If a field name is defined in the `variant_value` and the shredded field is non-null, the object is invalid and readers must either fail or return the value contained in `variant_value`. - -Each shredded field of the object is shredded as an optional field using the equivalent Parquet type from the table above using the field name. - -For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: -``` -optional group event (VARIANT) { - required binary metadata; - optional binary variant_value; - optional group typed_value { - optional binary event_type (STRING); - optional int64 event_ts (TIMESTAMP(true, MICROS)); - } -} -``` +A field's `value` and `typed_value` are set to null (missing) to indicate that the field does not exist in the variant. -A shredded field set to null to indicate that the field does not exist in the variant or is defined in `variant_value`. +Statistics for the `typed_value` column can be used for row group or page skipping when `value` is always null. ## Nesting ``` optional group event (VARIANT) { required binary metadata; - optional binary variant_value; + optional binary value; optional group typed_value { required group event_type { - optional binary variant_value; + optional binary value; optional binary typed_value (STRING); } required group event_ts { - optional binary variant_value; + optional binary value; optional int64 typed_value (TIMESTAMP(true, MICROS)); } required group location { - optional binary variant_value; + optional binary value; optional group typed_value { required group latitude { - optional binary variant_value; + optional binary value; optional double typed_value; } required group longitude { - optional binary variant_value; + optional binary value; optional double typed_value; } } } required group tags { - optional binary variant_value; + optional binary value; optional group typed_value (LIST) { repeated group list { required group element { - optional binary variant_value; + optional binary value; optional binary typed_value (STRING); } } @@ -231,82 +211,58 @@ optional group event (VARIANT) { } ``` - -``` -optional group event (VARIANT) { - required binary metadata; - optional binary variant_value; - optional group typed_value { - optional binary event_type (STRING); - optional int64 event_ts (TIMESTAMP(true, MICROS)); - optional group location { - optional double latitude; - optional double longitude; - } - optional group tags (LIST) { - repeated group list { - required group element { - optional binary variant_value; - optional binary typed_value (STRING); - } - } - } - } -} -``` - -| Variant Value | Top-level variant_value | b.variant_value | a.typed_value | a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes | +| Variant Value | Top-level value | b.variant_value | a.typed_value | a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes | |---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------| | {a: 123, b: {c: "hello"}} | null | null | 123 | null | hello | null | All values shredded | | {a: 1.23, b: {c: "123"}} | null | null | null | 1.23 | 123 | null | a is not an integer | | {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c set to non-null to indicate VariantNull | | {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to null, to indicate that c is missing | -| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field d is stored as variant_value | +| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field d is stored as value | | [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | null | null | null | null | null | Not an object | ## Data Skipping Shredded columns are expected to store statistics in the same format as a normal Parquet column. -In general, the engine can only skip a row group or page if all rows in the `variant_value` field are null, since it is possible for a `variant_get` expression to successfully cast a value from the `variant_value` to the target type. -For example, if `typed_value` is of type `int64`, then the string "123" might be contained in `variant_value`, which would not be reflected in statistics, but could be retained by a filter like `where variant_get(col, "$.field", "long") = 123`. -If `variant_value` is all-null, then the engine can prune pages or row groups based on `typed_value`. -This specification is not strict about what values may be stored in `variant_value` rather than `typed_value`, so it is not safe to skip rows based on `typed_value` unless the corresponding `variant_value` column is all-null, or the engine has specific knowledge of the behavior of the writer that produced the shredded data. +In general, the engine can only skip a row group or page if all rows in the `value` field are null, since it is possible for a `variant_get` expression to successfully cast a value from the `variant_value` to the target type. +For example, if `typed_value` is of type `int64`, then the string "123" might be contained in `value`, which would not be reflected in statistics, but could be retained by a filter like `where variant_get(col, "$.field", "long") = 123`. +If `value` is all-null, then the engine can prune pages or row groups based on `typed_value`. +This specification is not strict about what values may be stored in `value` rather than `typed_value`, so it is not safe to skip rows based on `typed_value` unless the corresponding `variant_value` column is all-null, or the engine has specific knowledge of the behavior of the writer that produced the shredded data. ## Reconstructing a Variant It is possible to recover a full Variant value using a recursive algorithm, where the initial call is to `ConstructVariant` with the top-level fields, which are assumed to be null if they are not present in the schema. ``` -# Constructs a Variant from `variant_value`, `object`, `array` and `typed_value`. +# Constructs a Variant from `value`, `object`, `array` and `typed_value`. # Only one of object, array and typed_value may be non-null. -def ConstructVariant(variant_value, object, array, typed_value): - if object is null and array is null and typed_value is null and variant_value is null: return VariantNull +def ConstructVariant(value, object, array, typed_value): + if object is null and array is null and typed_value is null and value is null: return VariantNull if object is not null: - return ConstructObject(variant_value, object) + return ConstructObject(value, object) elif array is not null: return ConstructArray(array) elif typed_value is not null: return cast(typed_value as Variant) else: - variant_value + value -# Construct an object from an `object` group, and a (possibly null) Variant variant_value -def ConstructObject(variant_value, object): - # If variant_value is present and is not an Object, then the result is ambiguous. - assert(variant_value is null or is_object(variant_value)) +# Construct an object from an `object` group, and a (possibly null) Variant value +def ConstructObject(value, object): + # If value is present and is not an Object, then the result is ambiguous. + assert(value is null or is_object(variant_value)) # Null fields in the object are missing from the reconstructed Variant. nonnull_object_fields = object.fields.filter(field -> field is not null) - all_keys = Union(variant_value.keys, non_null_object_fields) + all_keys = Union(value.keys, non_null_object_fields) return VariantObject(all_keys.map { key -> - if key in object: (key, ConstructVariant(object[key].variant_value, object[key].object, object[key].array, object[key].typed_value)) - else: (key, variant_value[key]) + if key in object: (key, ConstructVariant(object[key].value, object[key].object, object[key].array, object[key].typed_value)) + else: (key, value[key]) }) def ConstructArray(array): newVariantArray = VariantArray() for i in range(array.size): - newVariantArray.append(ConstructVariant(array[i].variant_value, array[i].object, array[i].array, array[i].typed_value) + newVariantArray.append(ConstructVariant(array[i].value, array[i].object, array[i].array, array[i].typed_value) ``` ## Nested Parquet Example @@ -335,26 +291,26 @@ The corresponding Parquet schema with "a" and "b" as leaf types is: ``` optional group variant_col { required binary metadata; - optional binary variant_value; + optional binary value; optional group array (LIST) { repeated group list { optional group element { - optional binary variant_value; + optional binary value; optional group object { optional group a { - optional binary variant_value; + optional binary value; optional group array (LIST) { repeated group list { optional group element { optional int64 typed_value; - optional binary variant_value; + optional binary value; } } } } optional group b { optional int64 typed_value; - optional binary variant_value; + optional binary value; } } } @@ -363,9 +319,9 @@ optional group variant_col { } ``` -In the above example schema, if "a" is an array containing a mix of integer and non-integer values, the engine will shred individual elements appropriately into either `typed_value` or `variant_value`. -If the top-level Variant is not an array (for example, an object), the engine cannot shred the value and it will store it in the top-level `variant_value`. -Similarly, if "a" is not an array, it will be stored in the `variant_value` under "a". +In the above example schema, if "a" is an array containing a mix of integer and non-integer values, the engine will shred individual elements appropriately into either `typed_value` or `value`. +If the top-level Variant is not an array (for example, an object), the engine cannot shred the value and it will store it in the top-level `value`. +Similarly, if "a" is not an array, it will be stored in the `value` under "a". Consider the following example: @@ -384,7 +340,7 @@ Consider the following example: ] ``` -The second array element can be fully shredded, but the first and third cannot be. The contents of `variant_col.array[*].variant_value` would be as follows: +The second array element can be fully shredded, but the first and third cannot be. The contents of `variant_col.array[*].value` would be as follows: ``` [ @@ -398,7 +354,7 @@ The second array element can be fully shredded, but the first and third cannot b Shredding is an optional feature of Variant, and readers must continue to be able to read a group containing only a `value` and `metadata` field. -Any fields in the same group as `typed_value`/`variant_value` that start with `_` (underscore) can be ignored. +Any fields in the same group as `typed_value`/`value` that start with `_` (underscore) can be ignored. This is intended to allow future backwards-compatible extensions. In particular, the field names `_metadata_key_paths` and any name starting with `_spark` are reserved, and should not be used by other implementations. Any extra field names that do not start with an underscore should be assumed to be backwards incompatible, and readers should fail when reading such a schema. From 73d42131114646d06e9d9748a14e83419bb6781a Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 13:14:31 -0700 Subject: [PATCH 03/15] More updates to the Variant spec. --- VariantShredding.md | 270 +++++++++++++++++++------------------------- 1 file changed, 119 insertions(+), 151 deletions(-) diff --git a/VariantShredding.md b/VariantShredding.md index 34fdf9fc..764a9be2 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -43,9 +43,9 @@ Similarly, for the query `SELECT * FROM tbl WHERE variant_get(variant_event, '$. ## Variant Metadata Variant metadata is stored in the top-level Variant group in a binary `metadata` column regardless of whether the Variant value is shredded. -All `value` columns within the Variant must use the same `metadata`. -All fields for a variant, whether shredded or not, must be present in the metadata. +All `value` columns within the Variant must use the same `metadata`. +All field names of a Variant, whether shredded or not, must be present in the metadata. ## Value Shredding @@ -61,14 +61,24 @@ optional group measurement (VARIANT) { } ``` +The series of measurements `34, null, "n/a", 100` would be stored as: + +| Value | `metadata` | `value` | `typed_value` | +|---------|------------------|-----------------------|---------------| +| 34 | `01 00` v1/empty | null | `34` | +| null | `01 00` v1/empty | `00` (null) | null | +| "n/a" | `01 00` v1/empty | `13 6E 2F 61` (`n/a`) | null | +| 100 | `01 00` v1/empty | null | `100` | + Both `value` and `typed_value` are optional fields used together to encode a single value. Values in the two fields must be interpreted according to the following table: -| `value` | `typed_value` | Meaning | -| null | null | The value is missing | -| non-null | null | The value is present and may be any type, including null | -| null | non-null | The value is present and the shredded type | -| non-null | non-null | The value is present and a partially shredded object | +| `value` | `typed_value` | Meaning | +|----------|---------------|---------| +| null | null | The value is missing | +| non-null | null | The value is present and may be any type, including null | +| null | non-null | The value is present and the shredded type | +| non-null | non-null | The value is present and a partially shredded object | An object is _partially shredded_ when the `value` is an object and the `typed_value` is a shredded object. @@ -102,7 +112,7 @@ Shredded values must use the following Parquet types: Primitive values can be shredded using the equivalent Parquet primitive type from the table above for `typed_object`. -Unless the value is shredded in an object field, `typed_value` or `value` (but not both) must be non-null. +Unless the value is shredded as an object (see [Objects](#objects)), `typed_value` or `value` (but not both) must be non-null. #### Arrays @@ -111,7 +121,8 @@ Arrays can be shredded using a 3-level Parquet list for `typed_value`. If the value is not an array, `typed_value` must be null. If the value is an array, `value` must be null. -The list `element` must be a required group that contains a `variant_type` (`binary`) and may contain a shredded `typed_value` field. +The list `element` must be a required group that contains optional `value` and `typed_value` fields. +The element's `value` field stores the element as Variant-encoded `binary` when the `typed_value` cannot represent it. For example, a `tags` Variant may be shredded as a list of strings using the following definition: ``` @@ -129,8 +140,18 @@ optional group tags (VARIANT) { } ``` -All elements of an array must be non-null, since `array` elements cannote be missing. -Either `typed_value` or `value` (but not both) must be non-null. +All elements of an array must be non-null because `array` elements in a Variant cannot be missing. +That is, either `typed_value` or `value` (but not both) must be non-null. +Null elements must be encoded in `value` as Variant null: basic type 0 (primitive) and physical type 0 (null). + +The series of `tags` arrays `["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null` would be stored as: + +| Array | `value` | `typed_value `| `typed_value...value` | `typed_value...typed_value` | +|----------------------------------|-------------|---------------|-----------------------|--------------------------------| +| `["comedy", "drama"]` | null | non-null | [null, null] | [`comedy`, `drama`] | +| `["horror", null]` | null | non-null | [null, `00`] | [`horror`, null] | +| `["comedy", "drama", "romance"]` | null | non-null | [null, null, null] | [`comedy`, `drama`, `romance`] | +| null | `00` (null) | null | | | #### Objects @@ -139,9 +160,10 @@ Fields of an object can be shredded using a Parquet group for `typed_value` that If the value is not an object, `typed_value` must be null. If the value is a partially shredded object, the `value` must not contain the shredded fields. -If shredded fields are present in the variant object, it is invalid and readers must either fail or use the values from the `value`. +If shredded fields are present in the variant object, it is invalid and readers must either fail or use the shredded values. Each shredded field in the `typed_value` group is represented as a required group that contains optional `value` and `typed_value` fields. +The `value` field stores the value as Variant-encoded `binary` when the `typed_value` cannot represent the field. This layout enables readers to skip data based on the field statistics for `value` and `typed_value`. For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: @@ -165,11 +187,28 @@ optional group event (VARIANT) { The group for each named field must be required. A field's `value` and `typed_value` are set to null (missing) to indicate that the field does not exist in the variant. - -Statistics for the `typed_value` column can be used for row group or page skipping when `value` is always null. +To encode a field that is present with a null value, the `value` must contain a Variant null: basic type 0 (primitive) and physical type 0 (null). + +The series of objects below would be stored as: + +| Event object | `value` | `typed_value` | `typed_value.event_type.value` | `typed_value.event_type.typed_value` | `typed_value.event_ts.value` | `typed_value.event_ts.typed_value` | Notes | +|------------------------------------------------------------------------------------|-----------------------------------|---------------|--------------------------------|--------------------------------------|------------------------------|------------------------------------|-----------------------------------------------| +| `{"event_type": "noop", "event_ts": 1729794114937}` | null | non-null | null | `noop` | null | 1729794114937 | Fully shredded object | +| `{"event_type": "login", "event_ts": 1729794146402, "email": "user@example.com"}` | `{"email": "user@example.com"}` | non-null | null | `login` | null | 1729794146402 | Partially shredded object | +| `{"error_msg": "malformed: ..."}` | `{"error_msg", "malformed: ..."}` | null | | | | | Object with no shredding | +| `"malformed: not an object"` | `malformed: not an object` | null | | | | | Not an object (stored as Variant string) | +| `{"event_ts": 1729794240241, "click": "_button"}` | `{"click": "_button"}` | non-null | null | null | null | 1729794240241 | Field `event_type` is missing | +| `{"event_type": null, "event_ts": 1729794954163}` | null | non-null | `00` (field exists, is null) | null | null | 1729794954163 | Field `event_type` is present and is null | +| null | `00` (null) | null | | | | | Object/value is null | +| missing | null | null | | | | | Object/value is missing | +| INVALID | `{"event_type": "login"}` | non-null | null | `login` | null | 1729795057774 | INVALID: Shredded field is present in `value` | ## Nesting +The `typed_value` associated with any Variant `value` field can be any shredded type according to the rules above. + +For example, the `event` object above may also shred sub-fields as object (`location`) or array (`tags`). + ``` optional group event (VARIANT) { required binary metadata; @@ -211,153 +250,82 @@ optional group event (VARIANT) { } ``` +# Data Skipping -| Variant Value | Top-level value | b.variant_value | a.typed_value | a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes | -|---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------| -| {a: 123, b: {c: "hello"}} | null | null | 123 | null | hello | null | All values shredded | -| {a: 1.23, b: {c: "123"}} | null | null | null | 1.23 | 123 | null | a is not an integer | -| {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c set to non-null to indicate VariantNull | -| {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to null, to indicate that c is missing | -| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field d is stored as value | -| [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | null | null | null | null | null | Not an object | - -## Data Skipping - -Shredded columns are expected to store statistics in the same format as a normal Parquet column. -In general, the engine can only skip a row group or page if all rows in the `value` field are null, since it is possible for a `variant_get` expression to successfully cast a value from the `variant_value` to the target type. -For example, if `typed_value` is of type `int64`, then the string "123" might be contained in `value`, which would not be reflected in statistics, but could be retained by a filter like `where variant_get(col, "$.field", "long") = 123`. -If `value` is all-null, then the engine can prune pages or row groups based on `typed_value`. -This specification is not strict about what values may be stored in `value` rather than `typed_value`, so it is not safe to skip rows based on `typed_value` unless the corresponding `variant_value` column is all-null, or the engine has specific knowledge of the behavior of the writer that produced the shredded data. - -## Reconstructing a Variant - -It is possible to recover a full Variant value using a recursive algorithm, where the initial call is to `ConstructVariant` with the top-level fields, which are assumed to be null if they are not present in the schema. - -``` -# Constructs a Variant from `value`, `object`, `array` and `typed_value`. -# Only one of object, array and typed_value may be non-null. -def ConstructVariant(value, object, array, typed_value): - if object is null and array is null and typed_value is null and value is null: return VariantNull - if object is not null: - return ConstructObject(value, object) - elif array is not null: - return ConstructArray(array) - elif typed_value is not null: - return cast(typed_value as Variant) - else: - value - -# Construct an object from an `object` group, and a (possibly null) Variant value -def ConstructObject(value, object): - # If value is present and is not an Object, then the result is ambiguous. - assert(value is null or is_object(variant_value)) - # Null fields in the object are missing from the reconstructed Variant. - nonnull_object_fields = object.fields.filter(field -> field is not null) - all_keys = Union(value.keys, non_null_object_fields) - return VariantObject(all_keys.map { key -> - if key in object: (key, ConstructVariant(object[key].value, object[key].object, object[key].array, object[key].typed_value)) - else: (key, value[key]) - }) - -def ConstructArray(array): - newVariantArray = VariantArray() - for i in range(array.size): - newVariantArray.append(ConstructVariant(array[i].value, array[i].object, array[i].array, array[i].typed_value) -``` - -## Nested Parquet Example - -This section describes a more deeply nested example, using a top-level array as the shredding type. - -Below is a sample of JSON that would be fully shredded in this example. -It contains an array of objects, containing an `a` field shredded as an array, and a `b` field shredded as an integer. +Statistics for `typed_value` columns can be used for file, row group, or page skipping when `value` is always null (missing). -``` -[ - { - "a": [1, 2, 3], - "b": 100 - }, - { - "a": [4, 5, 6], - "b": 200 - } -] -``` - - -The corresponding Parquet schema with "a" and "b" as leaf types is: +When the corresponding `value` column is all nulls, all values must be the shredded `typed_value` field's type. +Because the type is known, comparisons with values of that type are valid. +`IS NULL`/`IS NOT NULL` and `IS NAN`/`IS NOT NAN` filter results are also valid. -``` -optional group variant_col { - required binary metadata; - optional binary value; - optional group array (LIST) { - repeated group list { - optional group element { - optional binary value; - optional group object { - optional group a { - optional binary value; - optional group array (LIST) { - repeated group list { - optional group element { - optional int64 typed_value; - optional binary value; - } - } - } - } - optional group b { - optional int64 typed_value; - optional binary value; - } - } - } - } - } -} -``` +Comparisons with values of other types are not necessarily valid and data should not be skipped. -In the above example schema, if "a" is an array containing a mix of integer and non-integer values, the engine will shred individual elements appropriately into either `typed_value` or `value`. -If the top-level Variant is not an array (for example, an object), the engine cannot shred the value and it will store it in the top-level `value`. -Similarly, if "a" is not an array, it will be stored in the `value` under "a". +Casting behavior for Variant is delegated to processing engines. +For example, the interpretation of a string as a timestamp may depend on the engine's SQL session time zone. -Consider the following example: +## Reconstructing a Variant -``` -[ - { - "a": [1, 2, 3], - "b": 100, - "c": "unexpected" - }, - { - "a": [4, 5, 6], - "b": 200 - }, - "not an object" -] +It is possible to recover a full Variant value using a recursive algorithm, where the initial call is to `construct_variant` with the top-level Variant group fields. + +```python +def construct_variant(metadata, value, typed_value): + """Constructs a Variant from value and typed_value""" + if typed_value is not null: + if isinstance(typed_value, dict): + # this is a shredded object + object_fields = { + name: construct_variant(metadata, field.value, field.typed_value) + for (name, field) in typed_value + } + + if value is not null: + # this is a partially shredded object + assert isinstance(value, VariantObject), "partially shredded value must be an object" + assert typed_value.keys().isdisjoint(value.keys()), "object keys must be disjoint" + + # union the shredded fields and non-shredded fields + return VariantObject(metadata, object_fields).union(VariantObject(metadata, value)) + + else: + return VariantObject(metadata, object_fields) + + elif isinstance(typed_value, list): + # this is a shredded array + assert value is null, "shredded array must not conflict with variant value" + + elements = [ + construct_variant(metadata, elem.value, elem.typed_value) + for elem in list(typed_value) + ] + return VariantArray(metadata, elements) + + else: + # this is a shredded primitive + assert value is null, "shredded primitive must not conflict with variant value" + + return primitive_to_variant(typed_value) + + elif value is not null: + return Variant(metadata, value) + + else: + # value is missing + return None + +def primitive_to_variant(typed_value): + if isinstance(typed_value, int): + return VariantInteger(typed_value) + elif isinstance(typed_value, str): + return VariantString(typed_value) + ... ``` -The second array element can be fully shredded, but the first and third cannot be. The contents of `variant_col.array[*].value` would be as follows: - -``` -[ - { "c": "unexpected" }, - NULL, - "not an object" -] -``` ## Backward and forward compatibility -Shredding is an optional feature of Variant, and readers must continue to be able to read a group containing only a `value` and `metadata` field. - -Any fields in the same group as `typed_value`/`value` that start with `_` (underscore) can be ignored. -This is intended to allow future backwards-compatible extensions. -In particular, the field names `_metadata_key_paths` and any name starting with `_spark` are reserved, and should not be used by other implementations. -Any extra field names that do not start with an underscore should be assumed to be backwards incompatible, and readers should fail when reading such a schema. +Shredding is an optional feature of Variant, and readers must continue to be able to read a group containing only `value` and `metadata` fields. Engines without shredding support are not expected to be able to read Parquet files that use shredding. -Since different files may contain conflicting schemas (e.g. a `typed_value` column with incompatible types in two files), it may not be possible to infer or specify a single schema that would allow all Parquet files for a table to be read. +Different files may contain conflicting schemas. +That is, files may contain different `typed_value` columns for the same Variant with incompatible types. +It may not be possible to infer or specify a single shredded schema that would allow all Parquet files for a table to be read without reconstructing the value as a Variant. From e1d2b74ce2d317d30480d16fa0aab6e4b8c36a09 Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 13:22:21 -0700 Subject: [PATCH 04/15] Trim the intro --- VariantShredding.md | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/VariantShredding.md b/VariantShredding.md index 764a9be2..36b5ea38 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -25,20 +25,12 @@ The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. Query engines encode each Variant value in a self-describing format, and store it as a group containing required `value` and `metadata` binary fields in Parquet. Since data is often partially homogenous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. -We refer to this process as **shredding**. -Each Parquet file remains fully self-describing, with no additional metadata required to read or fully reconstruct the Variant data from the file. -Combining shredding with a binary residual provides the flexibility to represent complex, evolving data with an unbounded number of unique fields while limiting the size of file schemas, and retaining the performance benefits of a columnar format. +This process is **shredding**. -This document focuses on the shredding semantics, Parquet representation, implications for readers and writers, as well as the Variant reconstruction. -For now, it does not discuss which fields to shred, user-facing API changes, or any engine-specific considerations like how to use shredded columns. -The approach builds upon the [Variant Binary Encoding](VariantEncoding.md), and leverages the existing Parquet specification. +Shredding enables the use of of Parquet's columnar representation for more compact data encoding, the use of column statistics for data skipping, and partial projections from Parquet's columnar layout. -Shredding allows a query engine to reap the full benefits of Parquet's columnar representation, such as more compact data encoding, min/max statistics for data skipping, and I/O and CPU savings from pruning unnecessary fields not accessed by a query (including the non-shredded Variant binary data). -Without shredding, any query that accesses a Variant column must fetch all bytes of the full binary buffer. -With shredding, readers can get nearly equivalent performance as in a relational (scalar) data model. - -For example, `SELECT variant_get(variant_event, '$.event_ts', 'timestamp') FROM tbl` only needs to access `event_ts`, and the file scan could avoid fetching the rest of the Variant value if this field was shredded into a separate column in the Parquet schema. -Similarly, for the query `SELECT * FROM tbl WHERE variant_get(variant_event, '$.event_type', 'string') = 'signup'`, the scan could first decode the shredded `event_type` column, and only fetch/decode the full Variant event value for rows that pass the filter. +For example, the query `SELECT variant_get(event, '$.event_ts', 'timestamp') FROM tbl` only needs to load field `event_ts`, and shredding can enable columnar projection that ignores the rest of the `event` Variant. +Similarly, for the query `SELECT * FROM tbl WHERE variant_get(event, '$.event_type', 'string') = 'signup'`, the `event_type` shredded column metadata can be used for skipping and to lazily load the rest of the Variant. ## Variant Metadata From 6f13efb6c382ecc3c5101e86f84a29deb18b6ced Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 13:35:45 -0700 Subject: [PATCH 05/15] Update the encoding to capture required/optional value. --- VariantEncoding.md | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/VariantEncoding.md b/VariantEncoding.md index a40429f6..7ec8870e 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -39,24 +39,34 @@ Another motivation for the representation is that (aside from metadata) each nes For example, in a Variant containing an Array of Variant values, the representation of an inner Variant value, when paired with the metadata of the full variant, is itself a valid Variant. This document describes the Variant Binary Encoding scheme. -The [Variant Shredding spec](VariantShredding.md) describes the details of shredding Variant values as typed Parquet columns. +The [Variant Shredding specification](VariantShredding.md) describes the details of shredding Variant values as typed Parquet columns. ## Variant in Parquet A Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`. -The Variant group must be annotated with the `VARIANT` logical type. -Both fields `value` and `metadata` are of type `binary`. -The `metadata` field is required and must be a valid Variant metadata, as defined below. -The `value` field is optional. -When present, the `value` field must be a valid Variant value, as defined below. -The `value` field may be null only when parts of the Variant value are shredded according to the Variant Shreedding spec. -This is the expected representation in Parquet: +* The Variant group must be annotated with the `VARIANT` logical type. +* Both fields `value` and `metadata` must be of type `binary` (called `BYTE_ARRAY` in the Parquet thrift definition). +* The `metadata` field is required and must be a valid Variant metadata, as defined below. +* The `value` field is required for unshredded Variant values. +* The `value` field is optional when parts of the Variant value are shredded according to the [Variant Shredding specification](VariantShredding.md). +* When present, the `value` field must be a valid Variant value, as defined below. +This is the expected unshredded representation in Parquet: + +``` +optional group variant_name (VARIANT) { + required binary metadata; + required binary value; +} +``` + +This is an example representation of a shredded Variant in Parquet: ``` -optional group variant_event (VARIANT) { +optional group shredded_variant_name (VARIANT) { required binary metadata; optional binary value; + optional int64 typed_value; } ``` From ab126d0f29cc6dc25e215a4adfb7fbf874b26fe1 Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 13:39:24 -0700 Subject: [PATCH 06/15] Remove unnecessary required. --- VariantShredding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VariantShredding.md b/VariantShredding.md index 36b5ea38..3ee6d73e 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -23,7 +23,7 @@ > **This specification is still under active development, and has not been formally adopted.** The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. -Query engines encode each Variant value in a self-describing format, and store it as a group containing required `value` and `metadata` binary fields in Parquet. +Query engines encode each Variant value in a self-describing format, and store it as a group containing `value` and `metadata` binary fields in Parquet. Since data is often partially homogenous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. This process is **shredding**. From d4afb1aea118f42d8b742735b36c8336bcfb1f2d Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 13:49:41 -0700 Subject: [PATCH 07/15] Use typed_value when there is a conflict. --- VariantShredding.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/VariantShredding.md b/VariantShredding.md index 3ee6d73e..69ed455b 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -65,16 +65,16 @@ The series of measurements `34, null, "n/a", 100` would be stored as: Both `value` and `typed_value` are optional fields used together to encode a single value. Values in the two fields must be interpreted according to the following table: -| `value` | `typed_value` | Meaning | -|----------|---------------|---------| -| null | null | The value is missing | +| `value` | `typed_value` | Meaning | +|----------|---------------|----------------------------------------------------------| +| null | null | The value is missing | | non-null | null | The value is present and may be any type, including null | -| null | non-null | The value is present and the shredded type | -| non-null | non-null | The value is present and a partially shredded object | +| null | non-null | The value is present and is the shredded type | +| non-null | non-null | The value is present and a partially shredded object | An object is _partially shredded_ when the `value` is an object and the `typed_value` is a shredded object. -If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `value`. +If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `typed_value`. ### Shredded Value Types From 3462b46256f3d758e2a4e59767f5bb28c050e92a Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 13:53:43 -0700 Subject: [PATCH 08/15] Minor updates. --- VariantShredding.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/VariantShredding.md b/VariantShredding.md index 69ed455b..18f1bbde 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -70,7 +70,7 @@ Values in the two fields must be interpreted according to the following table: | null | null | The value is missing | | non-null | null | The value is present and may be any type, including null | | null | non-null | The value is present and is the shredded type | -| non-null | non-null | The value is present and a partially shredded object | +| non-null | non-null | The value is present and is a partially shredded object | An object is _partially shredded_ when the `value` is an object and the `typed_value` is a shredded object. @@ -262,7 +262,7 @@ It is possible to recover a full Variant value using a recursive algorithm, wher ```python def construct_variant(metadata, value, typed_value): """Constructs a Variant from value and typed_value""" - if typed_value is not null: + if typed_value is not None: if isinstance(typed_value, dict): # this is a shredded object object_fields = { @@ -270,7 +270,7 @@ def construct_variant(metadata, value, typed_value): for (name, field) in typed_value } - if value is not null: + if value is not None: # this is a partially shredded object assert isinstance(value, VariantObject), "partially shredded value must be an object" assert typed_value.keys().isdisjoint(value.keys()), "object keys must be disjoint" @@ -283,7 +283,7 @@ def construct_variant(metadata, value, typed_value): elif isinstance(typed_value, list): # this is a shredded array - assert value is null, "shredded array must not conflict with variant value" + assert value is None, "shredded array must not conflict with variant value" elements = [ construct_variant(metadata, elem.value, elem.typed_value) @@ -293,11 +293,11 @@ def construct_variant(metadata, value, typed_value): else: # this is a shredded primitive - assert value is null, "shredded primitive must not conflict with variant value" + assert value is None, "shredded primitive must not conflict with variant value" return primitive_to_variant(typed_value) - elif value is not null: + elif value is not None: return Variant(metadata, value) else: From 012eb3c17016411d1b707fd3ae33e7ac40f84e8d Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 24 Oct 2024 14:32:06 -0700 Subject: [PATCH 09/15] Clarify cases where a value is required but missing. --- VariantShredding.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/VariantShredding.md b/VariantShredding.md index 18f1bbde..60829cae 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -46,7 +46,7 @@ Each `value` field may have an associated shredded field named `typed_value` tha For example, a Variant field, `measurement` may be shredded as long values by adding `typed_value` with type `int64`: ``` -optional group measurement (VARIANT) { +required group measurement (VARIANT) { required binary metadata; optional binary value; optional int64 typed_value; @@ -76,6 +76,9 @@ An object is _partially shredded_ when the `value` is an object and the `typed_v If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `typed_value`. +If a Variant is missing in a context where a value is required, readers must either fail or return a Variant null: basic type 0 (primitive) and physical type 0 (null). +For example, if a Variant is required (like `measurement` above) and both `value` and `typed_value` are null, the returned `value` must be `00` (Variant null). + ### Shredded Value Types Shredded values must use the following Parquet types: From ce706e05b96e21a5b65163537c3f914667baefd2 Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Thu, 21 Nov 2024 16:57:48 -0800 Subject: [PATCH 10/15] Updates from reviews. --- VariantEncoding.md | 6 ++-- VariantShredding.md | 76 +++++++++++++++++++++++++++------------------ 2 files changed, 48 insertions(+), 34 deletions(-) diff --git a/VariantEncoding.md b/VariantEncoding.md index 7ec8870e..ccc2d91e 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -70,7 +70,7 @@ optional group shredded_variant_name (VARIANT) { } ``` -There are no restrictions on the repetition of Variant groups (required, optional, or repeated). +The `VARIANT` annotation places no additional restrictions on the repetition of Variant groups, but repetition may be restricted by containing types (such as `MAP` and `LIST`). The Variant group name is the name of the Variant column. ## Metadata encoding @@ -401,7 +401,7 @@ The Decimal type contains a scale, but no precision. The implied precision of a | Object | `2` | A collection of (string-key, variant-value) pairs | | Array | `3` | An ordered sequence of variant values | -| Logical Type | Physical Type | Type ID | Equivalent Parquet Type | Binary format | +| Variant Logical Type | Variant Physical Type | Type ID | Equivalent Parquet Type | Binary format | |----------------------|-----------------------------|---------|-----------------------------|---------------------------------------------------------------------------------------------------------------------| | NullType | null | `0` | UNKNOWN | none | | Boolean | boolean (True) | `1` | BOOLEAN | none | @@ -439,7 +439,7 @@ This includes the dictionary key string values, the "short string" values, and t ## Object field ID order and uniqueness -For objects, field IDs and offsets must be listed in the order of the corresponding field names, sorted lexicographically. +For objects, field IDs and offsets must be listed in the order of the corresponding field names, sorted lexicographically (using unsigned byte ordering for UTF-8). Note that the field values themselves are not required to follow this order. As a result, offsets will not necessarily be listed in ascending order. The field values are not required to be in the same order as the field IDs, to enable flexibility when constructing Variant values. diff --git a/VariantShredding.md b/VariantShredding.md index 60829cae..d5793dea 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -27,9 +27,9 @@ Query engines encode each Variant value in a self-describing format, and store i Since data is often partially homogenous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. This process is **shredding**. -Shredding enables the use of of Parquet's columnar representation for more compact data encoding, the use of column statistics for data skipping, and partial projections from Parquet's columnar layout. +Shredding enables the use of Parquet's columnar representation for more compact data encoding, column statistics for data skipping, and partial projections. -For example, the query `SELECT variant_get(event, '$.event_ts', 'timestamp') FROM tbl` only needs to load field `event_ts`, and shredding can enable columnar projection that ignores the rest of the `event` Variant. +For example, the query `SELECT variant_get(event, '$.event_ts', 'timestamp') FROM tbl` only needs to load field `event_ts`, and if that column is shredded, it can be read by columnar projection without reading or deserializing the rest of the `event` Variant. Similarly, for the query `SELECT * FROM tbl WHERE variant_get(event, '$.event_type', 'string') = 'signup'`, the `event_type` shredded column metadata can be used for skipping and to lazily load the rest of the Variant. ## Variant Metadata @@ -43,6 +43,7 @@ All field names of a Variant, whether shredded or not, must be present in the me Variant values are stored in Parquet fields named `value`. Each `value` field may have an associated shredded field named `typed_value` that stores the value when it matches a specific type. +When `typed_value` is present, readers **must** reconstruct shredded values according to this specification. For example, a Variant field, `measurement` may be shredded as long values by adding `typed_value` with type `int64`: ``` @@ -65,12 +66,12 @@ The series of measurements `34, null, "n/a", 100` would be stored as: Both `value` and `typed_value` are optional fields used together to encode a single value. Values in the two fields must be interpreted according to the following table: -| `value` | `typed_value` | Meaning | -|----------|---------------|----------------------------------------------------------| -| null | null | The value is missing | -| non-null | null | The value is present and may be any type, including null | -| null | non-null | The value is present and is the shredded type | -| non-null | non-null | The value is present and is a partially shredded object | +| `value` | `typed_value` | Meaning | +|----------|---------------|-------------------------------------------------------------| +| null | null | The value is missing; only valid for shredded object fields | +| non-null | null | The value is present and may be any type, including null | +| null | non-null | The value is present and is the shredded type | +| non-null | non-null | The value is present and is a partially shredded object | An object is _partially shredded_ when the `value` is an object and the `typed_value` is a shredded object. @@ -105,7 +106,7 @@ Shredded values must use the following Parquet types: #### Primitive Types -Primitive values can be shredded using the equivalent Parquet primitive type from the table above for `typed_object`. +Primitive values can be shredded using the equivalent Parquet primitive type from the table above for `typed_value`. Unless the value is shredded as an object (see [Objects](#objects)), `typed_value` or `value` (but not both) must be non-null. @@ -116,8 +117,10 @@ Arrays can be shredded using a 3-level Parquet list for `typed_value`. If the value is not an array, `typed_value` must be null. If the value is an array, `value` must be null. -The list `element` must be a required group that contains optional `value` and `typed_value` fields. -The element's `value` field stores the element as Variant-encoded `binary` when the `typed_value` cannot represent it. +The list `element` must be a required group that contains `value` and `typed_value` fields. +The element's `value` field stores the element as Variant-encoded `binary` when the `typed_value` is not present or cannot represent it. +The `typed_value` field may be omitted when not shredding elements as a specific type. +When `typed_value` is omitted, `value` must be `required`. For example, a `tags` Variant may be shredded as a list of strings using the following definition: ``` @@ -135,7 +138,7 @@ optional group tags (VARIANT) { } ``` -All elements of an array must be non-null because `array` elements in a Variant cannot be missing. +All elements of an array must be present (not missing) because the `array` Variant encoding does not allow missing elements. That is, either `typed_value` or `value` (but not both) must be non-null. Null elements must be encoded in `value` as Variant null: basic type 0 (primitive) and physical type 0 (null). @@ -152,15 +155,16 @@ The series of `tags` arrays `["comedy", "drama"], ["horror", null], ["comedy", " Fields of an object can be shredded using a Parquet group for `typed_value` that contains shredded fields. +If the value is an object, `typed_value` must be non-null. If the value is not an object, `typed_value` must be null. -If the value is a partially shredded object, the `value` must not contain the shredded fields. -If shredded fields are present in the variant object, it is invalid and readers must either fail or use the shredded values. - Each shredded field in the `typed_value` group is represented as a required group that contains optional `value` and `typed_value` fields. The `value` field stores the value as Variant-encoded `binary` when the `typed_value` cannot represent the field. This layout enables readers to skip data based on the field statistics for `value` and `typed_value`. +If the value is a partially shredded object, the `value` must not contain the shredded fields. +If shredded fields are present in the variant object, it is invalid and readers must either fail or use the shredded values. + For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: ``` optional group event (VARIANT) { @@ -186,21 +190,31 @@ To encode a field that is present with a null value, the `value` must contain a The series of objects below would be stored as: -| Event object | `value` | `typed_value` | `typed_value.event_type.value` | `typed_value.event_type.typed_value` | `typed_value.event_ts.value` | `typed_value.event_ts.typed_value` | Notes | -|------------------------------------------------------------------------------------|-----------------------------------|---------------|--------------------------------|--------------------------------------|------------------------------|------------------------------------|-----------------------------------------------| -| `{"event_type": "noop", "event_ts": 1729794114937}` | null | non-null | null | `noop` | null | 1729794114937 | Fully shredded object | -| `{"event_type": "login", "event_ts": 1729794146402, "email": "user@example.com"}` | `{"email": "user@example.com"}` | non-null | null | `login` | null | 1729794146402 | Partially shredded object | -| `{"error_msg": "malformed: ..."}` | `{"error_msg", "malformed: ..."}` | null | | | | | Object with no shredding | -| `"malformed: not an object"` | `malformed: not an object` | null | | | | | Not an object (stored as Variant string) | -| `{"event_ts": 1729794240241, "click": "_button"}` | `{"click": "_button"}` | non-null | null | null | null | 1729794240241 | Field `event_type` is missing | -| `{"event_type": null, "event_ts": 1729794954163}` | null | non-null | `00` (field exists, is null) | null | null | 1729794954163 | Field `event_type` is present and is null | -| null | `00` (null) | null | | | | | Object/value is null | -| missing | null | null | | | | | Object/value is missing | -| INVALID | `{"event_type": "login"}` | non-null | null | `login` | null | 1729795057774 | INVALID: Shredded field is present in `value` | +| Event object | `value` | `typed_value` | `typed_value.event_type.value` | `typed_value.event_type.typed_value` | `typed_value.event_ts.value` | `typed_value.event_ts.typed_value` | Notes | +|------------------------------------------------------------------------------------|-----------------------------------|---------------|--------------------------------|--------------------------------------|------------------------------|------------------------------------|--------------------------------------------------| +| `{"event_type": "noop", "event_ts": 1729794114937}` | null | non-null | null | `noop` | null | 1729794114937 | Fully shredded object | +| `{"event_type": "login", "event_ts": 1729794146402, "email": "user@example.com"}` | `{"email": "user@example.com"}` | non-null | null | `login` | null | 1729794146402 | Partially shredded object | +| `{"error_msg": "malformed: ..."}` | `{"error_msg", "malformed: ..."}` | non-null | null | null | null | null | Object with all shredded fields missing | +| `"malformed: not an object"` | `malformed: not an object` | null | | | | | Not an object (stored as Variant string) | +| `{"event_ts": 1729794240241, "click": "_button"}` | `{"click": "_button"}` | non-null | null | null | null | 1729794240241 | Field `event_type` is missing | +| `{"event_type": null, "event_ts": 1729794954163}` | null | non-null | `00` (field exists, is null) | null | null | 1729794954163 | Field `event_type` is present and is null | +| `{"event_type": "noop", "event_ts": "2024-10-24"` | null | non-null | null | `noop` | `"2024-10-24"` | null | Field `event_ts` is present but not a timestamp | +| `{ }` | null | non-null | null | null | null | null | Object is present but empty | +| null | `00` (null) | null | | | | | Object/value is null | +| missing | null | null | | | | | Object/value is missing | +| INVALID | `{"event_type": "login"}` | non-null | null | `login` | null | 1729795057774 | INVALID: Shredded field is present in `value` | +| INVALID | `"a"` | non-null | null | `login` | null | 1729795057774 | INVALID: `typed_value` is present for non-object | +| INVALID | `02 00` (object with 0 fields) | null | | | | | INVALID: `typed_value` is null for object | + +Invalid cases in the table above must not be produced by writers. +Readers must return an object when `typed_value` is non-null containing the shredded fields. +If `typed_value` is null and `value` is an object, readers may read the encoded object but are not required to do so. + +Readers can assume that a value is not an object if `typed_value` is null and that `typed_value` field values are correct; that is, readers do not need to read the `value` column if `typed_value` fields satisfy the required fields. ## Nesting -The `typed_value` associated with any Variant `value` field can be any shredded type according to the rules above. +The `typed_value` associated with any Variant `value` field can be any shredded type, as shown in the sections above. For example, the `event` object above may also shred sub-fields as object (`location`) or array (`tags`). @@ -258,12 +272,12 @@ Comparisons with values of other types are not necessarily valid and data should Casting behavior for Variant is delegated to processing engines. For example, the interpretation of a string as a timestamp may depend on the engine's SQL session time zone. -## Reconstructing a Variant +## Reconstructing a Shredded Variant -It is possible to recover a full Variant value using a recursive algorithm, where the initial call is to `construct_variant` with the top-level Variant group fields. +It is possible to recover an unshredded Variant value using a recursive algorithm, where the initial call is to `construct_variant` with the top-level Variant group fields. ```python -def construct_variant(metadata, value, typed_value): +def construct_variant(metadata: Metadata, value: Variant, typed_value: Any) -> Variant: """Constructs a Variant from value and typed_value""" if typed_value is not None: if isinstance(typed_value, dict): @@ -307,7 +321,7 @@ def construct_variant(metadata, value, typed_value): # value is missing return None -def primitive_to_variant(typed_value): +def primitive_to_variant(typed_value: Any): Variant: if isinstance(typed_value, int): return VariantInteger(typed_value) elif isinstance(typed_value, str): From 5cdd68259108f22fabcf4ab7b825c14231b5f9e0 Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Tue, 26 Nov 2024 11:57:55 -0800 Subject: [PATCH 11/15] Update from latest review comments. --- VariantEncoding.md | 12 +++++++---- VariantShredding.md | 52 +++++++++++++++++++++++---------------------- 2 files changed, 35 insertions(+), 29 deletions(-) diff --git a/VariantEncoding.md b/VariantEncoding.md index ccc2d91e..299cf653 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -39,6 +39,8 @@ Another motivation for the representation is that (aside from metadata) each nes For example, in a Variant containing an Array of Variant values, the representation of an inner Variant value, when paired with the metadata of the full variant, is itself a valid Variant. This document describes the Variant Binary Encoding scheme. +Variant fields can also be _shredded_. +Shredding refers to extracting some elements of the variant into separate columns for more efficient extraction/filter pushdown. The [Variant Shredding specification](VariantShredding.md) describes the details of shredding Variant values as typed Parquet columns. ## Variant in Parquet @@ -47,9 +49,8 @@ A Variant value in Parquet is represented by a group with 2 fields, named `value * The Variant group must be annotated with the `VARIANT` logical type. * Both fields `value` and `metadata` must be of type `binary` (called `BYTE_ARRAY` in the Parquet thrift definition). -* The `metadata` field is required and must be a valid Variant metadata, as defined below. -* The `value` field is required for unshredded Variant values. -* The `value` field is optional when parts of the Variant value are shredded according to the [Variant Shredding specification](VariantShredding.md). +* The `metadata` field is `required` and must be a valid Variant metadata, as defined below. +* The `value` field must be annotated as `required` for unshredded Variant values, or `optional` if parts of the value are [shredded](VariantShredding.md) as typed Parquet columns. * When present, the `value` field must be a valid Variant value, as defined below. This is the expected unshredded representation in Parquet: @@ -473,7 +474,7 @@ To maximize compatibility with readers that can process JSON but not Variant, th |---------------|-----------|----------------------------------------------------------|--------------------------------------| | Null type | null | `null` | `null` | | Boolean | boolean | `true` or `false` | `true` | -| Exact Numeric | number | Digits in fraction must match scale, no exponent | `34`, 34.00 | +| Exact Numeric | number | Digits in fraction must match scale, no exponent | `34`, `34.00` | | Float | number | Fraction must be present | `14.20` | | Double | number | Fraction must be present | `1.0` | | Date | string | ISO-8601 formatted date | `"2017-11-16"` | @@ -484,3 +485,6 @@ To maximize compatibility with readers that can process JSON but not Variant, th | Array | array | | `[34, "abc", "2017-11-16]` | | Object | object | | `{"id": 34, "data": "abc"}` | +Notes: + +* For timestamp and timestampntz, values must use microsecond precision and trailing 0s are required diff --git a/VariantShredding.md b/VariantShredding.md index d5793dea..5a1bfe25 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -84,25 +84,25 @@ For example, if a Variant is required (like `measurement` above) and both `value Shredded values must use the following Parquet types: -| Variant Type | Equivalent Parquet Type | -|-----------------------------|------------------------------| -| boolean | BOOLEAN | -| int8 | INT(8, signed=true) | -| int16 | INT(16, signed=true) | -| int32 | INT32 / INT(32, signed=true) | -| int64 | INT64 / INT(64, signed=true) | -| float | FLOAT | -| double | DOUBLE | -| decimal4 | DECIMAL(precision, scale) | -| decimal8 | DECIMAL(precision, scale) | -| decimal16 | DECIMAL(precision, scale) | -| date | DATE | -| timestamp | TIMESTAMP(true, MICROS) | -| timestamp without time zone | TIMESTAMP(false, MICROS) | -| binary | BINARY | -| string | STRING | -| array | LIST; see Arrays below | -| object | GROUP; see Objects below | +| Variant Type | Equivalent Parquet Type | +|-----------------------------|-----------------------------------| +| boolean | BOOLEAN | +| int8 | INT(8, signed=true) | +| int16 | INT(16, signed=true) | +| int32 | INT32 / INT(32, signed=true) | +| int64 | INT64 / INT(64, signed=true) | +| float | FLOAT | +| double | DOUBLE | +| decimal4 | INT32 / DECIMAL(precision, scale) | +| decimal8 | INT64 / DECIMAL(precision, scale) | +| decimal16 | DECIMAL(precision, scale) | +| date | DATE | +| timestamp | TIMESTAMP(true, MICROS) | +| timestamp without time zone | TIMESTAMP(false, MICROS) | +| binary | BINARY | +| string | STRING | +| array | LIST; see Arrays below | +| object | GROUP; see Objects below | #### Primitive Types @@ -112,12 +112,13 @@ Unless the value is shredded as an object (see [Objects](#objects)), `typed_valu #### Arrays -Arrays can be shredded using a 3-level Parquet list for `typed_value`. +Arrays can be shredded by using a 3-level Parquet list for `typed_value`. If the value is not an array, `typed_value` must be null. If the value is an array, `value` must be null. -The list `element` must be a required group that contains `value` and `typed_value` fields. +The list `element` must be a required group. +The `element` group can contain `value` and `typed_value` fields. The element's `value` field stores the element as Variant-encoded `binary` when the `typed_value` is not present or cannot represent it. The `typed_value` field may be omitted when not shredding elements as a specific type. When `typed_value` is omitted, `value` must be `required`. @@ -183,12 +184,12 @@ optional group event (VARIANT) { } ``` -The group for each named field must be required. +The group for each named field must use repetition level `required`. A field's `value` and `typed_value` are set to null (missing) to indicate that the field does not exist in the variant. To encode a field that is present with a null value, the `value` must contain a Variant null: basic type 0 (primitive) and physical type 0 (null). -The series of objects below would be stored as: +The table below shows how the series of objects in the first column would be stored: | Event object | `value` | `typed_value` | `typed_value.event_type.value` | `typed_value.event_type.typed_value` | `typed_value.event_ts.value` | `typed_value.event_ts.typed_value` | Notes | |------------------------------------------------------------------------------------|-----------------------------------|---------------|--------------------------------|--------------------------------------|------------------------------|------------------------------------|--------------------------------------------------| @@ -334,7 +335,8 @@ def primitive_to_variant(typed_value: Any): Variant: Shredding is an optional feature of Variant, and readers must continue to be able to read a group containing only `value` and `metadata` fields. -Engines without shredding support are not expected to be able to read Parquet files that use shredding. -Different files may contain conflicting schemas. +Engines that do not write shredded values must be able to read shredded values according to this spec or must fail. + +Different files may contain conflicting shredding schemas. That is, files may contain different `typed_value` columns for the same Variant with incompatible types. It may not be possible to infer or specify a single shredded schema that would allow all Parquet files for a table to be read without reconstructing the value as a Variant. From 2858477a8631b4d4ab3e6bef4c47f81f6fa4822d Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Tue, 10 Dec 2024 13:58:43 -0800 Subject: [PATCH 12/15] Handle infinities and NaN in the JSON encoding. --- VariantEncoding.md | 1 + 1 file changed, 1 insertion(+) diff --git a/VariantEncoding.md b/VariantEncoding.md index 299cf653..eb8ea93e 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -488,3 +488,4 @@ To maximize compatibility with readers that can process JSON but not Variant, th Notes: * For timestamp and timestampntz, values must use microsecond precision and trailing 0s are required +* For float and double, infinities and not a number values are encoded as strings: `"Infinity"`, `"-Infinity"`, and `"NaN"` From 4f5611d01e010029e8dc933146356758efb62a7d Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Tue, 10 Dec 2024 14:05:26 -0800 Subject: [PATCH 13/15] Specify columns accessed by name, not position. --- VariantShredding.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/VariantShredding.md b/VariantShredding.md index 5a1bfe25..57bcac2c 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -54,6 +54,8 @@ required group measurement (VARIANT) { } ``` +The Parquet columns used to store variant metadata and values must be accessed by name, not by position. + The series of measurements `34, null, "n/a", 100` would be stored as: | Value | `metadata` | `value` | `typed_value` | From 413139a9d1be9d8d61309089c5ab03f3d6f13f7a Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Tue, 10 Dec 2024 14:21:24 -0800 Subject: [PATCH 14/15] Clarifications. --- VariantShredding.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/VariantShredding.md b/VariantShredding.md index 57bcac2c..5fc0df06 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -160,13 +160,15 @@ Fields of an object can be shredded using a Parquet group for `typed_value` that If the value is an object, `typed_value` must be non-null. If the value is not an object, `typed_value` must be null. +Readers can assume that a value is not an object if `typed_value` is null and that `typed_value` field values are correct; that is, readers do not need to read the `value` column if `typed_value` fields satisfy the required fields. Each shredded field in the `typed_value` group is represented as a required group that contains optional `value` and `typed_value` fields. The `value` field stores the value as Variant-encoded `binary` when the `typed_value` cannot represent the field. This layout enables readers to skip data based on the field statistics for `value` and `typed_value`. -If the value is a partially shredded object, the `value` must not contain the shredded fields. -If shredded fields are present in the variant object, it is invalid and readers must either fail or use the shredded values. +The `value` column of a partially shredded object must never contain fields represented by the Parquet columns in `typed_value` (shredded fields). +If shredded fields are present in the variant object, it is invalid and readers must either fail or use the shredded field values. +Readers are not required to check for conflicts between `value` and the shredded fields in `typed_value` columns. For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: ``` @@ -213,8 +215,6 @@ Invalid cases in the table above must not be produced by writers. Readers must return an object when `typed_value` is non-null containing the shredded fields. If `typed_value` is null and `value` is an object, readers may read the encoded object but are not required to do so. -Readers can assume that a value is not an object if `typed_value` is null and that `typed_value` field values are correct; that is, readers do not need to read the `value` column if `typed_value` fields satisfy the required fields. - ## Nesting The `typed_value` associated with any Variant `value` field can be any shredded type, as shown in the sections above. From 9c283bcdfc323be28a32171165f496441e8c157a Mon Sep 17 00:00:00 2001 From: Ryan Blue Date: Wed, 8 Jan 2025 14:06:13 -0800 Subject: [PATCH 15/15] Update reader requirements to be more strict. --- VariantShredding.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/VariantShredding.md b/VariantShredding.md index 5fc0df06..046a4937 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -77,9 +77,10 @@ Values in the two fields must be interpreted according to the following table: An object is _partially shredded_ when the `value` is an object and the `typed_value` is a shredded object. -If both fields are non-null and either is not an object, the value is invalid. Readers must either fail or return the `typed_value`. +When `typed_value` is non-null, its value must always be used when reading. +Writers must not produce data where both `value` and `typed_value` are non-null, unless the Variant value is an object. -If a Variant is missing in a context where a value is required, readers must either fail or return a Variant null: basic type 0 (primitive) and physical type 0 (null). +If a Variant is missing in a context where a value is required, readers must return a Variant null (`00`): basic type 0 (primitive) and physical type 0 (null). For example, if a Variant is required (like `measurement` above) and both `value` and `typed_value` are null, the returned `value` must be `00` (Variant null). ### Shredded Value Types @@ -167,8 +168,7 @@ The `value` field stores the value as Variant-encoded `binary` when the `typed_v This layout enables readers to skip data based on the field statistics for `value` and `typed_value`. The `value` column of a partially shredded object must never contain fields represented by the Parquet columns in `typed_value` (shredded fields). -If shredded fields are present in the variant object, it is invalid and readers must either fail or use the shredded field values. -Readers are not required to check for conflicts between `value` and the shredded fields in `typed_value` columns. +The values stored in shredded fields must always be used when reading. For example, a Variant `event` field may shred `event_type` (`string`) and `event_ts` (`timestamp`) columns using the following definition: ``` @@ -208,12 +208,11 @@ The table below shows how the series of objects in the first column would be sto | null | `00` (null) | null | | | | | Object/value is null | | missing | null | null | | | | | Object/value is missing | | INVALID | `{"event_type": "login"}` | non-null | null | `login` | null | 1729795057774 | INVALID: Shredded field is present in `value` | -| INVALID | `"a"` | non-null | null | `login` | null | 1729795057774 | INVALID: `typed_value` is present for non-object | +| INVALID | `"a"` | non-null | null | null | null | null | INVALID: `typed_value` is present for non-object | | INVALID | `02 00` (object with 0 fields) | null | | | | | INVALID: `typed_value` is null for object | Invalid cases in the table above must not be produced by writers. Readers must return an object when `typed_value` is non-null containing the shredded fields. -If `typed_value` is null and `value` is an object, readers may read the encoded object but are not required to do so. ## Nesting