serde_yaml::Value does not perfectly round trip serialize.

[doug-q]

In paticular https://docs.rs/serde_yaml/latest/serde_yaml/value/struct.TaggedValue.html is deserialized as a Mapping. Observe this by uncommenting the generator in crate::proptest::any_serde_value (currently in review #981).

Ideas:

• use serde_json::Value for our "opaque" representation in CustomSerialized, CustomTypeArg, etc. we already use serde_json::Value for metadata.
• Fail to serialize TaggedValues.
• Try to fix the round trip.

[zrho]

What is the current state of discussion with respect to YAML? If we are moving to JSON (or at least away from YAML) anyway, there would be nothing to do for this issue.

Tagged values are a really nice feature of YAML that is unfortunately missing in JSON. Therefore JSON formats always have to make up their own encoding of tags, and there isn't a universally agreed upon way to do this. Msgpack has extensions, but they are not quite the same as tagged values, and there does not appear to be an standardised extension for tagged values. Therefore msgpack would do a similar encoding as JSON.

The Rust ecosystem of YAML libraries is in an unfortunate state. In particular, the serde_yaml crate is no longer maintained, marked as deprecated on crates.io and the repository is archived.

There is an additional option: Use a custom Value type which serialises and deserialises cleanly to JSON and msgpack, but otherwise maps more directly onto the sort of data that we deal with. I might, for instance, want to have metadata consisting of a type, or type scheme, or type row, etc.

[doug-q]

To be clear, the issue here is that the rust types CustomSerialized, CustomTypeArg, OpDef, OperationDeclaration store have fields containing values of type serde_yaml::Value. These values will not round round-trip-serialise through json, at least because of the TaggedValue construct of serde_yaml.

I suggest that we change to using serde_json::Value for all of these fields, so that we are able to have round-trip-serialisation through json. This will prevent confusing breakage for users.

What is the current state of discussion with respect to YAML? If we are moving to JSON (or at least away from YAML) anyway, there would be nothing to do for this issue.

I do not think it's settled, we do not have any declaritive-spec users yet I don't think. My view is that we should continue to accept yaml, possibly gated by a feature flag, and immediately convert to json on ingress, failing if there are any TaggedValues. My paragraph above explains why, even if we were to use no yaml, there is work to do for this issue.

Tagged values are a really nice feature of YAML that is unfortunately missing in JSON. Therefore JSON formats always have to make up their own encoding of tags, and there isn't a universally agreed upon way to do this. Msgpack has extensions, but they are not quite the same as tagged values, and there does not appear to be an standardised extension for tagged values. Therefore msgpack would do a similar encoding as JSON.

Agree, but we specify everything with pydantic and ensure we match that specification with serde annotations, so I think this is not an issue for us.

The Rust ecosystem of YAML libraries is in an unfortunate state. In particular, the serde_yaml crate is no longer maintained, marked as deprecated on crates.io and the repository is archived.
agree, very annoying.

There is an additional option: Use a custom Value type which serialises and deserialises cleanly to JSON and msgpack, but otherwise maps more directly onto the sort of data that we deal with. I might, for instance, want to have metadata consisting of a type, or type scheme, or type row, etc.

I think you mean Value type as in something analagous to serde_json::Value or serde_yaml::Value. (We also have ops::custom::Value :) ). I suggest that serde_json::Value is exactly this type you describe. It maps directly onto our various serialisable data types via serde. Perhaps I have misunderstood though?