Can data be appended to a struct while maintaining backwards compatibility?

[randyK-HH]

If a schema has been created and compiled using a union of simple structs, is it possible (in the future) to add new fields to those structs while maintaining backwards compatibility with past versions? I know that fields are easily added and deprecated in tables.

As an example, take the following simplified schema:

struct DataType1
{
    my_data        : [uint32:4];
}

struct DataType2
{
    my_data        : [uint32:16];
}

struct DataType3
{
    my_data        : [uint32:64];
}

struct DataType4
{
    field1 : uint8;
    field2 : uint8;
    field3 : uint16;
    field4 : uint32;
}

union DataStructs
{
    data_1 : DataType1,
    data_2 : DataType2,
    data_3 : DataType3,
    data_4 : DataType4,
}

table TableData {

    data_structs     : [DataStructs];           // Collection of unionized data structs

    raw_data         : [uint8];                 // Buffer for raw data
}

Can I add new fields (field5, field6) to DataType4 struct, recompile the schema, and expect the newly generated library files to be able to properly read serialized buffers generated with the old schema? Further, can I add new structs to the DataStructs union while maintaining backwards compatibility?

[mikkelfj]

No not really, but in some special cases it might accidentally work:
Structs are officially immutable in schema, but fields can be deprecated but still take up space.

Now, if you change the meaning of a field of same size, or repurpose a deprecated field of same size, then it can work but all parties must sort of either agree, or not look at the field at all - in which case access to other fields will be safe.

Now that is not what you ask. So, if you add fields later, at the end of a struct: You can compare this to how this would work in the C programming language: it can be work, but not if you embed the struct in another struct. In that case you must use pointers. The Flatbuffer equivalent is to use a table instead of a parent struct. So a table pointing to a growing struct will not cause damage, at least not in C.
I think it will probably also work in other languages, but since it is not officially allowed, it is not recommended, and it might not work (backwards compatible) depending on how a given language implements struct access.

So my advice would be to just not do it, and use tables instead. Or add reserved fields that you define later with same type.

As I am about to post this, I see you have edited the post, so if you only edited the end, this should be answered, except for unions:
You can always add new union types later up to the 255 variant limit.

[randyK-HH]

This and your answers below make sense. Thanks! Just wanted to get an informed opinion.

[mikkelfj]

As to struct field deprecation, it may be that that feature itself has been deprecated in the main FlatBuffer repo, I do not recall, but it was possible when I implemented it in C, and it is still supported in C, in addition to deprecating table fields.

[mikkelfj]

So explicitly in the case of your schema, you use a wrapping union around the struct you want to modify. This works almost like if you used a parent table reference to the struct and is thus safe in C at least. However in this case it would be obvious to create a new variant of the same struct. In C you could just cast one type to the other if indeed they are identical except the end.

[mikkelfj]

Many answerrs ... on adding a new union type, that wouldn't be fully backwards compatible, it would be safe, but old readers would not recognize a new union type.

[mikkelfj]

Just for completeness: I don't think extending a top-level struct would break any language implementation because it would just add extra data in the buffer in a place where padding might otherwise appear. So only verifiers that insist on zero filled padding would be broken, and I am not aware of such verifiers. Still not recommended, but could work if you are in a bind.