Questions about the GEDZIP format

[dthaler]

The GEDZIP format is in the FamilySearch GEDCOM 7.0.x spec, but I have a number of questions.

1. Can a GEDZIP file contain a GEDCOM 5.5.1 file as its gedcom.ged? (I would argue this would be useful.)
2. Can a GEDZIP file contain a GEDCOM 5.5.1 file as another local file? (I would argue this would be useful.)
3. Must all GEDCOM files (both gedcom.ged and other local files) in the zip archive use the same GEDCOM version? (I would argue the answer should be No.)
4. If the answer to 2 is Yes, then must all GEDCOM files in the zip archive use the same charset? (I would argue for the flexibility of No.)
5. The GEDCOM file contains a version number, but is it possible to version the GEDZIP format in the future or is the current set of requirements fixed for all time? For example in issue Media files called gedcom.ged incompatible with GEDZIP format #12 there was some discussion that could have resulted in a change, so I imagine there could be other questions in the future.

[tychonievich]

1. Can a GEDZIP file contain a GEDCOM 5.5.1 file as its gedcom.ged? (I would argue this would be useful.)

We discussed this during development and intentionally did not answer it. On the plus side, there is value in GEDZIP 5.5.1. On the minus side, some members of the steering committee felt that GEDZIP was 7.0's biggest selling point and didn't want people to use GEDZIP without using 7.0.

I think as worded, it is reasonable to say that the version of the GEDZIP is equal to the version of gedcom.ged: we document "The FamilySearch GEDZIP 7.0 file format" which has a FamilySearch GEDCOM 7.0 data stream inside it; by extension, it is reasonable to assume that a "The FamilySearch GEDZIP 5.5.1 file format" would have GEDCOM 5.5.1 inside it.

However, GEDZIP is defined in terms of 7.0 structures. While 5.5.1 has a similar structure (FILE), that structure is under-specified and can be taken to require absolute paths. To let GEDZIP work with 5.5.1 would require at least interpreting, if not editing, 5.5.1.

Pre-5.5.1 there is no FILE structure, so GEDZIP is not useful.

[dthaler]

Underspecification is a recipe for lack of interoperability, and for some other organization to define profiles that enable interoperability. I would prefer the main spec be explicit, even if it requires gedcom 7.0. Based on your answer to Q2, a workaround would be to include earlier gedcoms in the zip file, and just include a v7 gedcom.ged that contains only MULTIMEDIA_RECORDs pointer to the other gedcom's. This would seem to be legal already, and so if I understand correctly, it one can already use GEDZIP without using 7.0 for the actual genealogical data based on the Q2 answer.

[tychonievich]

I agree: we should be specific.

Having .gdz → 7.0 gedcom.ged → OBJE.FILE → 5.5.1 something.ged is legal, but doesn't fix the ambiguity problem with trying to put 5.5.1 inside GEDZIP.

In 5.5.1 the FILE <MULTIMEDIA_FILE_REFERENCE> construct requires a 1–30-character line value that is

A complete local or remote file reference to the auxiliary data to be linked to the GEDCOM context. Remote reference would include a network address where the multimedia data may be obtained.

If we want a 5.5.1 file inside a GEDZIP, either the 5.5.1 cannot contain any OBJE.FILE or we must clarify what this means in the GEDZIP context. The word "complete" is particularly troubling as it can reasonably be interpreted to mean "absolute".

I think such a clarification is doable, but we'd need to do it if we expected the 5.5.1's attachments to be reachable. Which means we'd have oi explicitly define GEDZIP for GEDCOM 5.5.1.

[clarkegj]

FamilySearch GEDCOM GEDZIP should only including Version 7+ .ged files in the package. I don't think we want to introduced the conversion of packaging an outdated GEDCOM file in a .gdz package. The definition, I thought was a .gdz file only included one (and only one) FamilySearch .ged file and included local files. If someone want to use the GEDZIP packaging they need to upgrade to Version 7. GEDZIP is a feature of only Version 7 and above. We need to provide all the reasons we can to upgrade to Version 7. Looking forward to discussing.

[tychonievich]

2. Can a GEDZIP file contain a GEDCOM 5.5.1 file as another local file? (I would argue this would be useful.)
3. Must all GEDCOM files (both gedcom.ged and other local files) in the zip archive use the same GEDCOM version? (I would argue the answer should be No.)
4. If Remove patch number from GEDC.VERS payload #2 is Yes, then must all GEDCOM files in the zip archive use the same charset? (I would argue for the flexibility of No.)

There is no limitation on files included alongside gedcom.ged except that they are linked from a g7:FILE in gedcom.ged

[tychonievich]

5. The GEDCOM file contains a version number, but is it possible to version the GEDZIP format in the future or is the current set of requirements fixed for all time? For example in issue Media files called gedcom.ged incompatible with GEDZIP format #12 there was some discussion that could have resulted in a change, so I imagine there could be other questions in the future.

The current numbering implies that GEDZIP and GEDCOM version numbers will be kept in sync. We could break that in the future, but I currently don't see a need to do so prior to a major release. Minor releases have to be backwards compatible, so all GEDZIP 7.x will have the same limitations on included files and permit files with just gedcom.ged that as GEDZIP 7.0 has.

My instinct is that we defer this question in the application/vnd.familysearch.gedzip+zip media type filing, not mentioning version numbers at all there.

[dthaler]

in the text/vnd.familysearch.gedcom application, we allow a version to be used in an Accept header to request a gedcom in a given version. Deferring the question means it would not be possible to request a gedzip in a given version.

[tychonievich]

I see four possible futures that can be read as consistent with the current 7.0 spec:

1. We define a different version number for GEDZIP. Thus, we might have GEDCOM v7.0 inside GEDZIP v7.1. The Accept header would presumably want to give both version numbers.

2. We define "GEDZIP version X means the contained gedcom.ged is version X". This means either

   1. We only change GEDZIP in fully compatible ways, so that a 7.0 GEDZIP parser can correctly parse all future GEDZIPs.

   2. We never change GEDZIP again. It is what it is.

   3. We let later GEDCOM versions use new features of GEDZIP: if you have a 7.0 GEDCOM file but GEDZIP 7.1 introduces some new feature, you either send it as a 7.1 GEDCOM file or don't use that feature.

I'm not sure which of these I like best. I'll need some time to think it over.

[tychonievich]

Discussed in steering committee meeting 2021-10-12
Decided we like gedcom.ged version == gedzip version.
This means that if gedzip changes in 7.1, we'd only use those with FamilySearch GEDCOM 7.1 files.

[dthaler]

@fisharebest do your source GEDCOM files contain any OBJE.FILE references?

[fisharebest]

Good question. Some of these GEDCOM files do indeed contain OBJE.FILE references - but I don't have the actual files.

Luckily(!), none of these OBJE.FILE references are to nested GEDCOM files :-)

[dthaler]

So that begs two more questions about the GEDZIP format:
6. Is it legal for the gedcom.ged file to have an OBJE.FILE local reference that is missing from the zip archive?
7. Is it legal for another local gedcom file (other than gedcom.ged) to have an OBJE.FILE local reference that is missing from the zip archive?

[dthaler]

For completeness, I might also ask:
8. Is it legal for another local gedcom file to have an OBJE.FILE local reference back to the main gedcom.ged file? I think the answer is yes. (This question becomes even more important if a future version moves other local files to a subfolder...)

[tychonievich]

6. As the 7.0 spec is written, no: local files cannot be missing. It defines a GEDZIP to have "an entry for each local file FILE structure in gedcom.ged." You'd have to convert any missing reference to a web-accessible file instead (e.g. http://example.com/file-not-found/OLD_FILE_NAME.TXT)
7. As the 7.0 spec is written, there are no constraints at all on files other than gedcom.ged, including on the links of other GEDCOM files.
8. As the 7.0 spec is written, there are no constraints at all on files other than gedcom.ged, including on the links of other GEDCOM files.

[dthaler]

Since it was decided that the GEDZIP version and gedcom.ged file version
must match, then the version parameter in the media type can be used to
specify how to interpret the GEDZIP format.

But since .gdz files come without a media type, we still need to specify
how to interpret a .gdz file when you don't know if it's 7.0 or some
future version.

So either the main file must always be called gedcom.ged in the base directory,
or future parsers will have to have a more complex algorithm to detect the GEDCOM version.

[dthaler]

Either:
a) We require gedcom.ged to always be in the same place with the same name in all future GEDZIP versions, or
b) We would need to specify a version detection algorithm for GEDZIP if that ever changed.

The choice of a vs b does affect the wording in the existing media type request's version parameter, so we should probably choose now.