OVERFLOW reporting in other output formats is undefined

[jwfraustro]

The reporting of overflow is ambiguous and undefined for a FORMAT/RESPONSEFORMAT other than VOTable. This line:

Reporting of overflow depends on the output format and is described in DALI.

would imply to the reader that the method to report an overflow in a format other than VOTable should be found in DALI. The TAP specification specifically states the following:

A TAP service must support VOTable as an output format, should support CSV and TSV output, and may support other
formats.

Which would imply that a method for CSV and TSV should also be defined, with the emphasis that the SHOULD implies a strong recommendation to support the formats (and their implementation details), and "other formats" left ambiguous, which is no matter. Indeed, the correct method to report an overflow in VOTable is defined in DALI:

https://github.com/ivoa-std/DALI/blob/c2cc2520f403f50f69a4f2b46ec523564089d807/DALI.tex#L1387-L1419

however, no other reporting methods for any other formats are described.

What is frustrating is that this implementation detail was previously explained in TAP 1.0, as shown here:

If the output format is VOTable, section 2.9.1 describes the method by which the overflow is reported. No method of reporting an overflow is defined for formats other than VOTable.

The removal of the second sentence here removed some rather useful information, and created the current effect-- how would a reader know that there is no other method for reporting OVERFLOW, period?

[msdemlei]

On Mon, Nov 13, 2023 at 10:19:55AM -0800, Joshua Fraustro wrote: The removal of the second sentence here removed some rather useful information, and created the current effect-- how would a reader know that there is no other method for reporting `OVERFLOW`, period?

Fair criticism. I suppose DALI should say that much now. Or, perhaps preferably, define a method that would work for "most" output formats. An obvious and trivial one would be to say that implementations must add a ".truncated" in front of the canonical extension of the file name returned (as transmitted in a content-disposition header). I frankly suspect that would be the most robust technique, and the one that our users would like the best, but I have the gut feeling that there might be hidden traps. An alternative would be a bespoke http header; I've not checked the pertinent RFCs whether that already exists. The one thing I'd try to avoid would be to rely on in-band signalling as in VOTable. Sure, for provenance it's nice if the document records that there's an incomplete result, but frankly, when someone commits a metadata massacre (such as using CSV or TSV), I suppose they won't worry about missing such a minor detail in the in-document metadata -- and anyway, I don't see us invent per-format tricks.

[pdowler]

http headers are committed and cannot be used to convey overflow unless one knows it is going to happen before starting to write the output... so in the general case a solution with headers can't be used and one needs to resort to an in-band indicator that output was truncated.

This text was indeed removed from TAP and reference to DALI used so that only a common mechanism would be added, in the right place. However, DALI does not make it clear that there is no overflow mechanism for CSV and TSV. iirc, they both support a header line but do not allow a footer line that is unambiguously not just another row.

The obvious thing would be to clarify this in DALI (will do). Maybe in the case of CSV and TSV we can re-introduce the same text in TAP since it's clear there is no viable solution there (well, there is one involving multi-part but it's pretty hostile to users wanting something simple and those are exactly the ones using these formats).