Add validateCOG method to GDALFileTileSource

[banesullivan]

This adds a validate_cog method to GDALFileTileSource to use the osgeo_utils script that will verify a COG.

I'm wondering if we should add another validate_* method to the TiledTiffDirectory that uses tifftools to check the IFDs and tiling of an image

[manthey]

We should figure out how to harmonize this with the "inefficient warning" here https://github.com/girder/large_image/blob/master/sources/tiff/large_image_source_tiff/__init__.py#L324-L339

[manthey]

If a file is small enough, even if it isn't a COG, we probably shouldn't consider it a problem (e.g., if a file is only a few kilobytes, the fact it isn't tiled or optimized is probably irrelevant). Since there are formats that are efficient that aren't tiff, so we probably can only intelligently detect that a file is inefficient in a small set of cases.

[banesullivan]

If a file is small enough, even if it isn't a COG, we probably shouldn't consider it a problem ...

I'm not sure if I'd like this new validate_cog method to handle that, I think this method should solely inform users on whether it is a valid COG or not regardless of size.

IMO, any downstream library that is calling this function should handle the cursory check of: is this under my file size threshold? no? then check if valid COG.

We should figure out how to harmonize this with the "inefficient warning"

Hm. should I add _checkForInefficientDirectories() as an abstract method on FileTileSource and let the GDAL source override it as:

    def _checkForInefficientDirectories(self, warn=True):
        try:
            return self.validate_cog(warn=warn)
        except TileSourcePyramidFormatError:
            pass
        self._inefficientWarning = True

[manthey]

Hm. should I add _checkForInefficientDirectories() as an abstract method on FileTileSource and let the GDAL source override it as:

    def _checkForInefficientDirectories(self, warn=True):
        try:
            return self.validate_cog(warn=warn)
        except TileSourcePyramidFormatError:
            pass
        self._inefficientWarning = True

Let's defer this to a later conversation or PR. There is another project where we may try to determine this for more different sources and we can take it back up then.

[banesullivan]

Let's defer this to a later conversation or PR. There is another project where we may try to determine this for more different sources and we can take it back up then.

Sounds good. I think it'd be nice to have a generalized efficiency check on the base class which the GDAL and TIFF sources can override with their respective checks.

Considering this, I renamed the exception type from [TileSourcePyramidFormatError to TileSourceInefficientError

[manthey]

I pushed a change to the test that checks which sources can read which files so that the new test files don't cause tests to fail.