Integration tests fail on macOS

[adamjstewart]

The parallel dataloader tests still fail on macOS:

$ pytest -m slow
...
self = <tests.samplers.test_single.TestGridGeoSampler object at 0x13524a160>, sampler = <torchgeo.samplers.GridGeoSampler object at 0x13524a2e0>, num_workers = 2

    @pytest.mark.slow
    @pytest.mark.parametrize("num_workers", [0, 1, 2])
    def test_dataloader(self, sampler: GridGeoSampler, num_workers: int) -> None:
        ds = CustomGeoDataset()
        dl = DataLoader(
            ds, sampler=sampler, num_workers=num_workers, collate_fn=stack_samples
        )
>       for _ in dl:

tests/samplers/test_single.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../.spack/.spack-env/view/lib/python3.8/site-packages/torch/utils/data/dataloader.py:359: in __iter__
    return self._get_iterator()
../.spack/.spack-env/view/lib/python3.8/site-packages/torch/utils/data/dataloader.py:305: in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
../.spack/.spack-env/view/lib/python3.8/site-packages/torch/utils/data/dataloader.py:918: in __init__
    w.start()
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
../.spack/.spack-env/._view/upu5ollpmvlcwd5wntc5ajlk2y3qkjpg/lib/python3.8/multiprocessing/reduction.py:60: in dump
    ForkingPickler(file, protocol).dump(obj)
torchgeo/datasets/geo.py:193: in __getstate__
    objects = self.index.intersection(self.index.bounds, objects=True)
../.spack/.spack-env/view/lib/python3.8/site-packages/rtree/index.py:680: in intersection
    return self._intersection_obj(coordinates, objects)
../.spack/.spack-env/view/lib/python3.8/site-packages/rtree/index.py:722: in _intersection_obj
    p_mins, p_maxs = self.get_coordinate_pointers(coordinates)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = rtree.index.Index(bounds=[1.7976931348623157e+308, -1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308], size=0)
coordinates = [1.7976931348623157e+308, 1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308, -1.7976931348623157e+308, -1.7976931348623157e+308]

    def get_coordinate_pointers(self, coordinates):
    
        try:
            iter(coordinates)
        except TypeError:
            raise TypeError('Bounds must be a sequence')
        dimension = self.properties.dimension
    
        mins = ctypes.c_double * dimension
        maxs = ctypes.c_double * dimension
    
        if not self.interleaved:
            coordinates = Index.interleave(coordinates)
    
        # it's a point make it into a bbox. [x, y] => [x, y, x, y]
        if len(coordinates) == dimension:
            coordinates += coordinates
    
        if len(coordinates) != dimension * 2:
            raise core.RTreeError(
                "Coordinates must be in the form "
                "(minx, miny, maxx, maxy) or (x, y) for 2D indexes")
    
        # so here all coords are in the form:
        # [xmin, ymin, zmin, xmax, ymax, zmax]
        for i in range(dimension):
            if not coordinates[i] <= coordinates[i + dimension]:
>               raise core.RTreeError(
                    "Coordinates must not have minimums more than maximums")
E               rtree.exceptions.RTreeError: Coordinates must not have minimums more than maximums

../.spack/.spack-env/view/lib/python3.8/site-packages/rtree/index.py:357: RTreeError

This bug was supposed to be fixed by #304, but this appears to be a new manifestation of #184. Also see Toblerity/rtree#87 for details.

We should:

1. Fix this bug
2. Test macOS and Windows on the release branch

[weiji14]

I've getting a similar error sometimes on Linux with the strange (1.7976931348623157e+308, -1.7976931348623157e+308) bounds. Is switching to interleaved=False the way to resolve this as you hinted at in #409 (comment)?

[adamjstewart]

I've getting a similar error sometimes on Linux

Hmm, that's strange. We run all integration tests on Linux and have never seen this error before. I was under the impression that this bug only affects macOS/Linux. Can you share a minimal script that reproduces this bug reliably (possibly with a fixed random seed)?

Is switching to interleaved=False the way to resolve this as you hinted at in #409 (comment)?

Unfortunately I don't think that would help with this particular issue. This issue affects both interleaved=True and interleaved=False (see Toblerity/rtree#204). The bigger issue is that when pickled, the rtree loses all contents (see Toblerity/rtree#87). The min/max value bounds issue is just a symptom of an empty index. This should have been fixed by #304 but I obviously missed something. I'll try to dig into this again in the next week if I find time.

[weiji14]

I've getting a similar error sometimes on Linux

Hmm, that's strange. We run all integration tests on Linux and have never seen this error before. I was under the impression that this bug only affects macOS/Linux. Can you share a minimal script that reproduces this bug reliably (possibly with a fixed random seed)?

It's a bit hard for me to reproduce because it relies on a custom dataset, but I'll post the error message here first:

  File "/home/username/Documents/github/project/project/neuralnet.py", line 720, in val_dataloader
    batch_sampler = torchgeo.samplers.RandomBatchGeoSampler(
  File "/home/username/miniconda3/envs/project/lib/python3.9/site-packages/torchgeo/samplers/batch.py", line 98, in __init__
    super().__init__(dataset, roi)
  File "/home/username/miniconda3/envs/project/lib/python3.9/site-packages/torchgeo/samplers/batch.py", line 41, in __init__
    roi = BoundingBox(*self.index.bounds)
  File "<string>", line 9, in __init__
  File "/home/username/miniconda3/envs/project/lib/python3.9/site-packages/torchgeo/datasets/utils.py", line 246, in __post_init__
    raise ValueError(
ValueError: Bounding box is invalid: 'minx=1.7976931348623157e+308' > 'maxx=-1.7976931348623157e+308'
*** Error in `python': double free or corruption (!prev): 0x0000556744a917d0 ***

After some troubleshooting, this seems to happen when I make an IntersectionDataset out of two datasets, dataset1 in a UTM projection, and dataset2 in EPSG:4326, and the automatic reprojection of dataset2 from EPSG:4326 to UTM creates an invalid dataset.index.bounds like [1.7976931348623157e+308, -1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308]. Maybe it's just a problem with my dataset that I need to figure out, but the way the bounds are returned with minx > maxx, miny > maxy does look like a problem.

Again, will see if I can manage to get a MWE to reproduce this, but I'll try and do a bit more digging around first.

[adamjstewart]

What's the length of this dataset? Those specific values tend to mean that the dataset is empty (i.e. there is no intersection between those datasets).

[weiji14]

What's the length of this dataset? Those specific values tend to mean that the dataset is empty (i.e. there is no intersection between those datasets).

Actually you're right, the size is 0. Doing print(intersection_dataset.index) gives rtree.index.Index(bounds=[1.7976931348623157e+308, -1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308], size=0). Let me double check what's wrong with my dataset, sorry for the false alarm!