Can osmupdate cause an "out of memory" error?

[zstadler]

Israel Hiking Map uses osmupdate to combine daily extracts from Geofabrik with minutely planet updates to achieve a continuously-updated map. In each update round, osmupdate downloads the required planet updates and creates an updated .pbf file based on the previously-used .pbf file.

However, if the updated .pbf file is provided to planetiler, it encounters an "out of memory" error. This error does not happen when using the .pbf file from Geofabrik.

[zstadler]

To reproduce:

1. In an empty directory, successfully run planetiler using the latest Geofabrik daily extract

docker run --rm -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest --download --area=israel-and-palestine

2. Download the boundary polygon of the extract

docker run --rm -v .:/tileset openmaptiles/openmaptiles-tools wget -c -O data/israel_and_palestine.poly https://download.geofabrik.de/asia/israel-and-palestine.poly data/sources/israel_and_palestine.osm.pbf

3. Create an updated osm.pbf file

docker run --rm -v .:/tileset openmaptiles/openmaptiles-tools osmupdate --verbose -B=data/israel_and_palestine.poly data/sources/israel_and_palestine.osm.pbf data/sources/israel_and_palestine_new.o5m
docker run --rm -v .:/tileset openmaptiles/openmaptiles-tools osmconvert --drop-broken-refs -B=data/israel_and_palestine.poly --hash-memory=2000 --verbose -o=data/sources/israel_and_palestine_new.osm.pbf data/sources/israel_and_palestine_new.o5m

4. Run maptiler on the updated osm.pbf file

docker run --rm -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest --area=israel_and_palestine_new --output=data/output_new.mbtiles 

and see this error

0:00:31 INF [osm_pass1] -
0:00:31 INF [osm_pass1] - Starting...
Worker osm_pass1_parse-3 died
java.lang.OutOfMemoryError: Java heap space

P.S.,
No other tool, including OpenMapTiles, has had any issue processing such updated osm.pbf files.

[msbarry]

Looks like osmconvert emits a PBF file with very few (10) large blocks whereas the original input file had many (1842) small blocks. Each block gets fully parsed/expanded to in-memory objects then processed by a worker thread so this basically ends up loading the entire thing in memory at once and -Xmx1g 1GB of ram is a pretty low limit for planetiler. You could either get it working by using -Xmx5g or limit to only having 1 thread expanding blocks at a time with --threads=1 then you could get away with -Xmx2g.

Alternatively there might be a config parameter to use to tell osmconvert to generate smaller blocks, but I'm not seeing anything obvious in the --help output.

[marqqs]

Hi there, I'm a bit confused now... :-)
Which PBF blob size you use is usually a matter of compression level you would like to achieve. According to PBF format specs, PBF-reading software should accept blobs up to a size of 32 MiB. If Planetiler does not adhere to these specs and therefore cannot cope with blobs of this size, what is the maximum blob size Planetiler can process?

[msbarry]

Planetiler can read PBF files with any size blob, you just need to give the JVM enough ram to process those larger blocks using the -Xmx jvm setting.

[zstadler]

I don't think there is an issue of incompatibility here.

It seems like the two programs are optimized for different goals: osmconvert is optimized for minimizing the size of the resulting pbf file while Planetiler is optimized for minimizing its execution time and memory requirements.

It would be great if we knew what blob size would make Planetiler "happy", and if osmconvert would accept a parameter that sets the blob size.

[marqqs]

@zstadler
It seems your use case has very specific needs regarding memory space, etc.
As @msbarry suggested it might be necessary to use the -Xmx option.

If you want conduct some experiments to determine the optimum fileblock size for your toolchain, you can do this by changing the constant pw__data_spaceM in osmconvert's source.