Generating mbtiles on HPE ProLiant MicroServer Gen10 Plus v2 (32 GB RAM, Intel Pentium E-2314)

[Alex-7]

With the Planetiler I can work with the PBF file up to about 4GB. I could generate France, South America, Germany, etc. but not the Europe or Africa, which are 26+ and 5+ GB.

I use the following command:
java -Xmx20g -jar planetiler.jar --area=australia-oceania --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --mbtiles=australia-oceania.mbtiles --nodemap-type=array --storage=mmap

In Europe I create mbtiles file for each country separately and then use the program tile-join to merge the mbtiles. It takes time, but it works correctly. Here is the resulting map: https://labellechose.ch/map/ . It is still work in progress, quite a few countries are still missing.

I wonder if I could somehow generate the Europe or Africa, or even better the whole planet, in one go. Time is not a problem. The micro-server does not take much of energy and it is in the homelab.

I could generate the planet with the sequentially-generate-planet-mbtiles program. It takes a day or two, but there are issues with the water shapes for some rivers on the resulting vector map. Planetiler, however, creates a correct vector map.

[msbarry]

What failure do you run into? You should be able to generate the entire planet with those settings and 32gb of ram without needing to stitch together multiple runs.

[Alex-7]

Thank you for your reply. I just started the planet with the following command:
java -Xmx20g -jar planetiler.jar --area=planet --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --mbtiles=planet.mbtiles --nodemap-type=array --storage=mmap
I will inform of the result as soon as I have it.

[msbarry]

Sounds good. Planetiler uses the RAM outside of what the jvm uses for the OS to cache the 80gb memory-mapped node location file, so with only 12gb extra it will run on the slow side during osm pass2 but should still finish.

[Alex-7]

These are the last lines before the Planetiler stops. It ran about 10 minutes and then stopped. But if I use an area which is 4 GB or less (I see the size at https://download.geofabrik.de/ ) than it works OK.

26:29 DEB - ✓ 361G storage on /mnt/sdb (/dev/sdb) requested for write phase disk, 3.1T available
0:26:29 DEB -  - 241G used for temporary feature storage
0:26:29 DEB -  - 120G used for mbtiles output
0:26:29 WAR - ❌️ 21G JVM heap requested for read phase, 21G available (increase JVM -Xmx setting)
0:26:29 WAR -    - 21G used for temporary profile storage
0:26:29 DEB - ✓ 93G storage on /mnt/sdb (/dev/sdb) requested for read phase, 3.1T available
0:26:29 DEB -  - 82G used for array node location cache (switch to sparsearray to reduce size)
0:26:29 DEB -  - 11G used for multipolygon way geometries
Exception in thread "main" java.lang.IllegalStateException: Insufficient resources for read phase, use the --force argument to continue anyway:
21G JVM heap requested for read phase, 21G available
        at com.onthegomap.planetiler.util.ResourceUsage.checkAgainstLimits(ResourceUsage.java:162)
        at com.onthegomap.planetiler.Planetiler.checkMemory(Planetiler.java:601)
        at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:520)
        at com.onthegomap.planetiler.basemap.BasemapMain.run(BasemapMain.java:51)
        at com.onthegomap.planetiler.basemap.BasemapMain.main(BasemapMain.java:14)
        at com.onthegomap.planetiler.Main.main(Main.java:50)

[msbarry]

It does a quick check to estimate if you've got enough resources before starting to avoid failing late in the process. Try giving it -Xmx21g or you could just run with --force to bypass the check. It's just an estimate so you'd probably be fine with -Xmx20g

[Alex-7]

OK. Thank you. I will do as advised and report later.

[msbarry]

Any luck?

[Alex-7]

I started it with -Xmx21g (without --force) and it is still running, several hours already. I keep my fingers crossed.

[Alex-7]

I am happy to report that the planet.mbtiles was well generated on the micro-server. I do not see any errors on the map.

I used an HDD, but next time I will use an SSD.

[msbarry]

Cool! How long did it end up taking?

[Alex-7]

It took more than a week. This server consumes less than 30 W; - I measured with a wattmeter. I had an impression that the HDD was the bottleneck. It is the good 4TB WD Red Label, but about 5 years old. And I did not do --building-merge-z13=false . I think that with the SSD it will be times faster.