Need help diagnosing failure when building tiles for North America

[diehl]

I attempted to build vector tiles for North America overnight with the following command:

docker run -e JAVA_TOOL_OPTIONS="-Xmx15g" -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest --download --area=north-america

And for some reason, the process seems to halt midway with no obvious error message as shown below.

...
1:39:45 INF [osm_pass2] -  nodes: [ 1.9B 100%    0/s ] 23G   ways: [ 172M 100%    0/s ] rels: [ 768k  42% 3.1k/s ] features: [ 690M  27k/s ] 51G   blocks: [ 267k 100%   <1/s ]
    cpus: 4.5 gc:  1% heap: 4.9G/16G direct: 55M postGC: 2.4G relInfo:  84M mpGeoms: 2.8G
    read( 0%) ->   (11/30) -> process(18% 21% 14% 16% 18% 21% 35% 39% 47% 21% 29% 35% 19% 15% 15% 16% 23% 16% 11%) -> (58k/70k) -> write(10%)
1:39:55 INF [osm_pass2] -  nodes: [ 1.9B 100%    0/s ] 23G   ways: [ 172M 100%    0/s ] rels: [ 821k  45% 5.2k/s ] features: [ 692M 147k/s ] 51G   blocks: [ 267k 100%   <1/s ]
    cpus: 4 gc:  1% heap: 2.5G/16G direct: 55M postGC: 2.5G relInfo:  84M mpGeoms: 2.8G
    read( 0%) ->   (11/30) -> process(25% 16% 16% 14% 13%  3% 16% 47% 18% 29% 20% 22% 15% 15% 17%  7% 19% 22% 11%) -> (60k/70k) -> write(49%)
1:40:05 INF [osm_pass2] -  nodes: [ 1.9B 100%    0/s ] 23G   ways: [ 172M 100%    0/s ] rels: [ 874k  48% 5.2k/s ] features: [ 693M 125k/s ] 51G   blocks: [ 267k 100%   <1/s ]
    cpus: 4.5 gc:  2% heap: 4.1G/16G direct: 55M postGC: 2.6G relInfo:  84M mpGeoms: 2.8G
    read( 0%) ->   (11/30) -> process(21% 15% 17% 25% 18%  2% 22% 32% 26% 20% 26% 23% 31%  9% 39%  5% 19% 22% 17%) -> (61k/70k) -> write(50%)
1:40:15 INF [osm_pass2] -  nodes: [ 1.9B 100%    0/s ] 23G   ways: [ 172M 100%    0/s ] rels: [ 924k  50% 4.9k/s ] features: [ 694M  85k/s ] 51G   blocks: [ 267k 100%   <1/s ]
    cpus: 5.8 gc:  5% heap: 4.3G/16G direct: 55M postGC: 2.8G relInfo:  84M mpGeoms: 2.8G
    read( 0%) ->   (11/30) -> process(36%  6% 24% 15% 19%  1% 19% 29% 31% 28% 15% 17% 33% 24% 39% 87% 23% 21% 27%) -> (60k/70k) -> write(46%)
(base) diehl@fusion planetiler %

Any suggestions on how I can diagnose what is going on here would be greatly appreciated!!

Update: Noticing an Exited(137) status for the Docker container. Going to try running this in the cloud on a larger machine. My iMac Pro has 32 GB which I thought would be sufficient.

[diehl]

Taking this to the cloud running on a VM with 64GB of RAM did the trick. I went back to my iMac Pro and tried running it again with --storage=mmap and it exited even faster. So strange.

[msbarry]

Yeah, it looks like docker must have killed it for some reason. May have exceeded memory or disk quota or something. 32gb is more than enough. You can do entire planet with only 8gb but the docker settings can be a bit finicky

[msbarry]

That's just a warning if you might not have enough resources to fail early if there's a chance it could fail later. Just run with --force to ignore it.

[diehl]

Got it - thanks @msbarry! I'll fire it up now.

[diehl]

Built a full planet map today in 4.5 hours on my iMac Pro and then a reduced resolution planet map in 1.5 hours. Truly amazing.

[diehl]

@msbarry You rock. So impressed by what you've built here. Planetiler is a gamechanger.

[msbarry]

Awesome! Glad it worked out