large api/map requests still result in "out of memory" messages

[chrisnatali]

Either address or limit number of nodes/area.

[chrisnatali]

This appears to be the reason the server hung last time: https://community.nitrous.io/posts/stability-and-a-linux-oom-killer-bug

Doesn't address why we're running out of memory, but updating to kernel > 3.13 should prevent us from getting locked out of the machine.

[chrisnatali]

Useful for watching resource usage of specific pids:

ps -ef | grep '\(map\)\|\(postgres\)' | awk 'BEGIN { ORS="," } {print $2}' | sed 's/,$//' > pids
htop -p `cat pids`

[chrisnatali]

Did some load testing of selcolumbia/osm-gridmaps-cgimap:0.3 (bounds restricted to 1000 sq degrees) vs osm-gridmaps-cgimap:0.4 (bounds restricted to 100 sq degrees). Found the former maxing out at over 14G while the latter remained at about 3G under the same load.

From this, it appears that cgimap loads all tiles for the bounding box into memory and also places that into postgres. That's consistent with the load profile, but will confirm.

This doc on osm quad_tile impl will be helpful.

[chrisnatali]

Helpful quick load test:

Put 20 http requests into a file

seq 20 | xargs -i echo "http://172.17.0.3/api/0.6/map?bbox=4.0,9.0,14.9,19.9" >> calls

Execute them in parallel

cat calls | xargs -i -P10 curl {} > /dev/null

Monitor docker container memory

while true; do 
    for f in `find /sys/fs/cgroup/memory/docker/* -type d`; do 
        cat $f/memory.usage_in_bytes 
    done 
    echo "---" 
    sleep 1 
done