Running $2.1\cdot 10^9$ simple cells blows the memory

[thorstenhater]

During a set of benchmarks in the last week I had the following occur

• being murdered by the OOM killer
• a vector cannot be (re)alloc'ed with exceeding v.max_size()

both happened on current master as of Sept 22 on GPU enabled libarbor.
Example running was ring.cpp from the busyring engine in NSuite

• GPU: cuda
• 2.1 Billion cells of the complex variety
• No spikes/samples stored

Investigation

Source Labels

Further experiments needed, but I suspect the spike source label map consolidation
during simulation setup. It relies on mpi::gather_all which builds a vector of all items
on all tasks. Each item is a (gid_t gid, std::string label, lid_t lo, lid_t hi) and in our example
each cell has one detector called 'detector' thus

• gid: 8B
• label: 8B
• lo: 4B
• hi: 4B

and in total $24\times 2.1 GB \simeq 48GB$ at the very minimum, ignoring all extra infrastructure
and overhead for

• std::string: likely a pointer and capacity size_t, maybe less depending on SSO
• std::unordered_map<gid, std::unordered_map<std::string, range>>, pointers, capacities, ...
• std::vector up to $\times 1.5 - 2$ depending on the implementation.
• padding/alignment of all data structures