Use RocksDB TransactionDB rather than OptimisticTransactionDB

[clemahieu]

The performance difference is minimal and it eliminates the possibility of misconfiguration of memtable size which could lead to out-of-memory or a crash.

[pwojcikdev]

It seems to me like the TransactionDB is the obvious choice for us, we should always favour more reliability over (slightly) more performance.

For reference for anyone else looking at this:

Internally, Transactions need to keep track of which keys have been written recently. The existing in-memory write buffers are re-used for this purpose. Transactions will still obey the existing max_write_buffer_number option when deciding how many write buffers to keep in memory. In addition, using transactions will not affect flushes or compactions.

It is possible that switching to using a [Optimistic]TransactionDB will use more memory than was used previously. If you have set a very large value for max_write_buffer_number, a typical RocksDB instance will never come close to this maximum memory limit. However, an [Optimistic]TransactionDB will try to use as many write buffers as allowed. But, this can be tuned by either reducing max_write_buffer_number or by setting max_write_buffer_size_to_maintain. See memtable for more details about max_write_buffer_size_to_maintain.

OptimisticTransactionDBs: At commit time, optimistic transactions will use the in-memory write buffers for conflict detection. For this to be successful, the data buffered must be older than the changes in the transaction. If not, Commit will fail. To decrease the likelihood of Commit failures due to insufficient buffer history, increase max_write_buffer_size_to_maintain.

TransactionDBs: If SetSnapshot() is used, Put/Delete/Merge/GetForUpdate operations will first check the in-memory buffers to do conflict detection. If there isn’t sufficient data history in the in-memory buffers, the SST files will then be checked. Increasing max_write_buffer_size_to_maintain will reduce the chance that SST files must be read during conflict detection.