Observable write stalls on high load

[v0y4g3r]

What type of bug is this?

Performance issue

What subsystems are affected?

Datanode

Minimal reproduce step

You can simply reproduce this in TSBS suite.

What did you expect to see?

There should be no peaks and valleys in requests handled per second.

What did you see instead?

When performs benchmark on disks with average performance (like AWS gp2/3 with no extra IOPS budget), we can observe noticeable write stall according to the metric mito_write_rows_total.

The stalls have a correlation in terms of time with fsync operations in WAL. Everytime the WAL rotates, it will allocate a new log file and fsync the previous log file to ensure durability. This will cause high IO util.

Some methods to mitigate these stalls:

• We may break the large fsync into frequent smaller fsyncs to amortize the cost
• Enable log recycle to reuse obsolate log files.

What operating system did you use?

NA