Tardis data questions

[bmouler]

Hi. First off, I wanted to thank you for making this repo available. Very cool stuff. 💪

I had a question regarding Tardis data import. In the docs, it is not clear which snapshot_mode to use with the tardis.convert method. Default settings seem to be OK but I wanted to know if you had any thoughts on the other modes.

On the same point, do we need to provide a snapshot to HftBacktest when using the same Tardis data processed with default snapshot_mode? It seems to work fine without it.

Thanks in advance!

[nkaz001]

As the document explains, since Tardis intentionally adds the Start-of-Day snapshot, not due to a message ID gap or disconnection, there might not be a need to process the SOD snapshot to build a complete order book. Please see https://docs.tardis.dev/historical-data-details#collected-order-book-data-details for more details.

I don't know the detailed implementation of Tardis, but if it's not due to a message ID gap, then it's okay to ignore the SOD snapshot provided by Tardis by using the ignore_sod mode. Additionally, taking a snapshot at every SOD can slow down backtesting a little bit, as it requires rebuilding the order book every SOD. So in this mode, you might have a slight speed increase.

You can also use the ignore mode to fully ignore any snapshot, not only SOD snapshot. You might lose some market depth information when a message ID gap happens, but if the trading asset is actively traded, the information gap will likely be recovered relatively quickly. This is also explained in detail in Databento's document: https://docs.databento.com/knowledge-base/new-users/normalization

Order book snapshots can be unnecessary for liquid products whose orders turnover very quickly, as pre-existing orders in the snapshot will eventually be filled or canceled - a process commonly referred to as natural refresh.

But, it's important to note that on the first day of your backtesting period, you still need the SOD snapshot. If you don't provide the SOD snapshot, at the beginning, you won't have access to the complete market depth, or at least you'll lose substantial market depth information until the market depth is sufficiently refreshed naturally through updates.

So if the dataset doesn't include the SOD snapshot, you need to provide it through the snapshot argument.

In live trading in Binance, you may encounter message ID gaps often, and immediate recovery by requesting a snapshot may fail from time to time. You need to compare the benefits of leaning towards natural refresh versus full recovery(it may induce some delay). Additionally, in the case of Binance, full depth snapshots are not provided, so there is no way to ensure the complete full market depth.