Best practises for defining crypto currencies & instruments for backtests, and future plan?

[yohplala]

Hi,
In backtest/data/providers.py and in model/currencies.pyx are pre-defined some cryptos and instruments.

For instance:

• in providers.py

   @staticmethod
   def adabtc_binance() -> CurrencySpot:
       """
       Return the Binance ADA/BTC instrument for backtesting.

       Returns
       -------
       CurrencySpot

       """
       return CurrencySpot(
           instrument_id=InstrumentId(
               symbol=Symbol("ADA/BTC"),
               venue=Venue("BINANCE"),
           ),
           local_symbol=Symbol("ADABTC"),
           base_currency=ADA,
           quote_currency=BTC,
           price_precision=8,
           size_precision=8,
           price_increment=Price(1e-08, precision=8),
           size_increment=Quantity(1e-08, precision=8),
           lot_size=None,
           max_quantity=Quantity.from_int(90000000),
           min_quantity=Quantity.from_int(1),
           max_notional=None,
           min_notional=Money(0.00010000, BTC),
           max_price=Price(1000, precision=8),
           min_price=Price(1e-8, precision=8),
           margin_init=Decimal("0"),
           margin_maint=Decimal("0"),
           maker_fee=Decimal("0.0010"),
           taker_fee=Decimal("0.0010"),
           ts_event=0,
           ts_init=0,
       )

• in currencies.pyx

ADA = Currency("ADA", precision=6, iso4217=0, name="Cardano", currency_type=CurrencyType.CRYPTO)

I feel a bit disturbed with that, this seems somewhat error prone (for instance #503).
This data is kind of configuration data.
Please, what are your plans regarding this data?

• As there are a lot of cryptos, do you have in mind to make the list grow, for instance thanks to new user manual contributions?
• Or (more interesting), do you have in mind at some point to automate the process, for instance as a step in the continuous integration procedure?
  I am thinking that it could be possible to have an automatic procedure that:
  • would query supported exchanges to know the list of cryptos (currently only binance and Bittrex, right?)
  • and write the list of currencies in a yaml file
  • for some other data (name of crypto), they could be queried from Coingecko I think.

At the moment, I plan to use a generic instrument, along a not so generic Currency with this script.
I am submitting it for advices / feedbacks. I clearly have no very deep understanding about how this data is further used (not at it yet).

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Tue Nov 30 20:30:00 2021
@author: yoh
Crypto generic instrument.
"""
from decimal import Decimal
from nautilus_trader.model.identifiers import Symbol, Venue
from nautilus_trader.model.currency import Currency
from nautilus_trader.model.instruments.currency import CurrencySpot
from nautilus_trader.model.identifiers import InstrumentId
from nautilus_trader.model.currencies import USDT
from nautilus_trader.model.objects import Money, Price, Quantity

CRYPTO_TYPE = USDT.currency_type
CRYPTO_MAP = {'ADA':'Cardano',
              'BCH':'Bitcoin Cash',
              'BNB':'Binance Coin',
              'BRZ':'Brazilian Digital Token',
              'BSV':'Bitcoin SV',
              'BTC':'Bitcoin',
              'DASH':'Dash',
              'DOGE':'DOGE',
              'DOT':'Polkadot',
              'EOS':'EOS',
              'ETH':'Ether',
              'FTT':'FTT',
              'LINK':'Chainlink',
              'LTC':'Litecoin',
              'TRYB':'BiLira',
              'VTC':'Vertcoin',
              'XLM':'Stellar Lumen',
              'XMR':'Monero',
              'XRP':'Ripple',
              'XTZ':'Tezos',
              'USDT':'Tether',
              'ZEC':'Zcash'}

def baseusdt_binance(base:str, precision:int=8) -> CurrencySpot:
    """
    Return Binance BASE/USDT instrument for backtesting.

    Returns
        CurrencySpot
    """
    base = base.upper()
    NAUT_BASE = Currency(base, precision=precision, iso4217=0,
                         name=CRYPTO_MAP[base], currency_type=CRYPTO_TYPE)
    return CurrencySpot(
        instrument_id=InstrumentId(symbol=Symbol(f'{base}/USDT'),
                                   venue=Venue('BINANCE')),
        local_symbol=Symbol(f'{base}USDT'),
        base_currency=NAUT_BASE,
        quote_currency=USDT,
        price_precision=precision,
        size_precision=precision,
        price_increment=Price(10**(-precision), precision=precision),
        size_increment=Quantity(10**(-precision), precision=precision),
        lot_size=None,
        max_quantity=Quantity(90000000, precision=precision),
        min_quantity=Quantity(10**(-precision), precision=precision),
        max_notional=None,
        min_notional=Money(1e-8, USDT),
        max_price=Price(1000000, precision=8),
        min_price=Price(1e-8, precision=8),
        margin_init=Decimal(0),
        margin_maint=Decimal(0),
        maker_fee=Decimal("0.001"),
        taker_fee=Decimal("0.001"),
        ts_event=0,
        ts_init=0,
    )

Thanks in advance for your comments.

[cjdsellers]

Hi @yohplala

So the TestInstrumentProvider is more to provide offline stub instruments for both backtests and unit tests.

What should be specified in the docs however, is that the preferred method for obtaining these instruments is actually through the BinanceInstrumentProvider or other providers specific to their exchanges. This will then request the instrument data over their API, which should alleviate some of your concerns about the correctness of the data.

Right now the BinanceInstrumentProvider only has an async method load_all_async you could run on an event_loop, I will add a convenience load_all synchronous method, the old integration used to have one of those.

Then you can just find your desired instrument from the 1700+ which will then be loaded (Spot).

[yohplala]

Thanks a lot @cjdsellers ! Ok, my python asyncio knowledge is fairly low, and this will be my next step then :)! Thanks again!

[graceyangfan]

@cjdsellers I fellow the steps to load instruments for binance：
(1)use provider to save all instruments from binance_testnet:

    catalog = ParquetDataCatalog("../catalog/.")
    write_objects(catalog,instruments)

(2)load from instruments from catalog

    parser = argparse.ArgumentParser()
    parser.add_argument('--symbol',default='BTCBUSD')
    parser.add_argument("--venue",default='BINANCE') 
    args = parser.parse_args()
    catalog = ParquetDataCatalog("../catalog/.")
    instrument_id = f"{args.symbol}-PERP.{args.venue}"
    instrument = catalog.instruments(instrument_ids=[instrument_id],as_nautilus=True)

I meet this error:

  File "/home/yang/nautilus_trader-develop/examples/catalog/tick2timebar.py", line 68, in <module>
    instrument = catalog.instruments(instrument_ids=[instrument_id],as_nautilus=True)
  File "/home/yang/nautilus_trader-develop/nautilus_trader/persistence/catalog/base.py", line 183, in instruments
    return self._query_subclasses(
  File "/home/yang/nautilus_trader-develop/nautilus_trader/persistence/catalog/parquet.py", line 198, in _query_subclasses
    df = self._query(
  File "/home/yang/nautilus_trader-develop/nautilus_trader/persistence/catalog/parquet.py", line 130, in _query
    return self._handle_table_nautilus(table=table, cls=cls, mappings=mappings)
  File "/home/yang/nautilus_trader-develop/nautilus_trader/persistence/catalog/parquet.py", line 178, in _handle_table_nautilus
    data = ParquetSerializer.deserialize(cls=cls, chunk=dicts)
  File "nautilus_trader/serialization/arrow/serializer.pyx", line 196, in nautilus_trader.serialization.arrow.serializer.ParquetSerializer.deserialize
  File "stringsource", line 69, in cfunc.to_py.__Pyx_CFunc_CryptoPerpetual__lParendict__rParen_to_py_6values.wrap
  File "nautilus_trader/model/instruments/crypto_perpetual.pyx", line 199, in nautilus_trader.model.instruments.crypto_perpetual.CryptoPerpetual.from_dict_c
TypeError: Expected bytes, got str

however drop as_nautilus=True this process works

by the way,I am using the latest develop version on Ubuntu 20.04