asyncio support?

[spchamp]

Has there perhaps been any discussion about introducing asyncio support for oandapyv20?

Alternate Approaches ??

As one hypothetically possible approach for asyncio support with the Oanda REST API in Python, today I was taking a look at using the Python (client) generator from the OpenAPI-generator project, starting with the v20-openapi spec from Oanda. The resulting Python code - as produced with templates in the OpenAPI generator Java libraries - it had a sort of optional asyncio feature. It was able to contact the Oanda server, but it looks like it may not have just matched the request URLs expected by the server. Candidly I haven't tried to debug this further, still quite new to this framework. The code doesn't work, as yet, but for purpose of example I might at least try to publish the tooling for the build.

If evaluating the approach of using OpenAPI-generator to produce AIO code: There is a shell script. It needs Java, Maven 3, and jq. When the script's generator command is applied as follows:

HERE=$PWD
mkdir -p ${HERE}/api/client
cd ${HERE}/api/client
${HERE}/scripts/openapi-generator-cli.sh generate -g python --additional-properties library=asyncio \
  --additional-properties disallowAdditionalPropertiesIfNotPresent=false \
  --additional-properties packageName=project_name \
  --additional-properties packageVersion=1.0.0 \
  --additional-properties projectName=project_name  \
  -i ${HERE}/build/v20-openapi-3.0.25/json/v20.json \
  --invoker-package project_name --skip-validate-spec

...then the generator will produce a complete Python project, including a pyproject.toml and test files, etc, in the current working directory. In the example, that would be api/client when the script is called

The generated code is usable for e.g list_accounts() in the following,

import project_name
from pprint import pprint

configuration = project_name.Configuration( host = 'https://api-fxpractice.oanda.com/' )
async with project_name.ApiClient(configuration) as api_client:
    api_instance = project_name.DefaultApi(api_client)
    auth = 'Bearer <private_bearer_token_from_oanda>'
    api_response = await api_instance.list_accounts(auth)
    pprint(api_response)

I'd seen some errors from this, previously, thus alternately the question about aio support here

After restarting the ipython session, it seems to work though. I'll try to publish the build configuration for this, perhaps it's not an entirely straightforward toolchain albeit.

Applications ?

Towards applying aio support for the Oanda REST API: Candidly, I was thinking about trying to mix asyncio with PySide6 (Qt for Python) starting with the Oanda REST support.

The idea here has been to develop an alternative to MetaTrader 5. While the MetaTrader 5 platform might offer some improvements for calculations, compared to MT4, and it offers support for OpenCL and DirectX but it seems that - perhaps due to the absence of broker support - MetaTrader 5 is principally unavailable for live trading with FX brokers licensed in the US.

afaict the earlier MT4 platform might not do really well with a moderate amount of data- and number-crunching for technical indicators. MT4 does OK with line drawing, but after some technical analysis methods developed by John F. Ehlers, MT4 may almost literally start to crunch.

I understand that it should be possible to use synchronous I/O throughout. Considering the possibility of opening a Qt GUI with a tab for each of a set of trading instruments, I thought it might be useful to start out with AIO if possible, whatever hacks might be needed to keep the Python aio parts synchronized with the Qt GUI.

On the Qt GUI side, there's also PythonQwt. It could probably be extended for a presentation of OHLC time series, as well as tick volume series and time-series data from technical indicators, in a Qt chart window.

Considering the possible complexity of the GUI features, I thought it might be conducive to start with the I/O features first, however.

Synchronous and Asynchronous Methods in the Same API ?

For the I/O components, considering the approach used in the code generated with OpenAPI-generator when applied as in the previous, perhaps it may be possible to implement both aio and synchronous I/O client methods in parallel. The code may seem to rely on the aspect that an async def method would return an async generator, such that can be applied as like an AIO Future under an await expression within some calling function, or otherwise applied within the aio framework.

So, perhaps it may be possible to implement that in parallel with a synchronous I/O implementation.

With apologies for if this may resemble something like a blog article: Thus the question of aio support ...?

[hootnot]

Question: you mention oandapyV20 but I get the impression you mean the V20 lib of OANDA, since you mention a method:

    api_response = await api_instance.list_accounts(auth)

in your example. Thing is that oandapyV20 uses request classes. There is no such method as list_accounts()

So, is your question for oandapyV20 or for V20?

[spchamp]

Apologies for my confusing approach. The example was from the source code for what's now been published as pyfx.dispatch.oanda. The API used there was generated with the OpenAPI Generator from an OpenAPI specification for v20 published by OANDA, last updated some years ago - more details available in the readme for the linked project.

At the time when I'd created the issue ticket asking about AIO support, I hadn't been able to get the generated source code running here. After restarting the iPython session and a little more debugging, however, it works now though - for the most part. Not every API method in the Oanda v20 OpenAPI specification is actually available as a REST endpoint, but I've managed to figure out how to use the API to print FX quotes information.

I appreciate that this was converted to a discussion item. Apologies for my rambling in the initial post.

[hootnot]

To answer your question:

I leave things to the developers to program async. code the way they like it. So, short term there will be no changes to oandapyV20 specifically related to asyncio

It is easy to wrap oandapyV20 in async. approaches. Personally I've used gevent a lot over the past years for async. coding.
Using Dramatiq or Celery might be an option.

An example to fetch prices using async (just a demo!):

import asyncio
import oandapyV20
import oandapyV20.endpoints.pricing as pricing
import random


# Define your API access credentials
ACCTID = "..."
TOKEN = "..."


tradables = ["AUD_CAD", "AUD_CHF", "AUD_JPY", "AUD_NZD", "AUD_USD",
             "CAD_CHF", "CAD_JPY", "CHF_JPY", "EUR_AUD", "EUR_CAD",
             "EUR_CHF", "EUR_GBP", "EUR_JPY", "EUR_NZD", "EUR_USD",
             "GBP_AUD", "GBP_CAD", "GBP_CHF", "GBP_JPY", "GBP_NZD",
             "GBP_USD", "NZD_CAD", "NZD_CHF", "NZD_JPY", "NZD_USD",
             "USD_CAD", "USD_CHF", "USD_JPY"]


async def fetch_market_data(client, instrument):
    
    # Create a request to get real-time pricing data
    # JUST one instrument, for async demo purpose (it can take a ',' delimited list of instruments)
    params = {"instruments": instrument}
    request = pricing.PricingInfo(accountID=ACCTID, params=params)
    # force some random processing time
    st = random.randint(0, 1)
    print("Req.: ", instrument, st)
    await asyncio.sleep(st)

    return client.request(request)


async def main(client):
    print("--------- async -------------")
    for task in asyncio.as_completed([fetch_market_data(client, instrument) for instrument in tradables]):
        response = await task
        for price in response['prices']:
            print(f"Instrument: {price['instrument']}, Bid: {price['bids'][0]['price']}, Ask: {price['asks'][0]['price']}")


if __name__ == '__main__':
    # Create an OANDA API client
    client = oandapyV20.API(access_token=TOKEN)

    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(client))

Result:

--------- async -------------
Req.:  EUR_NZD 1
Req.:  AUD_CAD 1
Req.:  EUR_USD 1
Req.:  AUD_CHF 0
Req.:  GBP_AUD 1
Req.:  AUD_JPY 0
Req.:  GBP_CAD 0
Req.:  AUD_NZD 0
Req.:  GBP_JPY 1
Req.:  AUD_USD 1
Req.:  GBP_NZD 0
Req.:  CAD_CHF 1
Req.:  GBP_USD 0
Req.:  CAD_JPY 1
Req.:  NZD_CAD 0
Req.:  CHF_JPY 1
Req.:  NZD_CHF 1
Req.:  EUR_AUD 0
Req.:  NZD_JPY 0
Req.:  GBP_CHF 0
Req.:  NZD_USD 1
Req.:  EUR_CAD 0
Req.:  USD_CAD 0
Req.:  EUR_CHF 0
Req.:  USD_CHF 0
Req.:  EUR_GBP 1
Req.:  USD_JPY 1
Req.:  EUR_JPY 1
Instrument: AUD_CHF, Bid: 0.56873, Ask: 0.57008
Instrument: AUD_JPY, Bid: 94.154, Ask: 94.370
Instrument: GBP_CAD, Bid: 1.69950, Ask: 1.70008
Instrument: AUD_NZD, Bid: 1.08267, Ask: 1.08484
Instrument: GBP_NZD, Bid: 2.11732, Ask: 2.11847
Instrument: GBP_USD, Bid: 1.24598, Ask: 1.24639
Instrument: NZD_CAD, Bid: 0.80240, Ask: 0.80271
Instrument: EUR_AUD, Bid: 1.67600, Ask: 1.67967
Instrument: NZD_JPY, Bid: 86.903, Ask: 87.069
Instrument: GBP_CHF, Bid: 1.11243, Ask: 1.11306
Instrument: EUR_CAD, Bid: 1.45912, Ask: 1.46006
Instrument: USD_CAD, Bid: 1.36397, Ask: 1.36418
Instrument: EUR_CHF, Bid: 0.95514, Ask: 0.95576
Instrument: USD_CHF, Bid: 0.89274, Ask: 0.89311
Instrument: EUR_NZD, Bid: 1.81794, Ask: 1.81930
Instrument: AUD_CAD, Bid: 0.86928, Ask: 0.87081
Instrument: EUR_USD, Bid: 1.06964, Ask: 1.07021
Instrument: GBP_AUD, Bid: 1.95203, Ask: 1.95613
Instrument: GBP_JPY, Bid: 184.152, Ask: 184.241
Instrument: AUD_USD, Bid: 0.63715, Ask: 0.63830
Instrument: CAD_CHF, Bid: 0.65446, Ask: 0.65481
Instrument: CAD_JPY, Bid: 108.342, Ask: 108.386
Instrument: CHF_JPY, Bid: 165.505, Ask: 165.572
Instrument: NZD_CHF, Bid: 0.52523, Ask: 0.52553
Instrument: NZD_USD, Bid: 0.58821, Ask: 0.58873
Instrument: EUR_GBP, Bid: 0.85839, Ask: 0.85896
Instrument: USD_JPY, Bid: 147.757, Ask: 147.847
Instrument: EUR_JPY, Bid: 158.103, Ask: 158.221

[spchamp]

Thanks for the example!

fwiw I wasn't able to figure out where all the pricing endpoints are as yet, lol. After getting the generated code running, I'd been seeing some errors about unavailable endpoints with the couple of pricing endpoints I'd tried, which were described in the original OANDA OpenAPI v20 specification, e.g /v3/instruments/<name>/price and /v3/instruments/pricing/range. These endpoints might not be listed at the OANDA v20 docs though they were in OANDA's v20 OpenAPI specification .... lol.

Once connecting with an HTTP/2 client, using HTTPX/HTTPCore, even the /v3/accounts/<acct>/pricing/stream endpoint seems to be unavailable. The server also produced a server error (internal error) when sending a client-specified timestamp format with the request.

I'm not sure how active their support might be, for this aspect of the OANDA framework. MT5 remains unavailable with brokers in the US, will keep hacking on something though lol. Thx for the example!

[hootnot]

FWIW: I'm currently looking into making a change to have the API client also support async. requests.

[spchamp]

Super!

I guess asyncio isn't necessarily all the thing, in all things?

Reading a little about green futures & green threads implementations recently, after the eventlet integration in futurist. If one could share some ideas: Futurist itself offers a green threads alternative to the multiprocessing Thread Pool Executor, based on eventlets. This is how I'd found the project, after looking for something to address a certain behavior of a Thread Pool Executor producing an atexit behavior that prevents the process from exiting. (There is a patch for this using system threads, or one could use green threads)

Futurist is probably portable to the alternate greenlets API. These green APIs don't seem to use asyncio for scheduling. Perhaps these green approaches may not add a lot of additional concerns for thread-safety, furthermore, as juxtaposed to the one-to-one threads and loops correlation in asyncio.

For green/asyncio interoperability there is an async layer available, based on greenlets. It's probably portable to gevent which also uses greenlets.

After the initial question, I'd developed something of a futures-oriented approach, it ultimately needed a concurrent.futures.Future implementation in addition to asyncio futures. Requests are sent under one (thread, loop) pair using a common httpx client and transport for http/2 request pooling when possible. Responses are each processed under a thread pool worker, with an asyncio loop pre-initilaized to each worker, then asyncio json processing under each loop using ijson (maybe kind of odd here, as the worker would be running at most one processing routine, I'd hoped it might help to balance the processing load for the other threads though, to use asyncio in the worker threads too)

Of course, the loops and futures under asyncio aren't rated "Thread-safe", and will often throw an exception if something sets a result or exception state from the wrong thread. So, there was the addition of a concurrent.futures.Future. This could be used to pass either an exception or a deserialized object back to the request caller, from within a response processor under a worker thread. The Future might even implement an __await__ function, e.g. return self.apoll().__await__() so the request caller can poll for the response asynchronously, without blocking the request thread. Maybe a greenlets-based approach could be a lot more straightforward.

With the greenlets API, gevent offers a system thread pool. The thread-safety caveats might be less limiting here.

I'm not sure if there's HTTP/2 client support in gevent itself. Maybe it could be fairly straightforward to wrap a synchronous HTTP/2 client in greenlet calls?

Hoping this helps. There are other alternatives for concurrent processing?

[hootnot]

I've been using oandapV20 with gevent over the past years. Works fine,

For now my focus is to get asyncio support in oandapyV20.
Basically it comes down to an additional client (the API class) that supports asyncio.
The current API class uses requests for the HTTP communication.
I've written an async variant that uses the httpx library.