Streaming large data sets back to a client

[simonjcarr]

I am reading data from a trino API. I don't know how big the data set is until trino has finished sending the data.

I would like to return a stream to the client and then write data to the stream as Adonis receives it from trino. We are using Adonis 4 on this project, not sure if that will make a difference.

Can anyone provide an example of how I should approach this? My experience with streams in general is limited.

Thanks

[thetutlage]

I am not aware of trino and its API. If they return back a ReadableStream, then you can simply pipe to the response.

trinostream.pipe(response.response)

[simonjcarr]

They don't return a stream, I have to poll the Trino API for results. If there are more results, the API response from Trino provides a nextUri link, which I make a GET request to fetch more data ....

I think I need to return a stream from Adonis to the client and then as data is made available by Trino, write it to the open stream, once no more data is available I would close the stream.

Do you think this is possible?

[simonjcarr]

I think in express I could keep writing respnose.write as more data becomes available and when no more data is available I could send response.end

How would I do this in Adonis?

[thetutlage]

Can you share a working example of how you would do it in express?

[simonjcarr]

I am working on a possible solution now, will let you know if I crack it and provide an example code or where ever I have got up to.