Character limit on select field

[Fingel]

Hi,

I'm trying to use the API to automate querying this stored query: https://lasair.roe.ac.uk/myquery/115/

Though the response from the server indicates that the size of the "select" parameter is too large:

{"selected":["Ensure this field has no more than 1024 characters."]}

Two things:

1. Is there a way around this size limit?
2. Even better would be if we could use the api to get results from a stored query instead of supplying the query directly. For example: GET https://lasair-iris.roe.ac.uk/api/query/stored/115 would return the results of that stored query. That way we can edit and visualize the query using your excellent online tool, and changes would be reflected in our code without having to update the query text stored elsewhere.

Thanks!

[RoyWilliams]

Hi Austin Lasair queries can be made into 'streaming' queries, meaning that the query is run in near real time as the alert flow, and the results flow out as kafka or email. Those results are also kept as a 'stream log'. For example the query 'SN-like candidates' is defined here https://lasair-iris.roe.ac.uk/query/2/ and its stream log is here https://lasair-iris.roe.ac.uk/streams/2SN-likecandidates/ You can run an ad hoc query via the API using the 'api/query' method, or you can fetch the stream from a stored query using the 'api/streams' method. In the near future, we will also offer push distribution via kafka. As you have requested, I have raised the charaacter limit to 4096 from 1024. I have also tried to rebuild your query on lasair-iris. The query is https://lasair-iris.roe.ac.uk/query/69/ and its stream will appear as https://lasair-iris.roe.ac.uk/streams/2Roys_TDE-candidates/ once something comes through from ZTF I have used some new recent features of Lasair. In particular: jdnow() returns the current julian day dmdt_g and dmdt_r are the latest values of delta magnitude over delta time (with positive values meaning it is getting brighter) g_minus_r is the g-r color, for the most recent night with both g and r detections Roy

…

--- Royal Observatory Edinburgh [email protected] 07542 869986

________________________________ From: Austin Riba [[email protected]] Sent: 23 February 2021 7:35 PM To: lsst-uk/lasair Cc: Subscribed Subject: [lsst-uk/lasair] Character limit on select field (#234) Hi, I'm trying to use the API to automate querying this stored query: https://lasair.roe.ac.uk/myquery/115/ Though the response from the server indicates that the size of the "select" parameter is too large: {"selected":["Ensure this field has no more than 1024 characters."]} Two things: 1. Is there a way around this size limit? 2. Even better would be if we could use the api to get results from a stored query instead of supplying the query directly. For example: GET https://lasair-iris.roe.ac.uk/api/query/stored/115 would return the results of that stored query. That way we can edit and visualize the query using your excellent online tool, and changes would be reflected in our code without having to update the query text stored elsewhere. Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#234>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACDZRUMFLJWCOIIRQNIVRD3TAP7QNANCNFSM4YDC2JKA>.

[Fingel]

Hi Roy,

I think I catch your drift. So instead of using the API to send the query directly (polling) we can just follow the stream log. This is very cool.

For some reason I had equated streaming == real time, but now I realize it's real time on your end only, and we can just follow the log.

Thank you!

[Fingel]

Hi Roy,

I think I'm a bit confused on the difference between lasair and lasair-iris. The query our project has been working on is here:

https://lasair.roe.ac.uk/streamdigest/13TDE-candidatesv3.1/

But it doesn't look like it's available as JSON anywhere. Additionally, the query uses the Candidate table, which doesn't appear on the Iris side.

Thanks for the copied query you created, we'll look at that, but it'd be good to get a better understanding of the differences.

[RoyWilliams]

Austin Lasair has been running since May 2018 and has nearly 3 -year light curves. Its software has not changed since April 2020. Queries that JOIN the objects and candidates are quite slow. The ingestion is slow, since everything goes through the MySQL database, which is a single machine and not scalable. Lasair-iris has been accumulating light curves for only a month, and has been a volatile prototype, although we expect/hope things to stabilise now. It has new features, such as the API. The ingestion is scalable by adding more cores, and we expect it to be able to handle LSST rates. There is still a central MySQL database, but it is smaller -- objects and features, no candidates). Further, the ingestion and streaming queries can continue even if the central MySQL machine is bogged down. The individual candidates -- and images -- are in the back (Cassandra and/or CephFS). Hope this helps Roy

…

--- Royal Observatory Edinburgh [email protected] 07542 869986

________________________________ From: Austin Riba [[email protected]] Sent: 24 February 2021 5:55 PM To: lsst-uk/lasair Cc: Roy Williams; Comment Subject: Re: [lsst-uk/lasair] Character limit on select field (#234) Hi Roy, I think I'm a bit confused on the difference between lasair and lasair-iris. The query our project has been working on is here: https://lasair.roe.ac.uk/streamdigest/13TDE-candidatesv3.1/ But it doesn't look like it's available as JSON anywhere. Additionally, the query uses the Candidate table, which doesn't appear on the Iris side. Thanks for the copied query you created, we'll look at that, but it'd be good to get a better understanding of the differences. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#234 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACDZRUPKAPKQR2G2EL2ZI3DTAU4Q5ANCNFSM4YDC2JKA>.