Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the client to deal with stream from the datatransfer service #151

Closed
1 of 3 tasks
JulienPeloton opened this issue Jan 16, 2023 · 0 comments · Fixed by #152
Closed
1 of 3 tasks

Update the client to deal with stream from the datatransfer service #151

JulienPeloton opened this issue Jan 16, 2023 · 0 comments · Fixed by #152

Comments

@JulienPeloton
Copy link
Member

JulienPeloton commented Jan 16, 2023

PoC:

fink_datatransfer.py -h
usage: fink_datatransfer.py [-h] [-topic TOPIC] [-limit LIMIT] [-outdir OUTDIR] [-partitionby PARTITIONBY] [-batchsize BATCHSIZE] [--restart_from_beginning]
                            [--verbose]

Kafka consumer to listen and archive Fink streams from the data transfer service

optional arguments:
  -h, --help            show this help message and exit
  -topic TOPIC          Topic name for the stream that contains the data.
  -limit LIMIT          If specified, download only `limit` alerts from the stream. Default is None, that is download all alerts.
  -outdir OUTDIR        Folder to store incoming alerts. It will be created if it does not exist.
  -partitionby PARTITIONBY
                        Partition data by `time` (year=YYYY/month=MM/day=DD), or `finkclass` (finkclass=CLASS), or `tnsclass` (tnsclass=CLASS). Default is
                        time.
  -batchsize BATCHSIZE  Maximum number of alert within the `maxtimeout` (see conf). Default is 1000 alerts.
  --restart_from_beginning
                        If specified, restart downloading from the 1st alert in the stream. Default is False.
  --verbose             If specified, print on screen information about the consuming.

Note that unlike the livestream service, the schema for these alerts is stored in a separate topic (topic_name_schema) that is polled simultaneously when retrieving alerts.

To do:

  • new script
  • documentation in the README
  • documentation in the readthedocs (along with the datatransfer service).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant