Added delay to trigger after block end and to prefer public/same block/same proposal calibrations. #403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…k/same proposal calibrations.
mgdaily
approved these changes
Dec 11, 2024
banzai/calibrations.py
Outdated
| dbs.get_master_cal_record(image, self.calibration_type, self.master_selection_criteria, | ||
| self.runtime_context.db_address, | ||
| use_only_older_calibrations=self.runtime_context.use_only_older_calibrations, | ||
| prefer_same_block=self.runtime_context.same_block_cals, |
There was a problem hiding this comment.
can we keep the names of the kwargs consistent with the runtime context attributes?
| @@ -74,6 +73,9 @@ class CalibrationImage(Base): | |||
| good_until = Column(DateTime, default=datetime.datetime(3000, 1, 1)) | |||
| good_after = Column(DateTime, default=datetime.datetime(1000, 1, 1)) | |||
| attributes = Column(JSON) | |||
| blockid = Column(Integer, nullable=True) | |||
| proposal = Column(String(50), nullable=True) | |||
| public_date = Column(DateTime, nullable=True) | |||
There was a problem hiding this comment.
We'll need to do 3 DB migrations here - one for each DB - BANZAI, B-NRES, B-Floyds. Just need to keep that in mind.
There was a problem hiding this comment.
For the future: alembic is a good solution for this sort of thing.
sfoale
reviewed
Dec 12, 2024
banzai/dbs.py
Outdated
| image_filter = db_session.query(CalibrationImage).filter(calibration_criteria & proposal_criteria) | ||
| calibration_image = image_filter.order_by(func.abs(CalibrationImage.dateobs - image.dateobs)).first() | ||
| if check_public: | ||
| calibration_criteria &= CalibrationImage.public_date <= datetime.datetime.utcnow() |
There was a problem hiding this comment.
utcnow() is deprecated - suggested replacement is now datetime.datetime.now(datetime.UTC)
jchate6
approved these changes
Dec 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds functionality to choose which calibration frames are chosen and when data is processed. This is mostly for BANZAI-FLOYDS. I don't expect BANZAI imaging to use this functionality.
- Delaying processing until after the block is complete.
I added functionality that would delay processing until after the block is done observing. This helps for FLOYDS when you want to use an arc and flat taken along with the current data without waiting for the full end of the night.
This is implemented by adding retry logic into the realtime utils, need_to_be_processed. In the case that the block end date is later than now, we use a celery task retry with a delay of 5 minutes after the end of the block. This will only affect certain observation types as set in the settings and has to be manually enabled from the command line.
- Prefering same block or same proposal calibrations
For imaging, all of the calibrations are taken under a public/observatory level program so we haven't had to worry about this. For FLOYDS it is common for the user to take their own calibrations. We then don't want other observations to use that calibration (unless it's public, see below). I have added in the db queries a chain of logic to first check for observations in the same block as the frame that is being processed, then the same proposal, then everything else. The knock on of this is that we now need to store the proposal and block id in the database. The database migration for this is not very complicated but will take a while to run at this point because we have so much data.
- Only use public calibrations
If calibrations from the same block or same proposal don't exist we want to fall back to public data, i.e. we don't want to process data with someone else's proprietary data. To enable this, we need to include the L1PUBDAT from the data in the database. This database migration can be easily completed with the one above.