Investigate the viability of "background matching" technique

[tms-epcc]

From Dec'22 quarterly update

• Investigate the viability of a second-order solution proposed by DM called "background matching" for possible use in full survey.
• this was a stretch goal for Phase B
• not a first order priority

[tms-epcc]

10/MAY/23

• This ticket corresponds to the following task as described in the FY23 Q2 March'23 quarterly update
  • Discuss "background-matching" sky-subtraction technique with Data Management to investigate viability.
  • this is a stretch goal and is not fundamental to the strategy
  • currently not convinced it will provide an improvement
  • met DM on 08/MAY/23 to discuss this
• This ticket also corresponds to the further task as described in the FY23 Q2 March'23 quarterly update
  • Continue ongoing work on LSB-friendly sky subtraction algorithms and work with DM on strategies for incorporation into pipeline.
  • This tasks will be ongoing for some time.
  • Discussions on incorporation will be ongoing. This will require DM approval. Expect more iteration on this with DM team.

[tms-epcc]

30/MAY/23

• working on other priorities so no progress

[tms-epcc]

15/JUN/23

• no progress due to other priorities

[tms-epcc]

04/JUL/23

• expect to start discussions with DM team on implementation
• automation a key consideration for metrics

[tms-epcc]

20/JUL/23

• continuing to investigate viability
• on hold
• this is a stretch goal

[tms-epcc]

11/APR/24

• This was originally a stretch goal but further investigations has taken place
• Aaron made synthetic images and assessed what is meant to happen and why.
• Able to get same sky across all coadds
• DM have some existing software that Aaron may finish if appropriate

[tms-epcc]

02/MAY/24

• progressing this
• there is a version in the LSST github.
• asked Robert Lupton about status of this but no response yet
• broadening strategy by considering other options
• challenge now is image quality which has a lot of factors

[tms-epcc]

23/MAY/24

• recvd confirmation that the DM Github package is doing background matching
• found that package had been tried on HSC pipeline but DM team found it did not work as they wanted so it has been set aside by DM
• One option is therefore for @aemerywatkins to get involved and take this package further
• For the moment will investigate this on HSC and will talk to Robert and Yusra on why exactly the package was shelved.

[tms-epcc]

17/JUN/24

• got technique working on own images
• indicates this is viable
• presented to Lee and others in DM
• will discuss more with Lee
• will then set up ticket to revise the code and then test on HSC images

[tms-epcc]

09/JUL/24

• See DM JIRA ticket - https://rubinobs.atlassian.net/browse/DM-44889
• existing code in LSST Github - matchbackgrounds.py written some time ago
• @aemerywatkins getting this working with Gen3
• mostly done - code executes apart from last part when writing images to disk

[tms-epcc]

23/JUL/24

• code is functioning
• run on series of images, output was sensible
• working with Lee on how to optimise and meet DM's code standards
• this approach is still the most attractive since it means people can reprocess images themselves rather than having to supercomputing time to do this.
• still to try running this at scale and obviously still to convince DM that this is the most appropriate technique
• so currently ironing out various details

[tms-epcc]

06/AUG/24

• Robert Lupton taking great interest and providing ideas
• Suggested try the technique on old M31 galaxy images since if it works on these it should work on many others
• So currently working on this

[tms-epcc]

24/SEP/24

• discussing with Lee that Robert is always asking about the project
• gave the M31 processing a try but pipeline crashing for various reasons so have put on hold for now
• there are now DE images of the Triffid nebula that have gone further on the pipeline so may try on that
• for now though Aaron continuing with development and working on the issues raised by Lee previously

[tms-epcc]

22/OCT/24

• image patterns are staying in the image w/o sky subtraction (see https://rubinobs.atlassian.net/browse/DM-46693)
  • currently under review
  • one further item to complete before merge
• Lee and Aaron met with Jim Bosch to discuss next steps
  • investigating whether depending on technique, crashes will occur in certain parts of sky

[tms-epcc]

12/NOV/24

• skycor modification needs review before merge
• considering Jim Bosch's ideas as well as crashes this includes concerns about contiguity
• will visit Princeton in Dec'24 and discuss further with Jim in-person

[tms-epcc]

28/NOV/24

• waiting on skycor change merge still; need Lee to look at this
• contiguity issue solved but there are some lingering onces
• will still discuss with Jim Bosch

[tms-epcc]

14/JAN/25

• skycor merge successful
• a bug discovered in the warping code and being investigated. This is preventing final implementation of the warping code.
• Still to discuss with Jim Bosch.