Neurips 2023 Submission

[voetberg]

@bnord

We want to submit this project for neurips 2023 - What should be included? How should we pitch it?

[bnord]

One thing is that we should review the literature on competitions and benchmark data

1. e.g., Plasticc about supernovae
2. e.g., weak lensing challenges -- e.g., GREAT08
3. e.g., kaggle set-ups

@rhea05 Could you share the link to the shared zotero library for the LSST white paper biz?

[bnord]

thank you!

[bnord]

I've read a lot of these papers, and it seems a lot of things come down to amount of time in a given part of the sky.
@ehneilsen Could you provide another perspective another major leading problems in scheduling? Do you know of any specific problems or papers as examples for us?

[voetberg]

I do have a collection of difference challenge outlines here and this blogpost (with a few dead links but what can you do)

[ehneilsen]

Yes, I have quite a few other papers worth looking at.
First, ApJ did a recent special issue on Rubin/LSST Survey Strategy Optimization, with many relevant papers.
I also strongly encourage everyone to read my tech note on DES observing strategy.
If you like, I can add these (and maybe a few others) to the TelescopePositioningChallenge zotero collection.

[bnord]

@ehneilsen yes, please add!

[bnord]

@ehneilsen Would you like to be involved in designing this competition?

[ehneilsen]

Involved, sure, but not deeply: I'm happy to advise, but am not able to commit the time to organize or manage.

[bnord]

Are we interested in managing this competition? Or should we just create the venue and let people self-organize?

I'm not sure I want to spend a lot of effort on this. It would be great if we could make it self-organized.
On the other hand, I'm not sure if it will succeed without some centralized organization.

[voetberg]

For additional reference - I did vaguely outline what would be required for a writeup in this md file. We can work off that file or just link a google doc instead

[ehneilsen]

Some initial thoughts:

1. We should do something better with our reward function. For a general competition, and a reward function that might be widely applicable (with minor variations), some much better alternatives to simply maximizing t_eff would be: for a given set of pointings, either maximizing t_eff of the field with the worst accumulated t_eff at the end of the survey; or maximizing the number of pointings whose combined t_eff exceeds some threshold.
2. We should include a weather model.
3. We should consider making the scheduler choose between filters.

[voetberg]

Do you have recommendations for how to start including a weather model? Mechanically/coding wise I don't think it would be hard with the set up I have but I'm a little at a loss for where to go to look for resources for this.

For your other two points - I can write some other example reward functions as subclasses of the existing survey class, and the functionally for filter switching is in the sim, but it can be highlighted further in the outline here.

[ehneilsen]

There are two aspects to the weather model: clouds and seeing.
For the clouds, you can take a look at what I put together for Rubin/LSST simulations here. These can then be turned into a reduction in t_eff (or a fraction of exposures rendered useless) using statistics from this tech note. In practice, for this exercise, I think this is far more elaborate than is needed, and also for the wrong location (Cerro Tololo, Chile, rather than Stone Edge Observatory).

You can, however, get weather data for locations in the U. S. from the NOAA here, with documentation here. I was able to download a CSV of the data for the past couple of years from the Santa Rosa airport weather station (which seems to be the one closest to SEO) this morning.

For our purposes, I think it's good enough to approximate all time as either "useless" (SCT, BKN, OVC anywhere in the "HourlySkyConditions" column, maybe) or "perfect" (anything else). The simplest thing to do, I think, would be to download as much weather data as we can from NOAA (the site claims to go back to 2000, but when I asked for all of that at once the request failed. Doing just a few recent years worked) and either replay it as is, or take statistics by month and use that to randomly generate as much data as we want (good enough for this project).

[ehneilsen]

Basing seeing conditions on real data for SEO will be harder, and we should probably not worry about it; let's just use uniform seeing.

[ehneilsen]

Note the usefulness of including weather conditions depends on what we are using as our reward.

If we continue to use maximizing total t_eff and nothing else, then adding weather shouldn't make any difference.

On the other hand, if we, say, want to maximize the minimum t_eff over a few different fields, it could make a significant difference.
For example, let's say:

• we can observe field A and field B,
• our reward is the total t_eff for the field for which we have the lowest total t_eff.
• we have two nights we are scheduled to observe, one in January and one in March.
• if weather is clear, field A is better observed in January, and field B is better observed in March.

If the weather is the same always, then the optimal is spending all of our time in January on field A, and all of our time in March on field B.

But, if there is weather, if we start observing in January and notice that the weather is clear, it might be good to spend at least some time observing field B in case March is clouded out. What fraction this should be depends on how likely it is to be cloudy in March. For example, if March is almost always entirely cloudy, we will want to spend a lot of time observing field B in January, while if it is almost always entirely clear, then we will want to stick to field A in January.

On the other hand, if January is clouded out, a good scheduler should be smart enough to split its time in March between fields A and B, and not just do B.