Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the max_steps_gap parameter and missing timestamp functionality in select_time_slice() #107

Open
dfulu opened this issue Dec 23, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@dfulu
Copy link
Member

dfulu commented Dec 23, 2024

In the slice_datasets_by_time() function, the call to select_time_slice() used for the satellite data uses max_steps_gap=2 - see here.

The value of max_steps_gap is entirely ignored in this function. This parameter is a carry over from ocf_datapipes and the slicing logic which uses this setting has not been ported across yet. Quite possibly we will never want to use this functionality since this was intended for use only in a production setting where data might be missing. We do this in the app now instead. During training, since we filter to t0 times where no satellite data may be missing, our current usage would never hit this anyway. We do not have the functionality in our t0 filtering to allow any missing time steps, so we should never need this.

I suggest we:

  • Strip out this parameter and the placeholder _sel_fillinterp() function which it was intended to be used with.
  • Plus remove the _sel_fillnan() function and the fill_selection parameter. I don't think we use these or will ever use these for training.
@dfulu dfulu added the bug Something isn't working label Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant