Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define a mechanism to log data as explicitly invalid #5293

Open
jleibs opened this issue Feb 26, 2024 · 1 comment
Open

Define a mechanism to log data as explicitly invalid #5293

jleibs opened this issue Feb 26, 2024 · 1 comment
Labels
🔩 data model enhancement New feature or request

Comments

@jleibs
Copy link
Member

jleibs commented Feb 26, 2024

Compared to "Clearing" data, which has semantics of "this data, which was valid, has now expired" sometimes it is useful to instead log that a component itself represents invalid data.

For example, when an algorithm doesn't converge or outputs a bad value. We know a concrete timestamp when the invalid data was computed, and it's helpful to be able to represent this in the timeline explicitly.

We already have two ways of representing non-data:

  • [ ] -- the data batch itself is empty
  • [ Null ] the data itself is unset

We need to think through if one of these can be unambiguously used in the case of "invalid" vs "unset" data, or if we would need to introduce an alternative parallel mechanism.

@jleibs jleibs added enhancement New feature or request 🔩 data model labels Feb 26, 2024
@Wumpf
Copy link
Member

Wumpf commented Feb 26, 2024

Very related. if not redundant to

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🔩 data model enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants