Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance metadata.json to use named thresholds and set threshold bounds #50

Open
gbowlin opened this issue Jul 18, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@gbowlin
Copy link
Contributor

gbowlin commented Jul 18, 2024

Is your feature request related to a problem? Please describe

Default score threshold values are read from metadata.json which is loaded from the config.yml's. When loaded, we look at the root thresholds node which is a list of numeric values (unsorted)

This means when we plot model performance, we just sort these high to low, and treat the highest threshold as "Red" and the second highest as "Orange", and we don't provide a name for the threshold like "Telephone Call", "Rapid Response", or "Screening"

This also means that we can only sport one global threshold, even though we could support more than one model output (score)

Describe the solution you'd like

Ideally metadata.json is improved to support different thresholds for different score names (if a model can predict more than one score) and different thresholds for different model outputs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant