scoringutils user feedback

[nikosbosse]

We'd like to hear your thoughts and ideas on scoringutils!

• What are you using scoringutils for?
• What are current pain points?
• What features / functionality do you most like?
• What features would you like to see?
• What ideas / visions do you have for scoringutils?
• Anything else you'd like to tell us.

Just post your thoughts and ideas here. All contributions welcome! If you hesitate whether or not to post something, please post it.

[elray1]

I just made a quick pass at a small forecast evaluation using scoringutils. Overall, using the package is much easier than I remembered it being a couple of years ago, last time I worked with it directly. Thanks for your work on it!! Here are a few misc. comments, all relatively minor.

• it would be nice if the names of columns providing the true_value, prediction, sample, quantile, and model were user-supplied arguments to the score function. I'd rather not have to rename columns in my data set.
• In docs for score: "For scoring integer and continuous forecasts a ‘sample’ column is needed" -- this is related to forecasts represented as a sample from a forecast distribution, but that's not clear from the text here.
• In docs for score: "‘quantile’: quantile to which the prediction corresponds." This is consistent with terminology that we adopted for the covid hub that I have come to regret. The quantity here is a probability level, and the prediction is a quantile at that probability level. In other words, if you have a distribution with cdf F and you evaluate the inverse of that cdf at a probability level p \in [0, 1], then x = F^{-1}(p) is a quantile. I believe that the column being documented here is the probability level p, not the quantile x (which is what is in the prediction column). I suggest renaming this column to something like quantile_level.
• Similarly, but with less strong feeling about the matter, I'd suggest renaming sample to sample_index or something like that. The sample is the number in the prediction column, the thing in what's currently called the sample column is a label or identifier for the sample.
• I wasn't totally sure how to interpret the "raw" outputs of score, before summarizing them. It might be nice to clearly define what those columns are. Maybe that is somewhere and I just missed it.
• Can I get at one-sided quantile coverage rates using this package? I didn't see a mechanism to do that.

[nikosbosse]

Thanks a lot, @elray1 ! These seem very useful.
We already decided to change the name from sample to sample_id and will follow your suggestion to rename quantile to quantile_level.
I'll create new issues based on your suggestions in the next couple of days.
In what kind of format are you interested in one-sided quantile coverage? Do you want it mainly for plotting, or are you intending to report it as a number for some quantiles?

I'm asking because we're currently planning to redesign score() such that it only returns one score per forecast (where at the moment it returns one score per quantile). This feels more natural in many instances, but also means that we have to rethink one-sided quantile coverage a bit.

[elray1]

I also just wrote up some thoughts about what it would take to create a link between hubverse and scoringutils here.