Dashboard showing Codespaces usage

[lucyb]

Dependent on #42

We are recording daily metrics about Codespaces usage, so we should make those available on a dashboard within Grafana. This will tell us if people are using Codespaces and how they're using it.

This text is based on this comment by Iain
We can derive the number of users that are developing their study code in Codespaces, over time. We hope this number is non-zero (someone is using a Codespace 🤞🏻) and is similar to the rate at which new studies are approved, albeit with a lag. For example, if a new study is approved every week for four weeks, then we hope that the number of users that are developing their study code in Codespaces will (eventually) increase to four. Knowing the user and repo will help us in the observation stage of the initiative: We will know who to ask about what, when we want to know about the experience of developing study code in Codespaces.

It would be useful to derive the distribution of time deltas between when a Codespace was created and when it was last used, as the distribution could help us calibrate our usage policy. For example, if the peak of the distribution was consistently low, then we could infer an ephemeral pattern of use. The current maximum retention period of 14 days would appropriate. However, if the peak of the distribution approached 14 days, then we should reevaluate the current maximum retention period, or at least our communication of it, to prevent users from loosing their work.

It would be useful to derive the distribution of time deltas between when a repo was created and when the associated Codespace was last used. We think this distribution will have positive skew -- that is, a large number of small deltas -- as this would demonstrate that new study code is being edited in Codespaces. However, we're very interested in repos to the right of the distribution, as these would demonstrate that old study code is being edited in Codespaces. These studies may be larger, more complex, and depend on older versions of our tools, and may help us address any challenges associated with developing study code in Codespaces sooner rather than later.

I expect this work will take around one day.

Acceptance criteria

• A Grafana dashboard that has one or more graphs showing Codespaces usage.
• The dashboard is accessible by the REX Team and Catherine.
• Catherine (or Lucy, if she's not available) has reviewed the dashboard.

[iaindillingham]

This depends on #42 being merged and deployed. @Jongmassey mentioned that the failed silently, so there isn't (yet) any data from which to create a dashboard.

[lucyb]

I've created a starter dashboard here: https://dashboards.opensafely.org/d/bdo2ig1sxis5ca/codespaces?orgId=1

[iaindillingham]

I've updated Lucy's starter dashboard, which now displays:

• a histogram and frequency distribution of Number of Codespaces by Lifespan (Days)
• a time series of Number of Codespaces Created by Day
• a table of Recent Use

I showed the tech team this dashboard in a recent weekly tech team meeting. @CLStables' feedback prompted me to add the table of Recent Use. I also showed co-pilots the dashboard in a recent co-pilot deep dive.

It's going to take more time than I anticipated to derive the distribution of time deltas between when a repo was created and when an associated Codespace was last used, so I've created ebmdatalab/metrics#187.

[iaindillingham]

@lucyb kindly reviewed the dashboard,¹ so I shall close this issue.

Footnotes

1. https://bennettoxford.slack.com/archives/C069SADHP1Q/p1718374014201719 ↩