05. How do you document your trust in an open source solution to satisfy a third-party inquiry?

[MichaelRimler]

e.g., if an organization is GCP audited on the integrity of data handling by a regulatory body?

[MichaelRimler]

Risk metric score: https://pharmar.github.io/riskmetric/
"The risk of using an R package is evaluated based on a number of metrics meant to evaluate development best practices, code documentation, community engagement and development sustainability."
Run your own tests after install - document results

valtools: https://phuse-org.github.io/valtools/
"valtools helps automate the validation of R packages used in clinical research and drug development: It provides useful templates and helper functions for tasks that arise during project set up and development of the validation framework."

[epijim]

I'd argue the most important validation hub effort would be the regulatory repo, as that would help make open source the process of assessing what trust we place in a package.

The PHUSE OS guidance touches on above, as well as other efforts. And also helps walkthrough what a user should think about when trying to understand the lifecycle and community behind a package: https://phuse-org.github.io/E2E-OS-Guidance/using.html

[epijim]

Also from the Roche perspective, there is a bias to theValidator over valtools, as we believe an R package inherently contains the information for validation already (assuming good docs and unit tests), thanks the machine readable roxygen tags and testthat unit tests. We also know though from the 4 companies that shared their process for assessing R packages, each company currently takes a very different approach.

Again - hoping the regulatory repo working group solves what is mostly arbitrary differences and harmonises the process.

[andyofsmeg]

I agree that an R package already contains the relevant information. For example, I would argue that in the context of general development/analysis user requirements and functional requirements are essentially the same thing (if I use the mean function then my requirement is to be able to calculate a mean). The argument is that R help files are structured such that the description contains the requirements and the tests test against these requirements.

That's assuming 'validation' is required in that sense.

[MichaelRimler]

Do people have any webinars/presentations to link on this one?

[rossfarrugia]

https://www.youtube.com/watch?v=ZfZpypQ1jSM
https://www.youtube.com/watch?v=xksxuvXVimM

[andyofsmeg]

You at least need the following with appropriately tracked review/sign-off as per internal QA policies:

• An overview that explains the overall approach (why user requirements don't make sense for the way the language is used, the context of additional QC processes, references to industry thinking like the R Validation Hub white paper, ...) This would need to cover topics like intended use and the approach to package dependencies.
• A document explaining why one might be willing to accept core R and perhaps the Tidyverse packages (along the lines of, "we've review these documents from R Foundation and Posit and accept that they follow good practice").
• Some kind of assessment report for each additional package. This probably includes some human-written interpretation in addition to any automated metrics. The R Validation Hub's Risk Assessment app provides a means to do this.

This concerns the review of the language and packages. Internal processes should govern additional documentation required to qualify/validate the environment.

You could potentially bypass human engagement for some packages, if they meet some risk threshold. The R Validation Hub invited several companies to share their experiences via case studies. Several were using the riskmetric score either as a way to quickly pass through the obvious (packages like dplyr which score very highly) or as a guide in an overall assessment. I think it's theoretically fine to rely on a score for those that score 'low risk' (so long as low risk is clearly defined), although the riskmetric package itself still has a lot of open issues to address (91 as I look today), including some important metrics that are still to be added. So as things stand my advice would be to use riskmetric to gather info but don't rely too heavily on the score itself and let a human review the data it gathers. This is where the risk assessment app comes in. Or else get involved in the development of the package and improve it!

[MikeKSmith-Pfizer]

Comments from PhUSE EU Connect OSTCDA workshop:

• Can industry work together to validate packages? i.e. can we reach consensus? How?
• Each company is likely to be validating a broadly similar set of packages. Can we share the effort?
• A common validation repository (e.g. Regulatory R Repository - through R Consortium R Validation working group) would help.
• Need agreement on the set of packages
• Need accountability for when something goes wrong
• Sustainability for this plus support would give trust in the packages hosted
• Sharing "validated" packages would allow us to focus on more "niche" packages which perhaps need more testing, scrutiny
• If there are issues during validation, are we looking under the hood of the packages and doing analysis to identify causes?
• We have a lot of metrics / attributes defined but we don't have enough data yet to say whether those metrics / attributes are good or bad at predicting quality.
• Takes some level of expertise to be able to critique and review packages (even to review results of validation testing). There are not enough people with this level of expertise within the industry.
• Different Pharma companies are sharing their lists of of validated packages, which is a big step forward.

[MikeKSmith-Pfizer]

For consideration... (my own views)

Different companies are likely to have different weightings (for the package attributes) to create their own "risk scores" for packages. This may make it difficult to contribute a regulatory R repository with a definitive score. But maybe we could work together to gather the attributes of packages and provide this as metadata for packages of interest. For example: code coverage, date since last version, package license type are attributes that can then be "scored". But instead of each party pulling this information to create our scores, if this was readily available as metadata in a repository then it would be easier to just apply our own scoring rules. If we can agree a common format and source for that metadata.