diff --git a/analysis.qmd b/analysis.qmd index ca555d4..52a0308 100644 --- a/analysis.qmd +++ b/analysis.qmd @@ -114,6 +114,7 @@ While the Scoping and Design stages identified and described risks and uncertain [Black box models](definitions_and_key_concepts.qmd/#black_box) such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. Assurance activities during the Analysis stage: + * may include performance testing in a live environment and * should include the verification steps set out in the Design Phase * should include validation and verification of automatic tests to ensure the model behave as expected diff --git a/analytical_lifecycle.qmd b/analytical_lifecycle.qmd index 769a539..7e1f2cb 100644 --- a/analytical_lifecycle.qmd +++ b/analytical_lifecycle.qmd @@ -91,7 +91,7 @@ Work in this stage can vary considerably depending on the commission, impact, ap After analysis results have been determined to meet the requirements, they are formally approved for dissemination during sign-off. Sign-off includes confirmation that the commission was met, documentation and evidence was captured, and appropriate assurance was conducted. This approval may be phased as work develops and insights are produced. -### Maintenance and continuous review +## Maintenance and continuous review The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made whether *The inputs used remain appropriate *The initial communication methods remain the best way to deliver the information diff --git a/definitions_and_key_concepts.qmd b/definitions_and_key_concepts.qmd index 108946a..9c2c38e 100644 --- a/definitions_and_key_concepts.qmd +++ b/definitions_and_key_concepts.qmd @@ -163,7 +163,7 @@ The AQuA book defines the following roles: See [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details. -## Third party +## Third party {.unnumbered} Any individual, or group of individuals that is not a member of the same group as the those commissioning analysis. E.g. working for a different government department, a different function or an outside company. ## Uncertainty {.unnumbered} diff --git a/proportionality.qmd b/proportionality.qmd index f81e242..33b3b35 100644 --- a/proportionality.qmd +++ b/proportionality.qmd @@ -147,7 +147,7 @@ Proportionate assurance of externally commissioned work is just as important as The difference will be in ensuring the assessment of risks and the applied mitigations are fully understood by the commissioner. -## Artificial intelligence and business risk +## Black-box models and business risk -Increasingly analysis may be underpinned by Artificial Intelligence (AI). With AI-informed analysis the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible. +Increasingly analysis may be underpinned by artificial intelligence or other forms of black-box models. With such models, the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible.