Re-rendered site

.quarto/cites/index.json

@@ -1 +1 @@
-{"accessibility_statement.qmd":[],"proportionality.qmd":[],"definitions_and_key_concepts.qmd":["glover2014","glover2014"],"foreword.qmd":[],"analytical_lifecycle.qmd":[],"quality_assurance_culture.qmd":[],"design.qmd":[],"forward.qmd":[],"engagement_and_scoping.qmd":[],"additional_resources.qmd":[],"analysis.qmd":[],"intro.qmd":[],"index.qmd":[],"improving_the_book.qmd":[],"summary.qmd":[],"references.qmd":["knuth84"],"delivery_and_communication.qmd":[]}
+{"accessibility_statement.qmd":[],"additional_resources.qmd":[],"summary.qmd":[],"design.qmd":[],"references.qmd":["knuth84"],"index.qmd":[],"definitions_and_key_concepts.qmd":["glover2014","glover2014"],"analysis.qmd":[],"intro.qmd":[],"forward.qmd":[],"analytical_lifecycle.qmd":[],"delivery_and_communication.qmd":[],"improving_the_book.qmd":[],"foreword.qmd":[],"quality_assurance_culture.qmd":[],"proportionality.qmd":[],"engagement_and_scoping.qmd":[]}

.quarto/xref/28f6248c

@@ -1 +1 @@
-{"headings":["who-is-the-aqua-book-for","why-should-i-pay-attention-to-this-guidance","how-to-use-this-book"],"entries":[],"options":{"chapters":true}}
+{"entries":[],"headings":["who-is-the-aqua-book-for","why-should-i-pay-attention-to-this-guidance","how-to-use-this-book"],"options":{"chapters":true}}

.quarto/xref/8968456b

@@ -1 +1 @@
-{"options":{"chapters":true},"headings":[],"entries":[]}
+{"entries":[],"options":{"chapters":true},"headings":[]}

.quarto/xref/a0b88893

@@ -1 +1 @@
-{"options":{"chapters":true},"entries":[],"headings":["analysis","assurance","assurance-activities","artificial-intelligence","black-box-models","business-critical-analysis","change-control","documentation","specification-documentation","design-documentation","assumptions-log","decisions-log","data-log","user-technical-documentation","assurance-statement","materiality","multi-use-models","principles-of-analytical-quality-assurance","quality-analysis","reproducible-analytical-pipelines","roles-and-responsibilities","third-party","uncertainty","validation","verification","version-control"]}
+{"headings":["analysis","assurance","assurance-activities","artificial-intelligence","black-box-models","business-critical-analysis","change-control","documentation","specification-documentation","design-documentation","assumptions-log","decisions-log","data-log","user-technical-documentation","assurance-statement","materiality","multi-use-models","principles-of-analytical-quality-assurance","quality-analysis","reproducible-analytical-pipelines","roles-and-responsibilities","third-party","uncertainty","validation","verification","version-control"],"entries":[],"options":{"chapters":true}}

docs/analysis.html

@@ -275,7 +275,7 @@ <h3 data-number="8.2.2" class="anchored" data-anchor-id="the-analysts-responsibi
 <ul>
 <li><p>The Analyst shall follow the conduct the verification and validation activities that were designed as part of the analytical plan in <a href="design.html#the-design-stage">the design stage</a>. They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.</p></li>
 <li><p>When the analysis includes coding, the Analyst shall proportionately follow <a href="#assurance-of-code">best practice for code development</a>.</p></li>
-<li><p>The Analyst shall produce documentation of the data and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.</p></li>
+<li><p>The Analyst shall produce documentation of the data (see <a href="https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework">The Government Data Quality Framework</a>) and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.</p></li>
 <li><p>The Analyst shall document any changes to the analytical plan in a proportionate manner.</p></li>
 <li><p>The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner’s needs or whether there are any new requirements.</p></li>
 </ul>
@@ -308,7 +308,7 @@ <h3 data-number="8.3.1" class="anchored" data-anchor-id="verification-and-valida
 </ul>
 <p>Validation refers to testing whether the product meets the requirements of users. Hence, it is important to involve the users in the process. <a href="https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_">Methods for validation</a> include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility.</p>
 <p>Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. Techniques that may be useful in validation of models are reviewed by <a href="https://www.informs-sim.org/wsc11papers/016.pdf">Sargent (2011)</a>.</p>
-<p>The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the specification document to assure that the analysis meets the specification.</p>
+<p>The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the <a href="delivery_and_communication.html#specification-documentation">specification document</a> to assure that the analysis meets the specification.</p>
 </section>
 <section id="data-validity-and-data-considerations" class="level3" data-number="8.3.2">
 <h3 data-number="8.3.2" class="anchored" data-anchor-id="data-validity-and-data-considerations"><span class="header-section-number">8.3.2</span> Data validity and data considerations</h3>
@@ -337,9 +337,9 @@ <h2 data-number="8.4" class="anchored" data-anchor-id="documention-in-the-analys
 <ul>
 <li>Maintain appropriate records of the work;</li>
 <li>Fully document any code following agreed standards;</li>
-<li>Log the data, assumptions and inputs used in the analysis, and decisions made (see <a href="#definitions_and_key_concepts/#def_documentation">documentation</a>);</li>
+<li>Log the data, assumptions and inputs used in the analysis, and decisions made (see <a href="#definitions_and_key_concepts.html/#documentation">documentation</a>);</li>
 <li>Record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its impact on the analysis;</li>
-<li>Produce <a href="#definitions_and_key_concepts/#def_documentation">user and technical documentation</a>).</li>
+<li>Produce <a href="#definitions_and_key_concepts.html#user-technical-documentation">user and technical documentation</a>. For modelling, the Analyst may include a model map that describes data flows and transformations.</li>
 </ul>
 </section>
 <section id="treatment-of-uncertainty-in-the-analysis-stage" class="level2" data-number="8.5">

docs/analytical_lifecycle.html

@@ -209,6 +209,7 @@ <h2 id="toc-title">Table of contents</h2>
   <li><a href="#delivery-communication-and-sign-off" id="toc-delivery-communication-and-sign-off" class="nav-link" data-scroll-target="#delivery-communication-and-sign-off"><span class="header-section-number">5.2.4</span> Delivery, communication and sign-off</a></li>
   <li><a href="#maintenance-and-continuous-review" id="toc-maintenance-and-continuous-review" class="nav-link" data-scroll-target="#maintenance-and-continuous-review"><span class="header-section-number">5.2.5</span> Maintenance and continuous review</a></li>
   </ul></li>
+  <li><a href="#urgent-analysis" id="toc-urgent-analysis" class="nav-link" data-scroll-target="#urgent-analysis"><span class="header-section-number">5.3</span> Urgent analysis</a></li>
   </ul>
 </nav>
     </div>
@@ -338,9 +339,13 @@ <h3 data-number="5.2.4" class="anchored" data-anchor-id="delivery-communication-
 <h3 data-number="5.2.5" class="anchored" data-anchor-id="maintenance-and-continuous-review"><span class="header-section-number">5.2.5</span> Maintenance and continuous review</h3>
 <p>The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made whether <em>The inputs used remain appropriate </em>The initial communication methods remain the best way to deliver the information <em>Any software relied on continues to be supported and up to date </em>The model continues to be calibrated appropriately (this is particularly important for <a href="./definitions_and_key_concepts.html#black_box_models">black box models</a>)</p>
 <p>Additionally, a robust version control process should be in place to ensure any changes to the analysis are appropriately assured.</p>
+</section>
+</section>
+<section id="urgent-analysis" class="level2" data-number="5.3">
+<h2 data-number="5.3" class="anchored" data-anchor-id="urgent-analysis"><span class="header-section-number">5.3</span> Urgent analysis</h2>
+<p>In some cases there may be a need for urgent analysis that cannot follow all the steps in this guide i.e.&nbsp;where the need for analysis outweighs the risk of poor quality. In this case analysts should follow the <a href="https://www.gov.uk/government/publications/urgent-data-quality-assurance-guidance/urgent-data-quality-assurance-guidance">Urgent data quality assurance guidance</a>.</p>
 
 
-</section>
 </section>
 
 </main> <!-- /main -->

docs/definitions_and_key_concepts.html

@@ -365,7 +365,7 @@ <h3 class="unnumbered anchored" data-anchor-id="data-log">Data log</h3>
 </section>
 <section id="user-technical-documentation" class="level3 unnumbered">
 <h3 class="unnumbered anchored" data-anchor-id="user-technical-documentation">User / technical documentation</h3>
-<p>All analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial.</p>
+<p>All analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. For modelling, the Analyst may include a model map that describes data flows and transformations.</p>
 </section>
 <section id="assurance-statement" class="level3 unnumbered">
 <h3 class="unnumbered anchored" data-anchor-id="assurance-statement">Assurance statement</h3>

docs/design.html

@@ -202,7 +202,7 @@ <h2 id="toc-title">Table of contents</h2>
   <ul>
   <li><a href="#introduction-and-overview" id="toc-introduction-and-overview" class="nav-link active" data-scroll-target="#introduction-and-overview"><span class="header-section-number">7.1</span> Introduction and overview</a>
   <ul class="collapse">
-  <li><a href="#the-design-stage" id="toc-the-design-stage" class="nav-link" data-scroll-target="#the-design-stage"><span class="header-section-number">7.1.1</span> The design stage</a></li>
+  <li><a href="#the-analytical-plan" id="toc-the-analytical-plan" class="nav-link" data-scroll-target="#the-analytical-plan"><span class="header-section-number">7.1.1</span> The analytical plan</a></li>
   </ul></li>
   <li><a href="#roles-and-responsibilities-in-the-design-stage" id="toc-roles-and-responsibilities-in-the-design-stage" class="nav-link" data-scroll-target="#roles-and-responsibilities-in-the-design-stage"><span class="header-section-number">7.2</span> Roles and responsibilities in the design stage</a>
   <ul class="collapse">
@@ -258,8 +258,8 @@ <h1 class="title"><span class="chapter-number">7</span>&nbsp; <span class="chapt
 <section id="introduction-and-overview" class="level2" data-number="7.1">
 <h2 data-number="7.1" class="anchored" data-anchor-id="introduction-and-overview"><span class="header-section-number">7.1</span> Introduction and overview</h2>
 <p>The design stage is where the Analyst translates the scope for the analysis agreed with the Commissioner into an actionable analytical plan. This chapter sets out recommended practices around designing the analysis and associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design, and design of multi-use and AI models.</p>
-<section id="the-design-stage" class="level3" data-number="7.1.1">
-<h3 data-number="7.1.1" class="anchored" data-anchor-id="the-design-stage"><span class="header-section-number">7.1.1</span> The design stage</h3>
+<section id="the-analytical-plan" class="level3" data-number="7.1.1">
+<h3 data-number="7.1.1" class="anchored" data-anchor-id="the-analytical-plan"><span class="header-section-number">7.1.1</span> The analytical plan</h3>
 <p>The development of the analytical plan should consider:</p>
 <ul>
 <li>Methodology for producing results, including the treatment of uncertainty;</li>
@@ -272,7 +272,7 @@ <h3 data-number="7.1.1" class="anchored" data-anchor-id="the-design-stage"><span
 <li>Communication between stakeholders;</li>
 <li>Verification and validation procedures during the project lifetime;</li>
 <li>Documentation to be delivered;</li>
-<li>Process for updating the analytical Plan;</li>
+<li>Process for updating the analytical plan;</li>
 <li>Process for ongoing review and maintenance of models, including reviewing inputs and calibrations and ensuring that software relied on continues to be supported and up to date;</li>
 <li>Ethics;</li>
 <li>Reporting;</li>
@@ -310,10 +310,11 @@ <h3 data-number="7.2.1" class="anchored" data-anchor-id="the-commissioners-respo
 <h3 data-number="7.2.2" class="anchored" data-anchor-id="the-analysts-responsibilities-during-the-design-stage"><span class="header-section-number">7.2.2</span> The Analyst’s responsibilities during the design stage</h3>
 <p>The Analyst should:</p>
 <ul>
-<li>develop the method and plan to address the Commissioner’s needs,</li>
-<li>develop the assurance requirements will be and</li>
-<li>plan in sufficient time for the assurance activity.</li>
-<li>document the analytical plan in a proportionate manner.</li>
+<li>develop the method and plan to address the Commissioner’s needs;</li>
+<li>establish assurance requirements;</li>
+<li>develop the plan for proportionate verification and validation - see <a href="https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf">National Audit Office Framework to review models</a>;</li>
+<li>plan in sufficient time for the assurance activity;</li>
+<li>document the analytical plan in a proportionate manner; and,</li>
 <li>follow any organisation governance procedures for project design.</li>
 </ul>
 </section>
@@ -340,7 +341,8 @@ <h2 data-number="7.3" class="anchored" data-anchor-id="assurance-activities-in-t
 </section>
 <section id="documention-in-the-design-stage" class="level2" data-number="7.4">
 <h2 data-number="7.4" class="anchored" data-anchor-id="documention-in-the-design-stage"><span class="header-section-number">7.4</span> Documention in the design stage</h2>
-<p>The design process should be documented in a proportionate manner. A design document that records the analytical plan should be produced by the Analyst, and signed-off by the Commissioner. The design document may be reviewed by the Assurer.</p>
+<p>The design process should be documented in a proportionate manner. A design document that records the <a href="#the-analytical-plan">analytical plan</a> should be produced by the Analyst and signed-off by the Commissioner. The design document may be reviewed by the Assurer.</p>
+<p>For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage.</p>
 <p>It is best practice to use formal version control to track changes in the design document.</p>
 </section>
 <section id="treatment-of-uncertainty-in-the-design-stage" class="level2" data-number="7.5">
@@ -353,13 +355,13 @@ <h2 data-number="7.6" class="anchored" data-anchor-id="black-box-models-and-the-
 <p>This <a href="https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance">guidance</a> outlines considerations for the design of AI models, including risk assessment, impact assessment, bias audits and compliance audits.</p>
 <p>In the Design of AI/ML models, the Analyst should:</p>
 <ul>
-<li>define the situation they wish to model</li>
-<li>the prediction they wish to make</li>
-<li>the data that could be used to make the prediction</li>
-<li>carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach</li>
-<li>consider how to separate the data for the design and testing of models - it’s usual to design a model with a fraction of the data and then test it with the data that was not used in the design</li>
-<li>consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions</li>
-<li>consider whether to refer the model to their ethics committee, or similar</li>
+<li>define the situation they wish to model;</li>
+<li>the prediction they wish to make;</li>
+<li>the data that could be used to make the prediction;</li>
+<li>carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach;</li>
+<li>consider how to separate the data for the design and testing of models - it’s usual to design a model with a fraction of the data and then test it with the data that was not used in the design;</li>
+<li>consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions; and,</li>
+<li>consider whether to refer the model to their ethics committee, or a similar group - see <a href="https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020">Data Ethics Framework - Guidance</a>.</li>
 </ul>
 </section>
 <section id="multi-use-models-and-the-design-stage" class="level2" data-number="7.7">