diff --git a/.quarto/cites/index.json b/.quarto/cites/index.json index 72fe2ca..3cc6b63 100644 --- a/.quarto/cites/index.json +++ b/.quarto/cites/index.json @@ -1 +1 @@ -{"additional_resources.qmd":[],"references.qmd":["knuth84"],"summary.qmd":[],"intro.qmd":[],"forward.qmd":[],"engagement_and_scoping.qmd":[],"analytical_lifecycle.qmd":[],"definitions_and_key_concepts.qmd":["glover2014","glover2014"],"design.qmd":[],"delivery_and_communication.qmd":[],"quality_assurance_culture.qmd":[],"foreword.qmd":[],"proportionality.qmd":[],"accessibility_statement.qmd":[],"index.qmd":[],"analysis.qmd":[]} +{"delivery_and_communication.qmd":[],"analysis.qmd":[],"intro.qmd":[],"foreword.qmd":[],"summary.qmd":[],"definitions_and_key_concepts.qmd":["glover2014","glover2014"],"index.qmd":[],"quality_assurance_culture.qmd":[],"references.qmd":["knuth84"],"design.qmd":[],"proportionality.qmd":[],"analytical_lifecycle.qmd":[],"forward.qmd":[],"engagement_and_scoping.qmd":[],"additional_resources.qmd":[],"accessibility_statement.qmd":[],"improving_the_book.qmd":[]} diff --git a/.quarto/idx/definitions_and_key_concepts.qmd.json b/.quarto/idx/definitions_and_key_concepts.qmd.json index 99b5863..25a70c3 100644 --- a/.quarto/idx/definitions_and_key_concepts.qmd.json +++ b/.quarto/idx/definitions_and_key_concepts.qmd.json @@ -1 +1 @@ -{"title":"Definitions and key concepts","markdown":{"headingText":"Definitions and key concepts","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nThis chapter sets out definitions and key concepts that are used throughout the rest of the book. \n\n## Analysis {.unnumbered}\n\nAnalysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. \n\nExamples of types of analysis that are frequently encountered in government are: \n\n* actuarial \n* data science \n* economic \n* financial \n* geographical \n* operational research \n* scientific, technical and engineering research \n* statistical \n* social research \n\n## Assurance {.unnumbered}\n\nAnalytical assurance is the process and set of practices to ensure that the analysis is fit for purpose. \n\n## Assurance activities {.unnumbered}\n\nAssurance activities are any actions carried out in order to validate and verify analysis. \n\nFor example: \n\n* analyst testing \n* peer review \n* reconciliation of results to independent sources \n\n## Artificial Intelligence {.unnumbered}\n\nArtificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [black box models](#black_box_models)\n\n\n## Black box models {.unnumbered}\n\nBlack box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future. \n\n\n## Business critical analysis {.unnumbered}\n\nBusiness critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal impact for the public sector. \n\nThe first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM). \n\n## Change control {.unnumbered}\n\nChange control is the set of processes followed when changes are made to a piece of analysis. For example, authorising and accepting changes, version numbering, documentation, assurance of changes. \n\n## Documentation {.unnumbered}\n\n### Specification documentation {.unnumbered}\n\nSpecifications capture initial engagements with the commissioner. They describe the question, the context, and any boundaries of the analysis. This provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner.\n\n### Design documentation {.unnumbered}\n\nDesign documents describe the analytical plan, including the methodology, inputs, and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the Analytical Assurer to verify whether the analysis meets the specified requirements. For more information on the design documentation, see the [Design](design.qmd) chapter.\n\n### Assumptions log {.unnumbered}\n\nA register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its impact and reliability and set out when it was made, why it was made, who made it and who signed it off.\n\n### Decisions log {.unnumbered}\n\nA register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off.\n\n### Data log {.unnumbered}\n\nA register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. \n\n\n### User / technical documentation {.unnumbered}\n\nAll analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. \n\n### Assurance statement {.unnumbered}\n\nA brief description of the analytical assurance that have been performed to assure the analysis. The statement should refer to known limitations and conditions associated with the analysis.\n\n::: {.callout-tip}\n# Example of publishing quality assurance tools\nThe Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensureanalysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. \n\n:::\n\n## Materiality {.unnumbered}\n\n[Materiality](https://en.wikipedia.org/wiki/Materiality_(auditing)) is a concept or convention in auditing and accounting relating to the importance of a feature. Information is said to be material if omitting it or misstating it could influence decisions that users make. Materiality is \"an entity-specific aspect of relevance, based on size, magnitude or both\".\n\n## Multi-use models {.unnumbered}\n\nSome models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**. \n\nOften, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area. \n\n## Principles of analytical quality assurance {.unnumbered}\n\nNo single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. However, the following principles support commissioning and production of fit-for-purpose analysis: \n\n**Proportionate:** Quality assurance effort should be appropriate to the risk associated with the intended use of the analysis and the complexity of the analytical approach. These risks include financial, legal, operational and reputational impacts. More details can be found in chapter [3] \n\n**Assurance throughout development:** Quality assurance should be considered throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs. More details on the analysis life cycle can be seen in chapter [5]. \n\n**Verification and validation:** Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It should also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation). Validation and verification are covered in more depth in chapters [5-9]. \n\n**Accept that uncertainty is inherent** in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty . Further support can be found in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io) \n\n**Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. \n\n::: {.callout-tip collapse=\"true\"}\n### RIGOUR\nThroughout all the stages of an analytical project, the analyst should ask questions of their own analysis. The helpful mnemonic \"RIGOUR\" may assist:\n\n* **R**epeatable\n* **I**ndependent\n* **G**rounded in reality\n* **O**bjective\n* **U**ncertainty-managed\n* **R**obust\n\n**Repeatable:** For an analytical process to be considered valid we might reasonably expect that the analysis produces the same outputs for the same inputs and constraints. Different analysts might approach the analytical problem in different ways, while methods might include randomised processes. In such cases, exact matches are not guaranteed or expected. Taking this into account, repeatability means that if an approach is repeated the results should be as expected. \n\n**Independent:** Analysis should be free of prejudice or bias. Care should be taken to balance views appropriately across all stakeholders and experts. \n\n**Grounded in reality:** Quality analysis takes the Commissioner and Analyst on a journey as views and perceptions are challenged and connections are made between the analysis and its real consequences. Connecting with reality like this guards against failing to properly grasp the context of the problem that is being analysed. \n\n**Objective:** Effective engagement and suitable challenge reduce the risk of bias and enables the Commissioner and the Analyst to be clear about the interpretation of results. \n\n**Uncertainty-managed:** Uncertainty is identified, managed and communicated throughout the analytical process. \n\n**Robust:** Analytical results are error free in the context of residual uncertainty and accepted limitations that make sure the analysis is used appropriately. \n\n:::\n\n## Quality analysis {.unnumbered}\n\nQuality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its impact, adequately communicated, documented and accepted by its commissioners. \n\n## Reproducible analytical pipelines {.unnumbered}\n\n[Reproducible Analytical Pipelines (RAPs)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) are automated analytical processes. They incorporate elements of software engineering best practice to ensure that the pipelines are reproducible, auditable, efficient, and high quality.\n\n## Roles and responsibilities {.unnumbered}\n\nThe AQuA book defines the following roles:\n\n* **Commissioner**\n* **Analyst** \n* **Assurer** \n* **Approver** \n\nSee [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details.\n\n## Third party\nAny individual, or group of individuals that is not a member of the same group as the those commissioning analysis. E.g. working for a different government department, a different function or an outside company.\n\n## Uncertainty {.unnumbered}\n\nThe outcome of a decision is never known perfectly in advance. For each option within analysis, a range of real outcomes is possible: the outcome is uncertain. \n\n::: {.callout-note}\n### Defining uncertainty\n\n[Wikipedia defines uncertainty](https://en.wikipedia.org/wiki/Uncertainty) as referring to [epistemic](https://en.wikipedia.org/wiki/Epistemology) situations involving imperfect or unknown information. It applies to predictions of future events, to physical measurements that are already made, or to the unknown. \n\n:::\n\nThere are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will impact the analytical approach and assurance activities required. \n\nThe [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty.\n\n## Validation {.unnumbered}\n\nEnsuring the analysis meets the needs of its intended users and the intended use environment. See @glover2014 for more information.\n\n## Verification {.unnumbered}\n\nEnsuring the analysis meets it specified design requirements. See @glover2014 for more information.\n\n## Version control {.unnumbered}\n\nIt is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom. \n\n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"definitions_and_key_concepts.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file +{"title":"Definitions and key concepts","markdown":{"headingText":"Definitions and key concepts","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nThis chapter sets out definitions and key concepts that are used throughout the rest of the book. \n\n## Analysis {.unnumbered}\n\nAnalysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. \n\nExamples of types of analysis that are frequently encountered in government are: \n\n* actuarial \n* data science \n* economic \n* financial \n* geographical \n* operational research \n* scientific, technical and engineering research \n* statistical \n* social research \n\n## Assurance {.unnumbered}\n\nAnalytical assurance is the process and set of practices to ensure that the analysis is fit for purpose. \n\n## Assurance activities {.unnumbered}\n\nAssurance activities are any actions carried out in order to validate and verify analysis. \n\nFor example: \n\n* analyst testing \n* peer review \n* reconciliation of results to independent sources \n\n## Artificial Intelligence {.unnumbered}\n\nArtificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [black box models](#black_box_models)\n\n\n## Black box models {.unnumbered}\n\nBlack box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future. \n\n\n## Business critical analysis {.unnumbered}\n\nBusiness critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal impact for the public sector. \n\nThe first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM). \n\n## Change control {.unnumbered}\n\nChange control is the set of processes followed when changes are made to a piece of analysis. For example, authorising and accepting changes, version numbering, documentation, assurance of changes. \n\n## Documentation {.unnumbered}\n\n### Specification documentation {.unnumbered}\n\nSpecifications capture initial engagements with the commissioner. They describe the question, the context, and any boundaries of the analysis. This provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner.\n\n### Design documentation {.unnumbered}\n\nDesign documents describe the analytical plan, including the methodology, inputs, and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the Analytical Assurer to verify whether the analysis meets the specified requirements. For more information on the design documentation, see the [Design](design.qmd) chapter.\n\n### Assumptions log {.unnumbered}\n\nA register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its impact and reliability and set out when it was made, why it was made, who made it and who signed it off.\n\n### Decisions log {.unnumbered}\n\nA register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off.\n\n### Data log {.unnumbered}\n\nA register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. \n\n\n### User / technical documentation {.unnumbered}\n\nAll analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. \n\n### Assurance statement {.unnumbered}\n\nA brief description of the analytical assurance that have been performed to assure the analysis. The statement should refer to known limitations and conditions associated with the analysis.\n\n::: {.callout-tip}\n# Example of publishing quality assurance tools\nThe Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensureanalysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. \n\n:::\n\n## Materiality {.unnumbered}\n\n[Materiality](https://en.wikipedia.org/wiki/Materiality_(auditing)) is a concept or convention in auditing and accounting relating to the importance of a feature. Information is said to be material if omitting it or misstating it could influence decisions that users make. Materiality is \"an entity-specific aspect of relevance, based on size, magnitude or both\".\n\n## Multi-use models {.unnumbered}\n\nSome models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**. \n\nOften, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area. \n\n## Principles of analytical quality assurance {.unnumbered}\n\nNo single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. However, the following principles support commissioning and production of fit-for-purpose analysis: \n\n**Proportionate:** Quality assurance effort should be appropriate to the risk associated with the intended use of the analysis and the complexity of the analytical approach. These risks include financial, legal, operational and reputational impacts. More details can be found in chapter [3] \n\n**Assurance throughout development:** Quality assurance should be considered throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs. More details on the analysis life cycle can be seen in chapter [5]. \n\n**Verification and validation:** Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It should also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation). Validation and verification are covered in more depth in chapters [5-9]. \n\n**Accept that uncertainty is inherent** in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty . Further support can be found in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io) \n\n**Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. \n\n::: {.callout-tip collapse=\"true\"}\n### RIGOUR\nThroughout all the stages of an analytical project, the analyst should ask questions of their own analysis. The helpful mnemonic \"RIGOUR\" may assist:\n\n* **R**epeatable\n* **I**ndependent\n* **G**rounded in reality\n* **O**bjective\n* **U**ncertainty-managed\n* **R**obust\n\n**Repeatable:** For an analytical process to be considered valid we might reasonably expect that the analysis produces the same outputs for the same inputs and constraints. Different analysts might approach the analytical problem in different ways, while methods might include randomised processes. In such cases, exact matches are not guaranteed or expected. Taking this into account, repeatability means that if an approach is repeated the results should be as expected. \n\n**Independent:** Analysis should be free of prejudice or bias. Care should be taken to balance views appropriately across all stakeholders and experts. \n\n**Grounded in reality:** Quality analysis takes the Commissioner and Analyst on a journey as views and perceptions are challenged and connections are made between the analysis and its real consequences. Connecting with reality like this guards against failing to properly grasp the context of the problem that is being analysed. \n\n**Objective:** Effective engagement and suitable challenge reduce the risk of bias and enables the Commissioner and the Analyst to be clear about the interpretation of results. \n\n**Uncertainty-managed:** Uncertainty is identified, managed and communicated throughout the analytical process. \n\n**Robust:** Analytical results are error free in the context of residual uncertainty and accepted limitations that make sure the analysis is used appropriately. \n\n:::\n\n## Quality analysis {.unnumbered}\n\nQuality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its impact, adequately communicated, documented and accepted by its commissioners. \n\n## Reproducible analytical pipelines {.unnumbered}\n\n[Reproducible Analytical Pipelines (RAPs)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) are automated analytical processes. They incorporate elements of software engineering best practice to ensure that the pipelines are reproducible, auditable, efficient, and high quality.\n\n## Roles and responsibilities {.unnumbered}\n\nThe AQuA book defines the following roles:\n\n* **Commissioner**\n* **Analyst** \n* **Assurer** \n* **Approver** \n\nSee [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details.\n\n## Third party\nAny individual, or group of individuals that is not a member of the same group as the those commissioning analysis. E.g. working for a different government department, a different function or an outside company.\n\n## Uncertainty {.unnumbered}\n\nUncertainties are things that are not known, or are in a state of doubt, or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme, piece of analysis meeting its objectives.[^1]\n\n[^1]: https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf\n\nThere are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will impact the analytical approach and assurance activities required. \n\nThe [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty.\n\n## Validation {.unnumbered}\n\nEnsuring the analysis meets the needs of its intended users and the intended use environment. See @glover2014 for more information.\n\n## Verification {.unnumbered}\n\nEnsuring the analysis meets it specified design requirements. See @glover2014 for more information.\n\n## Version control {.unnumbered}\n\nIt is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom. \n\n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"definitions_and_key_concepts.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file diff --git a/.quarto/idx/index.qmd.json b/.quarto/idx/index.qmd.json index c9bcfa0..f68ff78 100644 --- a/.quarto/idx/index.qmd.json +++ b/.quarto/idx/index.qmd.json @@ -1 +1 @@ -{"title":"Preface","markdown":{"headingText":"Preface","headingAttr":{"id":"","classes":["unnumbered"],"keyvalue":[]},"containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nPlaceholder text","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"index.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file +{"title":"Preface","markdown":{"headingText":"Preface","headingAttr":{"id":"","classes":["unnumbered"],"keyvalue":[]},"containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\nPlaceholder text\n\n# Acknowledgements {.unnumbered}\nThe AQuA Book is the work of many authors from across the Government Analytical Functions. The original version of the book was compiled by the Quality Assurance Working Group set up after Sir Nicholas Macpherson's review of modelling in government. This revised edition of the book was produced by a task and finish group drawn from across the Government Analytical Functionsy. We would like to thank everybody who has given of their time and expertise to produce the revised edition and give a special mention to the committee who gave their time and expetise to this book:\n\n\n* Faye Clancy\t\n* Will England\n* Andrew Friedman\t\n* Nick Harris\n* Jordan Low\t\n* James McGlade\n* Ian Mitchell\n* Iris Oren\n* Adam Powell\n* Martin Ralphs\n* Philippa Robinson\n* Sarjeet Soni\n* Lorna Wilson\n* Rebecca Wodcke\n\nAlec Waterhouse\nDate Month Year\n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"index.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file diff --git a/.quarto/idx/intro.qmd.json b/.quarto/idx/intro.qmd.json index 454099f..7ef43d0 100644 --- a/.quarto/idx/intro.qmd.json +++ b/.quarto/idx/intro.qmd.json @@ -1 +1 @@ -{"title":"Introduction","markdown":{"headingText":"Introduction","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\nThe AQuA book provides guidance on producing quality analysis for government. It aims to support well informed decision making to deliver better outcomes and improve the lives of citizens.\n\nThe AQuA Book has made a significant contribution to the cultural change in assurance practices in government. It is about the process for assuring analytical evidence in all forms. It sets out the core framework for assuring all forms of analytical evidence. \n\nThe last version of the **Analytical Quality Assurance (AQuA) Handbook** was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then, assurance has become part of the fabric of good practice for developing evidence to support policy development, delivery and operational excellence. \n\nThe world of analysis has developed since we published the first edition of the AQuA Book. \n\nIncreasingly in our data driven world, insights from analysis underpin almost all policies and support operational excellence. This underlines the continuing importance of assuring our evidence. In parallel our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. Since then we have broadened the range of methods to include open-source software, machine learning and artificial intelligence. \n\nUsers of the AQuA book have pointed out some things we did not cover in the first edition and areas where guidance was unclear or insufficient. In this edition we have added guidance on: \n\n* multi-use models - large models used for many purposes with many stakeholders; \n* assuring \"black box\"[^1] analysis, including artificial intelligence; \n* development, maintenance and continuous review; \n* working with third parties such as contractors and academic groups and, \n* publishing models. \n\nWe provide improved guidance on what [a proportionate approach to assurance means](https://best-practice-and-impact.github.io/aqua_book_revision/proportionality.html) and have made the whole guide applicable to all types of analysis.\n\nThe AQuA Book describes what you need to do but not how to do it, although it does contain many worked examples. Large organisations will have their own processes and practices covering “the how”. \n\nFor those of you who do not work in places with bespoke guidance you will find a collection of helpful resources in chapter 10.\n\nThe AQuA Book is a key supporting guide for the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). This Standard refers extensively to the AQuA book and notes that \"detailed guidance on the analytical cycle and management of analysis, included in the Aqua Book should be followed.\"\n\nIt is also referred to by the [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020), the [Magenta Book](https://www.gov.uk/government/publications/the-magenta-book) and other Functional Standards, such as the [Finance Function](https://www.gov.uk/government/publications/government-finance-standards-page) Standards. \n\n\n## Who is the AQuA Book for?\n\nIn this edition we have tried to make our guidance relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose and not just about the checks after the analysis has been completed. \n\nWe would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision making in government. Our intended audience includes: \n\n* Users of analysis – helping you to get the most out of your commission; \n* Those who carry out analysis such as members of the government analytical professions, including:\n + operational researchers, statisticians and economists;\n + geographers;\n + finance professionals making spending forecasts;\n + actuaries;\n + social researchers carrying out qualitative research;\n + data scientists developing advanced analytics;\n + and anyone else carrying out analysis. \n* Senior leaders with an interest in analytical assurance. \n\n## Why should I pay attention to this guidance? \n\nHere are a few reasons why.\n\n* **Your analytical insights will be used for major decisions and operations.** You need to do your best to get them right, thus minimising the risk of being complicit in causing operational, business or reputational damage; \n* **Trust is hard to obtain but easy to lose.** A simple error that could have been prevented by assurance could lead to your and your team’s work being doubted; \n* **Prevention is better than cure.** Analysis is more likely to be right first time when you consider quality from the start. Having appropriate quality assurance in place helps to manage mistakes, handle changes to requirements and ensure appropriate re-use; \n* **Delivering quality analysis provides the confidence that is needed for transparency and public openness;** \n* **Assurance is required for audit purposes**[^2]; and \n* **Professional pride in your work.** \n\n## How to use this book\n\nThe first four chapters of this book cover definitions and overarching themes, whilst the second half of the book goes into more detail on the [analytical life cycle](analytical_lifecycle.qmd). This can be pictured as follows:\n![Figure 1 - The analytical cycle](analytical_lifecycle.jpg){fig-alt=\"This diagram shows the cycle of activities that make up a typical analytical workflow and shows how these map to the chapters of the AQuA Book. The cycle moves from the start point to engagement and scoping, then design, analysis and finally delivery and communication. After the final stage the cycle either ends at sign off or returns to the engagement and scoping stage. The figure explains that the cycle is often iterative.\"}\n\nEach chapter in the second half of the book is structured as follows:\n\n* Introduction and overview\n* Roles and responsibilities \n* Assurance activities\n* Documentation\n* Uncertainty\n* Black box models \n* Multi-use models\n* Any other elements specific to the stage of the life cycle\n\nThis guidance uses the following terms to indicate whether recommendations are mandatory or advisory. \n\nThe terms are: \n\n* **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances, at all times \n* **‘should’** denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis \n* **‘may’** denotes approval \n* **‘might’** denotes a possibility \n* **‘can’** denotes both capability and possibility \n* **is/are** is used for a description \n\nThese are the same terms as those in the [UK Government Functional Standards](https://www.gov.uk/government/collections/functional-standards). \n\n# Acknowledgements {.unnumbered}\n\nThe AQuA Book is the work of many authors from across the Government Analysis Function. The original version of the book was compiled by the Quality Assurance Working Group set up after Sir Nicholas Macpherson's review of modelling in government. This revised edition of the book was produced by a task and finish group drawn from across the Analysis Function community. We would like to thank everybody who has given of their time and expertise to produce the revised edition.\n\n\n[^1]: [Black box](https://en.wikipedia.org/wiki/Black_box): system which can be viewed in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings.\n[^2]: [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money ), Annex 4.2 Use of models \n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"intro.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file +{"title":"Introduction","markdown":{"headingText":"Introduction","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\nThe AQuA book provides guidance on producing quality analysis for government. It aims to support well informed decision making to deliver better outcomes and improve the lives of citizens.\n\nThe AQuA Book has made a significant contribution to the cultural change in assurance practices in government. It is about the process for assuring analytical evidence in all forms. It sets out the core framework for assuring all forms of analytical evidence. \n\nThe last version of the **Analytical Quality Assurance (AQuA) Handbook** was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then, assurance has become part of the fabric of good practice for developing evidence to support policy development, delivery and operational excellence. \n\nThe world of analysis has developed since we published the first edition of the AQuA Book. \n\nIncreasingly in our data driven world, insights from analysis underpin almost all policies and support operational excellence. This underlines the continuing importance of assuring our evidence. In parallel our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. Since then we have broadened the range of methods to include open-source software, machine learning and artificial intelligence. \n\nUsers of the AQuA book have pointed out some things we did not cover in the first edition and areas where guidance was unclear or insufficient. In this edition we have added guidance on: \n\n* multi-use models - large models used for many purposes with many stakeholders; \n* assuring \"black box\"[^1] analysis, including artificial intelligence; \n* development, maintenance and continuous review; \n* working with third parties such as contractors and academic groups and, \n* publishing models. \n\nWe provide improved guidance on what [a proportionate approach to assurance means](https://best-practice-and-impact.github.io/aqua_book_revision/proportionality.html) and have made the whole guide applicable to all types of analysis.\n\nThe AQuA Book describes what you need to do but not how to do it, although it does contain many worked examples. Large organisations will have their own processes and practices covering “the how”. \n\nFor those of you who do not work in places with bespoke guidance you will find a collection of helpful resources in chapter 10.\n\nThe AQuA Book is a key supporting guide for the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). This Standard refers extensively to the AQuA book and notes that \"detailed guidance on the analytical cycle and management of analysis, included in the Aqua Book should be followed.\"\n\nIt is also referred to by the [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020), the [Magenta Book](https://www.gov.uk/government/publications/the-magenta-book) and other Functional Standards, such as the [Finance Function](https://www.gov.uk/government/publications/government-finance-standards-page) Standards. \n\n\n## Who is the AQuA Book for?\n\nIn this edition we have tried to make our guidance relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose and not just about the checks after the analysis has been completed. \n\nWe would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision making in government. Our intended audience includes: \n\n* Users of analysis – helping you to get the most out of your commission; \n* Those who carry out analysis such as members of the government analytical professions, including:\n + operational researchers, statisticians and economists;\n + geographers;\n + finance professionals;\n + actuaries;\n + social researchers carrying out qualitative research;\n + data scientists developing advanced analytics;\n + and anyone else carrying out analysis. \n* Senior leaders with an interest in analytical assurance. \n\n## Why should I pay attention to this guidance? \n\nHere are a few reasons why.\n\n* **Your analytical insights will be used for major decisions and operations.** You need to do your best to get them right, thus minimising the risk of being complicit in causing operational, business or reputational damage; \n* **Trust is hard to obtain but easy to lose.** A simple error that could have been prevented by assurance could lead to your and your team’s work being doubted; \n* **Prevention is better than cure.** Analysis is more likely to be right first time when you consider quality from the start. Having appropriate quality assurance in place helps to manage mistakes, handle changes to requirements and ensure appropriate re-use; \n* **Delivering quality analysis provides the confidence that is needed for transparency and public openness;** \n* **Assurance is required for audit purposes**[^2]; and \n* **Professional pride in your work.** \n\n## How to use this book\n\nThe first four chapters of this book cover definitions and overarching themes, whilst the second half of the book goes into more detail on the [analytical life cycle](analytical_lifecycle.qmd). This can be pictured as follows:\n![Figure 1 - The analytical cycle](analytical_lifecycle.jpg){fig-alt=\"This diagram shows the cycle of activities that make up a typical analytical workflow and shows how these map to the chapters of the AQuA Book. The cycle moves from the start point to engagement and scoping, then design, analysis and finally delivery and communication. After the final stage the cycle either ends at sign off or returns to the engagement and scoping stage. The figure explains that the cycle is often iterative.\"}\n\nEach chapter in the second half of the book is structured as follows:\n\n* Introduction and overview\n* Roles and responsibilities \n* Assurance activities\n* Documentation\n* Uncertainty\n* Black box models \n* Multi-use models\n* Any other elements specific to the stage of the life cycle\n\nThis guidance uses the following terms to indicate whether recommendations are mandatory or advisory. \n\nThe terms are: \n\n* **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances, at all times \n* **‘should’** denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis \n* **‘may’** denotes approval \n* **‘might’** denotes a possibility \n* **‘can’** denotes both capability and possibility \n* **is/are** is used for a description \n\nThese are the same terms as those in the [UK Government Functional Standards](https://www.gov.uk/government/collections/functional-standards). \n\n[^1]: [Black box](https://en.wikipedia.org/wiki/Black_box): system which can be viewed in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings.\n[^2]: [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money ), Annex 4.2 Use of models \n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"intro.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file diff --git a/.quarto/xref/28f6248c b/.quarto/xref/28f6248c index ddac277..829c30f 100644 --- a/.quarto/xref/28f6248c +++ b/.quarto/xref/28f6248c @@ -1 +1 @@ -{"entries":[],"headings":["who-is-the-aqua-book-for","why-should-i-pay-attention-to-this-guidance","how-to-use-this-book","acknowledgements"],"options":{"chapters":true}} \ No newline at end of file +{"entries":[],"options":{"chapters":true},"headings":["who-is-the-aqua-book-for","why-should-i-pay-attention-to-this-guidance","how-to-use-this-book"]} \ No newline at end of file diff --git a/.quarto/xref/2d689abf b/.quarto/xref/2d689abf index 4609b44..cb01cd9 100644 --- a/.quarto/xref/2d689abf +++ b/.quarto/xref/2d689abf @@ -1 +1 @@ -{"entries":[],"headings":["preface"],"options":{"chapters":true}} \ No newline at end of file +{"options":{"chapters":true},"headings":["preface","acknowledgements"],"entries":[]} \ No newline at end of file diff --git a/.quarto/xref/8968456b b/.quarto/xref/8968456b index 20fe9e1..b25df94 100644 --- a/.quarto/xref/8968456b +++ b/.quarto/xref/8968456b @@ -1 +1 @@ -{"entries":[],"options":{"chapters":true},"headings":[]} \ No newline at end of file +{"options":{"chapters":true},"entries":[],"headings":[]} \ No newline at end of file diff --git a/.quarto/xref/INDEX b/.quarto/xref/INDEX index d303cd9..53ef899 100644 --- a/.quarto/xref/INDEX +++ b/.quarto/xref/INDEX @@ -46,5 +46,8 @@ }, "foreword.qmd": { "foreword.html": "305b8fa7" + }, + "improving_the_book.qmd": { + "improving_the_book.html": "e0de93da" } } \ No newline at end of file diff --git a/.quarto/xref/a0b88893 b/.quarto/xref/a0b88893 index 190cfc7..0679425 100644 --- a/.quarto/xref/a0b88893 +++ b/.quarto/xref/a0b88893 @@ -1 +1 @@ -{"entries":[],"options":{"chapters":true},"headings":["analysis","assurance","assurance-activities","artificial-intelligence","black-box-models","business-critical-analysis","change-control","documentation","specification-documentation","design-documentation","assumptions-log","decisions-log","data-log","user-technical-documentation","assurance-statement","materiality","multi-use-models","principles-of-analytical-quality-assurance","quality-analysis","reproducible-analytical-pipelines","roles-and-responsibilities","third-party","uncertainty","validation","verification","version-control"]} \ No newline at end of file +{"options":{"chapters":true},"entries":[],"headings":["analysis","assurance","assurance-activities","artificial-intelligence","black-box-models","business-critical-analysis","change-control","documentation","specification-documentation","design-documentation","assumptions-log","decisions-log","data-log","user-technical-documentation","assurance-statement","materiality","multi-use-models","principles-of-analytical-quality-assurance","quality-analysis","reproducible-analytical-pipelines","roles-and-responsibilities","third-party","uncertainty","validation","verification","version-control"]} \ No newline at end of file diff --git a/_quarto.yml b/_quarto.yml index 24b1172..93ad88d 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -5,7 +5,7 @@ project: book: title: "The AQuA Book" author: "Government Analysis Function" - date: "10/08/2024" + date: "29/10/2024" chapters: - index.qmd - foreword.qmd diff --git a/docs/Figure 3-1 Types of Assurance Alternative 3.jpg b/docs/Figure 3-1 Types of Assurance Alternative 3.jpg new file mode 100644 index 0000000..e461521 Binary files /dev/null and b/docs/Figure 3-1 Types of Assurance Alternative 3.jpg differ diff --git a/docs/additional_resources.html b/docs/additional_resources.html index f041554..de3f829 100644 --- a/docs/additional_resources.html +++ b/docs/additional_resources.html @@ -30,7 +30,7 @@ - + @@ -180,7 +180,7 @@
  • - + Accessibility statement
    • @@ -720,7 +720,7 @@

    - + Accessibility statement diff --git a/docs/analysis.html b/docs/analysis.html index c2a0244..1b8e32d 100644 --- a/docs/analysis.html +++ b/docs/analysis.html @@ -180,7 +180,7 @@
  • - + Accessibility statement
    • @@ -200,25 +200,24 @@

    Table of contents

    @@ -261,29 +260,18 @@

    8 

    8.1 Introduction and overview

    The analysis stage is where planned analysis is undertaken and assured, and progress and relevance are monitored. During this stage, the design may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the design or scoping stage.

    -
    -

    8.1.1 The Analyst’s responsibilities during the analysis stage

    -
      -

    • The Analyst shall follow the assurance plan, and conduct the specified verification and validation. They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.

      • -

    • The Analyst shall produce documentation of the data and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.

      • -

    • The Analyst shall document any changes to the analytical plan in a proportionate manner.

      • -

    • The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner’s needs or whether there are any new requirements.

      • -
    -
    -
    -

    8.1.2 The Assurer’s responsibilities during the analysis stage

    -

    The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required.

    -

    The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology.

    -
    -

    8.1.3 The Commissioner’s responsibilities during the analysis stage

    +
    +

    8.2 Roles and responsibilities in the analysis stage

    +
    +

    8.2.1 The Commissioner’s responsibilities during the analysis stage

    • The Commissioner should be available to provide input and clarifications to the Analyst.

    • The Commissioner’s should review any changes in design or methodology that the Analyst brings to their attention.

    -
    -

    8.1.4 The Analyst’s responsibilities during the analysis stage

    +
    +

    8.2.2 The Analyst’s responsibilities during the analysis stage

    • The Analyst shall follow the conduct the verification and validation activities that were designed as part of the analytical plan in the design stage. They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.

    • When the analysis includes coding, the Analyst shall proportionately follow best practice for code development.

      • @@ -292,28 +280,28 @@

      -

      8.1.5 The Assurer’s responsibilities during the analysis stage

      +
      +

      8.2.3 The Assurer’s responsibilities during the analysis stage

      • The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required.

        • -

      • When the analysis includes coding, the Assurer shall review that the work proportionately adheres to coding best practice.

        • +

      • When the analysis includes coding, the Assurer shall review that the work proportionately adheres to best practice for code development.

      • The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology.

      -
      -

      8.1.6 The Approver’s responsibilities during the analysis stage

      +
      +

      8.2.4 The Approver’s responsibilities during the analysis stage

      The Approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage.

      -
      -

      8.2 Assurance activities in the analysis stage

      -
      -

      8.2.1 Verification and validation

      +
      +

      8.3 Assurance activities in the analysis stage

      +
      +

      8.3.1 Verification and validation

      Verification that the implemented methodology meets the intended plan should be incorporated as part the analysis. Whitener and Balci (1989) review verification techniques in relation to simulation modelling, but these techniques extend to analysis more broadly. These include:

      • Informal analysis: techniques that rely on human reasoning and subjectivity.
      • Static analysis: tests that the implementation of the analysis before it is run. For example, checking that code adheres to code conventions, structural analysis of the code by examining graphs of control and data flows, .
        • -
      • Dynamic analysis: tests the behaviour of the system or code to find errors that arise during execution. This includes unit testing, integration testing and stress testing
        • +
      • Dynamic analysis: tests the behaviour of the system, model or code to find errors that arise during execution. This includes unit testing, integration testing and stress testing
      • Symbolic analysis: particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during the execution of a model. Includes path tracing and cause-effect testing (see Whitener and Balci (1989) )
      • Constraint analysis: particularly relevant to modelling and tests the implementation of constraints during model execution. This includes checking the assertions of the model and boundary analysis.
      • Formal analysis: tests logical correctness through formal verification such as logic or mathematical proofs
        • @@ -322,8 +310,8 @@

        Sargent (2011).

        The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the specification document to assure that the analysis meets the specification.

      -
      -

      8.2.2 Data validity and data considerations

      +
      +

      8.3.2 Data validity and data considerations

      Testing data validity (i.e. that data meet the specification for which they are used) is a key part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics (outliers, trends, expected distributions etc), and assuring robust data management practices (e.g. automating data creation and data sourcing).

      It is rare to have the perfect dataset for an analytical commission. Reasons for this include:

        @@ -338,13 +326,13 @@

        -

        8.2.3 Assurance of code

        -

        The Duck Book provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on structuring code, producing documentation, using version control, data management, testing, peer review, and automation.

        +
        +

        8.3.3 Assurance of code

        +

        The Duck Book provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on structuring code, producing documentation, using version control, data management, testing, peer review, and automation. The Analysit shall follow the guidance for good quality code development in a proportionate manner, and the Assurer shall review this accordingly.

      -
      -

      8.3 Documention in the analysis stage

      +
      +

      8.4 Documention in the analysis stage

      The Analyst should:

      • Maintain appropriate records of the work;
        • @@ -354,16 +342,17 @@

        user and technical documentation).

      -
      -

      8.4 Treatment of uncertainty in the analysis stage

      +
      +

      8.5 Treatment of uncertainty in the analysis stage

      While the Scoping and Design stages identified and described risks and uncertainties, the Analysis stage aims to assess and quantify the impact of uncertainty on the analytical outcome and their contribution to the range and likelihoods of possible outcomes. The Uncertainty Toolkit for Analysts reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty.

      -
      -

      8.5 Black box models and the analysis stage

      -

      Bblack box models such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. Assurance activities during the Analysis stage include performance testing and formal verification. See the Introduction to AI Assurance for further details.

      +
      +

      8.6 Black box models and the analysis stage

      +

      Black box models such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis.

      +

      Assurance activities during the Analysis stage: * may include performance testing in a live environment and * should include the verification steps set out in the Design Phase * should include validation and verification of automatic tests to ensure the model behave as expected See the Introduction to AI Assurance for further details.

      -
      -

      8.6 Multi-use models and the analysis stage

      +
      +

      8.7 Multi-use models and the analysis stage

      In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This calls for mechanisms for assuring that the changes integrate into the larger model as expected, for example, through the use of test-suites.

      diff --git a/docs/analytical_lifecycle.html b/docs/analytical_lifecycle.html index 3f1603a..b484b04 100644 --- a/docs/analytical_lifecycle.html +++ b/docs/analytical_lifecycle.html @@ -180,7 +180,7 @@
    • - + Accessibility statement
      • @@ -336,7 +336,7 @@

      5.2.5 Maintenance and continuous review

      -

      The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made whether The inputs used remain appropriate The initial communication methods remain the best way to deliver the information *Any software relied on continues to be supported and up to date

      +

      The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made whether The inputs used remain appropriate The initial communication methods remain the best way to deliver the information Any software relied on continues to be supported and up to date The model continues to be calibrated appropriately (this is particularly important for black box models)

      Additionally, a robust version control process should be in place to ensure any changes to the analysis are appropriately assured.

      diff --git a/docs/definitions_and_key_concepts.html b/docs/definitions_and_key_concepts.html index ac64b7f..57f918a 100644 --- a/docs/definitions_and_key_concepts.html +++ b/docs/definitions_and_key_concepts.html @@ -200,7 +200,7 @@
    • - + Accessibility statement
      • @@ -459,20 +459,7 @@

      Uncertainty

      -

      The outcome of a decision is never known perfectly in advance. For each option within analysis, a range of real outcomes is possible: the outcome is uncertain.

      -
      -
      -
      - -
      -
      -Defining uncertainty -
      -
      -
      -

      Wikipedia defines uncertainty as referring to epistemic situations involving imperfect or unknown information. It applies to predictions of future events, to physical measurements that are already made, or to the unknown.

      -
      -
      +

      Uncertainties are things that are not known, or are in a state of doubt, or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme, piece of analysis meeting its objectives.1

      There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will impact the analytical approach and assurance activities required.

      The Uncertainty Toolkit for Analysts in Government is a tool produced by a cross government group to help assessing and communicating uncertainty.

      @@ -495,6 +482,12 @@

      Version control

      +
      +
      +
        +

      1. https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf↩︎

        2. +
      +
      - + @@ -199,7 +199,7 @@
    • - + Accessibility statement
      • @@ -666,7 +666,7 @@

      References