Introduction
Think about and deliver appropriate (proportionate) levels of assurance for your analysis. There is a need to be confident in analysis delivered, but there is no point spending months assuring simple analysis that will inform a decision that will make minimal impact.
-Table 1 provides a list of key factors that should be considered when determining what level of assurance is appropriate.
+Table 3-1 provides a list of key factors that should be considered when determining what level of assurance is appropriate.
Further detail and considerations may be found on the Analysis Function’s Quality Questions and Red Flags page.
-Table 1 - Factors for determining appropriate assurance
+Table 3-1 - Factors for determining appropriate assurance
@@ -298,25 +298,25 @@
-Figure 1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The key message is the need for more assurance interventions increases with the complexity of, and the business risk associated with analysis.
+Figure 3-1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The key message is the need for more assurance interventions increases with the complexity of, and the business risk associated with analysis.
-The interventions in Figure 1 must not be viewed in isolation. Some complex and risky analysis that would benefit from an external review will still require the interventions closer to the axes, for example version control and analyst led testing.
+The interventions in Figure 3-1 must not be viewed in isolation. Some complex and risky analysis that would benefit from an external review will still require the interventions closer to the axes, for example version control and analyst led testing.
One way to view assurance interventions would be to consider individual interventions in a layer. An individual intervention will reduce risks in a particular area, but will leave many other risks unmitigated. Adding more interventions (layers) will start to increase coverage and reduce overall risk.
-The total elimination of risk will never be achievable, and the balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. In practice there are many different techniques that need to be considered and implemented as appropriate. Refer to the table in chapter 10.
+The total elimination of risk will never be achievable, so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. In practice there are many different techniques that need to be considered and implemented as appropriate. Refer to the table in chapter 10.
Many of these interventions are mentioned elsewhere in the AQUA Book, and are not repeated here.
Structured assessment of business risk and complexity
To guide what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential impact of analysis errors, and the likelihood of errors occurring. In situations where the potential business impact is high, it is more important that the likelihood of errors is reduced.
-This can be visualised by considering the situation as a risk matrix, illustrated in Table X. The impact of the analysis will usually be beyond the control of the analyst to change, so there will be few options to move an assessment down the table. However, there will usually be treatments (or mitigations), involving additional assurance measures, that will allow the assessed business risk to move to the left.
+This can be visualised by considering the situation as a risk matrix, illustrated in Table 3-2. The impact of the analysis will usually be beyond the control of the analyst to change, so there will be few options to move an assessment down the table. However, there will usually be treatments (or mitigations), involving additional assurance measures, that will allow the assessed business risk to move to the left.
-Table X - Example of a risk matrix
+Table 3-2 - Example of a risk matrix
@@ -461,10 +461,10 @@
-Table X - Responses to risk assessment levels
+Table 3-1 - Responses to risk assessment levels
@@ -500,8 +500,8 @@
- Artificial intelligence
+
+ Artificial intelligence and business risk
Increasingly analysis may be underpinned by Artificial Intelligence (AI). With AI-informed analysis the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible.
diff --git a/docs/search.json b/docs/search.json
index edb920f..e4181c4 100644
--- a/docs/search.json
+++ b/docs/search.json
@@ -234,7 +234,7 @@
"href": "proportionality.html",
"title": "3 Proportionality",
"section": "",
- "text": "3.1 Introduction\nThink about and deliver appropriate (proportionate) levels of assurance for your analysis. There is a need to be confident in analysis delivered, but there is no point spending months assuring simple analysis that will inform a decision that will make minimal impact.\nTable 1 provides a list of key factors that should be considered when determining what level of assurance is appropriate.\nFurther detail and considerations may be found on the Analysis Function’s Quality Questions and Red Flags page.\nFigure 1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The key message is the need for more assurance interventions increases with the complexity of, and the business risk associated with analysis.\nThe interventions in Figure 1 must not be viewed in isolation. Some complex and risky analysis that would benefit from an external review will still require the interventions closer to the axes, for example version control and analyst led testing.\nOne way to view assurance interventions would be to consider individual interventions in a layer. An individual intervention will reduce risks in a particular area, but will leave many other risks unmitigated. Adding more interventions (layers) will start to increase coverage and reduce overall risk.\nThe total elimination of risk will never be achievable, and the balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. In practice there are many different techniques that need to be considered and implemented as appropriate. Refer to the table in chapter 10.\nMany of these interventions are mentioned elsewhere in the AQUA Book, and are not repeated here.",
+ "text": "3.1 Introduction\nThink about and deliver appropriate (proportionate) levels of assurance for your analysis. There is a need to be confident in analysis delivered, but there is no point spending months assuring simple analysis that will inform a decision that will make minimal impact.\nTable 3-1 provides a list of key factors that should be considered when determining what level of assurance is appropriate.\nFurther detail and considerations may be found on the Analysis Function’s Quality Questions and Red Flags page.\nFigure 3-1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The key message is the need for more assurance interventions increases with the complexity of, and the business risk associated with analysis.\nThe interventions in Figure 3-1 must not be viewed in isolation. Some complex and risky analysis that would benefit from an external review will still require the interventions closer to the axes, for example version control and analyst led testing.\nOne way to view assurance interventions would be to consider individual interventions in a layer. An individual intervention will reduce risks in a particular area, but will leave many other risks unmitigated. Adding more interventions (layers) will start to increase coverage and reduce overall risk.\nThe total elimination of risk will never be achievable, so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. In practice there are many different techniques that need to be considered and implemented as appropriate. Refer to the table in chapter 10.\nMany of these interventions are mentioned elsewhere in the AQUA Book, and are not repeated here.",
"crumbs": [
"3 Proportionality"
]
@@ -244,7 +244,7 @@
"href": "proportionality.html#introduction",
"title": "3 Proportionality",
"section": "",
- "text": "Table 1 - Factors for determining appropriate assurance\n\n\n\n\n\n\nFactor\nComments\n\n\n\n\nBusiness criticality\nDifferent issues will vary their financial, legal, operational, political and reputational impacts.\n\n\nRelevance of the analysis to the decision making process\nWhen analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the evidence base.\n\n\nType and complexity of analysis\nHighly complex analysis requires more effort to assure. The nature of that analysis may also require the engagement of appropriate subject matter experts.\n\n\nNovelty of approach\nA previously untried method requires more assurance. Confidence will grow as the technique is repeatedly tested.\n\n\nPrecision of the analysis outputs\nImprecise analysis can require different assurance than precise analysis, e.g. because of inherent limitations of the analytical technique, or lack of data on assumptions.\n\n\nAmount of resource available for the analysis and assurance\nThe value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Approaches that underpin a lot of different things may require greater levels of QA than might be suggested by any individual decision they support.\n\n\nLongevity of the analysis\nChange control.\n\n\nRepeat runs for the same analysis\nControl and assurance of data and parameters for each run\n\n\n\n\n\n\n\nFigure 1 - Types of quality assurance",
+ "text": "Table 3-1 - Factors for determining appropriate assurance\n\n\n\n\n\n\nFactor\nComments\n\n\n\n\nBusiness criticality\nDifferent issues will vary their financial, legal, operational, political and reputational impacts.\n\n\nRelevance of the analysis to the decision making process\nWhen analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the evidence base.\n\n\nType and complexity of analysis\nHighly complex analysis requires more effort to assure. The nature of that analysis may also require the engagement of appropriate subject matter experts.\n\n\nNovelty of approach\nA previously untried method requires more assurance. Confidence will grow as the technique is repeatedly tested.\n\n\nPrecision of the analysis outputs\nImprecise analysis can require different assurance than precise analysis, e.g. because of inherent limitations of the analytical technique, or lack of data on assumptions.\n\n\nAmount of resource available for the analysis and assurance\nThe value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Approaches that underpin a lot of different things may require greater levels of QA than might be suggested by any individual decision they support.\n\n\nLongevity of the analysis\nChange control.\n\n\nRepeat runs for the same analysis\nControl and assurance of data and parameters for each run\n\n\n\n\n\n\n\nFigure 3-1 - Types of quality assurance",
"crumbs": [
"3 Proportionality"
]
@@ -254,7 +254,7 @@
"href": "proportionality.html#structured-assessment-of-business-risk-and-complexity",
"title": "3 Proportionality",
"section": "3.2 Structured assessment of business risk and complexity",
- "text": "3.2 Structured assessment of business risk and complexity\nTo guide what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential impact of analysis errors, and the likelihood of errors occurring. In situations where the potential business impact is high, it is more important that the likelihood of errors is reduced.\nThis can be visualised by considering the situation as a risk matrix, illustrated in Table X. The impact of the analysis will usually be beyond the control of the analyst to change, so there will be few options to move an assessment down the table. However, there will usually be treatments (or mitigations), involving additional assurance measures, that will allow the assessed business risk to move to the left.\n\n\nTable X - Example of a risk matrix\n\n\n\n\n\n\n\nLikelihood of errors occurring\n\n\n\nImpact of errors occurring\n\n\n\n\n\n\nHighly unlikely\n\n\nUnlikely\n\n\nRealistic possibility\n\n\nLikely or probably\n\n\nHighly likely\n\n\n\n\n\n\nCritical\n\n\nMedium\n\n\nMedium\n\n\nHigh\n\n\nHigh\n\n\nHigh\n\n\n\n\n\n\nSevere\n\n\nLow\n\n\nMedium\n\n\nMedium\n\n\nHigh\n\n\nHigh\n\n\n\n\n\n\nMajor\n\n\nLow\n\n\nMedium\n\n\nMedium\n\n\nMedium\n\n\nHigh\n\n\n\n\n\n\nModerate\n\n\nVery Low\n\n\nLow\n\n\nMedium\n\n\nMedium\n\n\nMedium\n\n\n\n\n\n\nMinor\n\n\nVery Low\n\n\nVery Low\n\n\nLow\n\n\nLow\n\n\nMedium\n\n\n\nTable X shows appropriate responses to a risk assessment. Where business risk is high, appropriate treatment(s) must be considered to reduce the probability of errors occurring. The choice of treatment will depend on the mitigations already in place and on the complexity of the analysis (see Figure 1). For a situation where simple analysis is being employed, a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range internal assurance measures, options like external peer review may be necessary. For situations where the business risk is ‘very low’, there would be very little benefit in applying further assurance mitigations, . Aalthough to avoid the risk growing it remains important to ensure existing or planned mitigations aren’t lost.\nIn cases where there is a need for analysis, but there are also significant time and/or resource constraints, it may not be possible to do as much assurance as usual. In these situations, the focus should be on areas of greatest risk. These risks and limitations must also be communicated, along with appropriate caveats.\n\nTable X - Responses to risk assessment levels\n\n\n\n\n\n\nAssessed risk\nMitigations to consider\n\n\n\n\nHigh\nThe risk should not be tolerated. New assurance measures must be considered to treat (mitigate) the likelihood of errors occurring. If treatment isn’t an option, consideration must be be given to terminating or transferring the (analysis) risk. If it remains necessary to tolerate the risk the SRO needs to fully understand the risk.\n\n\nMedium\nThe risk should not be tolerated without SRO agreement. New assurance measures should be put in place to treat (mitigate) the likelihood of errors occurring. Continue with planned or existing mitigations.\n\n\nLow\nThe risk can be tolerated. Existing or planned mitigations should be continued, and new treatments may be considered.\n\n\nVery Low\nThe risk can be tolerated. Existing/planned mitigations measures should be continued.",
+ "text": "3.2 Structured assessment of business risk and complexity\nTo guide what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential impact of analysis errors, and the likelihood of errors occurring. In situations where the potential business impact is high, it is more important that the likelihood of errors is reduced.\nThis can be visualised by considering the situation as a risk matrix, illustrated in Table 3-2. The impact of the analysis will usually be beyond the control of the analyst to change, so there will be few options to move an assessment down the table. However, there will usually be treatments (or mitigations), involving additional assurance measures, that will allow the assessed business risk to move to the left.\n\n\nTable 3-2 - Example of a risk matrix\n\n\n\n\n\n\n\nLikelihood of errors occurring\n\n\n\nImpact of errors occurring\n\n\n\n\n\n\nHighly unlikely\n\n\nUnlikely\n\n\nRealistic possibility\n\n\nLikely or probably\n\n\nHighly likely\n\n\n\n\n\n\nCritical\n\n\nMedium\n\n\nMedium\n\n\nHigh\n\n\nHigh\n\n\nHigh\n\n\n\n\n\n\nSevere\n\n\nLow\n\n\nMedium\n\n\nMedium\n\n\nHigh\n\n\nHigh\n\n\n\n\n\n\nMajor\n\n\nLow\n\n\nMedium\n\n\nMedium\n\n\nMedium\n\n\nHigh\n\n\n\n\n\n\nModerate\n\n\nVery Low\n\n\nLow\n\n\nMedium\n\n\nMedium\n\n\nMedium\n\n\n\n\n\n\nMinor\n\n\nVery Low\n\n\nVery Low\n\n\nLow\n\n\nLow\n\n\nMedium\n\n\n\nTable 3-2 shows appropriate responses to a risk assessment. Where business risk is high, appropriate treatment(s) must be considered to reduce the probability of errors occurring. The choice of treatment will depend on the mitigations already in place and on the complexity of the analysis (see Figure 3-1). For a situation where simple analysis is being employed, a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range internal assurance measures, options like external peer review may be necessary. For situations where the business risk is ‘very low’, there would be very little benefit in applying further assurance mitigations, . Aalthough to avoid the risk growing it remains important to ensure existing or planned mitigations aren’t lost.\nIn cases where there is a need for analysis, but there are also significant time and/or resource constraints, it may not be possible to do as much assurance as usual. In these situations, the focus should be on areas of greatest risk. These risks and limitations must also be communicated, along with appropriate caveats.\n\nTable 3-1 - Responses to risk assessment levels\n\n\n\n\n\n\nAssessed risk\nMitigations to consider\n\n\n\n\nHigh\nThe risk should not be tolerated. New assurance measures must be considered to treat (mitigate) the likelihood of errors occurring. If treatment isn’t an option, consideration must be be given to terminating or transferring the (analysis) risk. If it remains necessary to tolerate the risk the SRO needs to fully understand the risk.\n\n\nMedium\nThe risk should not be tolerated without SRO agreement. New assurance measures should be put in place to treat (mitigate) the likelihood of errors occurring. Continue with planned or existing mitigations.\n\n\nLow\nThe risk can be tolerated. Existing or planned mitigations should be continued, and new treatments may be considered.\n\n\nVery Low\nThe risk can be tolerated. Existing/planned mitigations measures should be continued.",
"crumbs": [
"3 Proportionality"
]
@@ -270,11 +270,11 @@
]
},
{
- "objectID": "proportionality.html#artificial-intelligence",
- "href": "proportionality.html#artificial-intelligence",
+ "objectID": "proportionality.html#artificial-intelligence-and-business-risk",
+ "href": "proportionality.html#artificial-intelligence-and-business-risk",
"title": "3 Proportionality",
- "section": "3.4 Artificial intelligence",
- "text": "3.4 Artificial intelligence\nIncreasingly analysis may be underpinned by Artificial Intelligence (AI). With AI-informed analysis the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible.",
+ "section": "3.4 Artificial intelligence and business risk",
+ "text": "3.4 Artificial intelligence and business risk\nIncreasingly analysis may be underpinned by Artificial Intelligence (AI). With AI-informed analysis the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible.",
"crumbs": [
"3 Proportionality"
]
@@ -374,7 +374,7 @@
"href": "engagement_and_scoping.html#the-commissioners-responsibilities-during-engagement-and-scoping",
"title": "6 Engagement and scoping",
"section": "6.2 The Commissioner’s responsibilities during engagement and scoping",
- "text": "6.2 The Commissioner’s responsibilities during engagement and scoping\nDuring engagement, the Commissioner and the Analyst shape the analysis by developing a shared understanding of the problem and the context.\nThe Commissioner is responsible for ensuring that:\n\nKey aspects of the problem, scope, and programme constraints, are captured and clearly communicated to the Analyst.\n\nThey are available to actively engage with the Analyst to appropriately shape the work.\n\nThere is sufficient governance in place to support the analysis and its role in the wider project or programme. This is particularly important if the analysis supports business critical decisions. This proportionality may need to be revisited at the design stage if a novel or riskier approach is required (for example if AI models are used).\n\nThey understand risks where time and resource pressures constrain the approach.\n\nThey request information on uncertainty from analysts and challenge them when it is absent, inadequate or ambiguous.\n\nCommunicate to the analyst any sources of uncertainty they have identified as part of their wider considerations.\n\nIf possible, indicate in advance the consequences for decision-making of different degrees of uncertainty, as this may enable the analyst to conduct their analysis at a proportionate level.",
+ "text": "6.2 The Commissioner’s responsibilities during engagement and scoping\nDuring engagement, the Commissioner and the Analyst shape the analysis by developing a shared understanding of the problem and the context.\nThe Commissioner is responsible for ensuring that:\n\nKey aspects of the problem, scope, and programme constraints, are captured and clearly communicated to the Analyst.\n\nThey are available to actively engage with the Analyst to appropriately shape the work.\n\nThere is sufficient governance in place to support the analysis and its role in the wider project or programme. This is particularly important if the analysis supports business critical decisions. This proportionality may need to be revisited at the design stage if a novel or riskier approach is required (for example if AI models are used).\n\nThey understand risks where time and resource pressures constrain the approach.\n\nThey request information on uncertainty from analysts and challenge them when it is absent, inadequate or ambiguous.\n\nCommunicate to the analyst any sources of uncertainty they have identified as part of their wider considerations.\n\nIf possible, indicate in advance the consequences for decision-making of different degrees of uncertainty, as this may enable the analyst to conduct their analysis at a proportionate level.\n\nCommisioner should sign off on the specification",
"crumbs": [
"6 Engagement and scoping"
]
|
