04.01-idp-a2i-with-custom-rules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Customize Business Rules for Intelligent Document Processing with Human Review and BI Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon Textract (https://aws.amazon.com/textract/) lets you easily extract text from various documents, and Amazon Augmented AI (https://aws.amazon.com/augmented-ai/) (A2I) enables you to implement a human review of machine learning predictions. The default Amazon A2I template allows you to build a human review pipeline based on conditions such as when the extraction confidence score is lower than a pre-defined threshold, required keys are missing, or randomly assigning documents to human review. But sometimes, customers need the document processing pipeline to support flexible business rules, such as validating the string format, verifying the data type and range, and cross fields validation. This post shows how you can leverage Amazon Textract and Amazon A2I to customize a generic document processing pipeline supporting flexible business rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The following diagram illustrates the workflow of the pipeline that supports customized business rules:\n",
    "![a2i with custom rules](./images/a2i-custom-rule.png)\n",
    "\n",
    "- [Step 1: Setup notebook](#step1)\n",
    "- [Step 2: Extract text from sample documents using Amazon Textract](#step2)\n",
    "- [Step 3: Validate rules and route document to A2I](#step3)\n",
    "- [Step 4: Build BI dashboard using QuickSight](#step4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Sample Document\n",
    "The documents processed in the sample solution is the Tax Form 990 (https://en.wikipedia.org/wiki/Form_990), a US IRS form that provides the public with financial information about a non-profit organization. We will only cover the extraction logic for some of the fields on the 1st page as example since this post focuses on the end-to-end pipeline. \n",
    "\n",
    "<div class=\"alert alert-warning\"> <h4><strong>💡 NOTE</strong>\n",
    "</h4>\n",
    "    The document is from the publicly available IRS (Internal Revenue Service) <a href=\"https://www.irs.gov/charities-non-profits/form-990-series-downloads\">website</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Setup notebook <a id=\"step1\"></a>\n",
    "\n",
    "In this step, we will import some necessary libraries that will be used throughout this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install awscli --upgrade\n",
    "!pip install botocore --upgrade\n",
    "!pip install boto3 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import botocore\n",
    "import sagemaker as sm\n",
    "import os\n",
    "import io\n",
    "import datetime\n",
    "\n",
    "# variables\n",
    "data_bucket = sm.Session().default_bucket()\n",
    "region = boto3.session.Session().region_name\n",
    "\n",
    "os.environ[\"BUCKET\"] = data_bucket\n",
    "os.environ[\"REGION\"] = region\n",
    "role = sm.get_execution_role()\n",
    "\n",
    "print(f\"SageMaker role is: {role}\\nDefault SageMaker Bucket: s3://{data_bucket}\")\n",
    "\n",
    "s3=boto3.client('s3')\n",
    "textract = boto3.client('textract', region_name=region)\n",
    "comprehend=boto3.client('comprehend', region_name=region)\n",
    "sagemaker=boto3.client('sagemaker', region_name=region)\n",
    "a2i=boto3.client('sagemaker-a2i-runtime', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Step 2: Extract text from sample documents using Amazon Textract<a id=\"step2\"></a>\n",
    "\n",
    "In this section we  use Amazon Textract's `analyze_document` API to extract the text information for the 990 Tax Form Page 1 document. We will also map the data to a JSON data model. This data model will be used to validate business rules in the later steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, define a JSON structure `page1` map to the fields on the 990 Tax form on page 1, so we can apply business rules in the later steps. \n",
    "\n",
    "The below image shows how the text on page 1 maps to the JSON fields.\n",
    "\n",
    "![990 page1 mapping](./images/a2i-page1-data-model-mapping.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below `page1` object will hold the Textract extraction result. \n",
    "Following the same pattern, you can expand this JSON to include more fields. For this lab, the sample code only extracts and maps a partial page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# JSON structure to hold the Page 1 extraction result\n",
    "page1 = {\n",
    "          \"dln\": None,\n",
    "          \"omb_no\": None,\n",
    "          \"b.address_change\": None,\n",
    "          \"b.name_change\": None,\n",
    "          \"c.org_name\": None,\n",
    "          \"c.street_no\": None,\n",
    "          \"d.employer_id\": None,\n",
    "          \"e.phone_number\": None,\n",
    "          \"i.501_c_3\": None,\n",
    "          \"i.501_c\": None,\n",
    "          \"part1.1\": None,\n",
    "          \"part1.3\": None,\n",
    "          \"part1.8_pre_yr\": None,\n",
    "          \"part1.8_cur_yr\": None,\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's start to prepare extraction by uploading the sample document to the S3 bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_key = 'idp/textract/990-sample-page-1.jpg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload images to S3 bucket:\n",
    "!aws s3 cp a2idata/990-sample-page-1.jpg s3://{data_bucket}/{s3_key} --only-show-errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get image meta data\n",
    "from PIL import Image\n",
    "im = Image.open('a2idata/990-sample-page-1.jpg')\n",
    "image_width, image_height = im.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now call Textrat's `analyze_document` API Query feature to extract fields by asking specific questions. You do not need to know the structure of the data in the document (table, form, implied field, nested data) or worry about variations across document versions and formats. Queries leverages a combination of visual, spatial, and language cues to extract the information you seek with high accuracy.\n",
    "\n",
    "In the below code, 14 Textract Query questions map to the fields in the JSON structure defined earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = textract.analyze_document(\n",
    "            Document={'S3Object': {'Bucket': data_bucket, 'Name': s3_key}},\n",
    "            FeatureTypes=[\"QUERIES\"],\n",
    "            QueriesConfig={\n",
    "                    'Queries': [\n",
    "                        {\n",
    "                            'Text': 'What is the DLN?',\n",
    "                            'Alias': 'DLN_NO'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text': 'What is the OMB No?',\n",
    "                            'Alias': 'OMB_NO'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text': 'Does the address changed?',\n",
    "                            'Alias': 'B_ADDRESS_CHANGED'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text': 'Does the name changed?',\n",
    "                            'Alias': 'B_NAME_CHANGED'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text': 'What is the name of organzation?',\n",
    "                            'Alias': 'C_ORG_NAME'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text': 'What is the Number and street?',\n",
    "                            'Alias': 'C_STREET_NUMBER'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'What is the Employer identification number?',\n",
    "                            'Alias': 'D_EMPLOYER_ID'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'What is Telephone Number?',\n",
    "                            'Alias': 'E_PHONE'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'Does 501(cx3) checked?',\n",
    "                            'Alias': 'I_501_CX3'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'Does 501(c) checked?',\n",
    "                            'Alias': 'I_501_C'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'What is Breifly describe the organization''s mission or most significant activities?',\n",
    "                            'Alias': 'PART_1_1'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'What is Number of voting members of the governing body?',\n",
    "                            'Alias': 'PART_1_3'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'What is 8 contributes and grants for Prior Year?',\n",
    "                            'Alias': 'PART_1_8_PRIOR_YEAR'\n",
    "                        },\n",
    "                        {\n",
    "                            'Text':'What is 8 contributes and grants for Current Year?',\n",
    "                            'Alias': 'PART_1_8_CURRENT_YEAR'\n",
    "                        },\n",
    "                    ]\n",
    "                }\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Textract JSON response is relatively large. \n",
    "You can use the below code snippet to save it as a JSON file under the same directory called: `textract-response.json`. \n",
    "\n",
    "Open the file in a new SageMaker Studio tab (or you preferred IDE) for easy reviewing and searching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, os\n",
    "with open('textract-response.json','w') as f:\n",
    "    f.write(json.dumps(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's parse the Textract response to populate the values to the `page1` object defined earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility functions to allocate fields from Textract response JSON\n",
    "# Find the Query item in block. Return text and confidence score\n",
    "# return tuple contains (parsed_value, confidence_score, raw_block)\n",
    "def get_query_ref(id):\n",
    "    for b in response[\"Blocks\"]:\n",
    "        if b[\"BlockType\"] == \"QUERY_RESULT\" and b[\"Id\"] == id:\n",
    "            return {\n",
    "                        \"value\": b.get(\"Text\"), \n",
    "                        \"confidence\": b.get(\"Confidence\"), \n",
    "                        \"block\": b\n",
    "                    }\n",
    "    return None\n",
    "        \n",
    "def get_query_answer(q_alias):\n",
    "    for b in response[\"Blocks\"]:\n",
    "        if b[\"BlockType\"] == \"QUERY\" and b[\"Query\"][\"Alias\"] == q_alias:\n",
    "                ref_id = b[\"Relationships\"][0][\"Ids\"][0]\n",
    "                return get_query_ref(ref_id)\n",
    "    return None\n",
    "\n",
    "# Populate Textract Query results to the page1 object\n",
    "page1['dln'] = get_query_answer('DLN_NO')\n",
    "page1['omb_no'] = get_query_answer('OMB_NO')\n",
    "page1['b.address_change'] = get_query_answer('B_ADDRESS_CHANGED')\n",
    "page1['b.name_change'] = get_query_answer('B_NAME_CHANGED')\n",
    "page1['c.org_name'] = get_query_answer('C_ORG_NAME')\n",
    "page1['c.street_no'] = get_query_answer('C_STREET_NUMBER')\n",
    "page1['d.employer_id'] = get_query_answer('D_EMPLOYER_ID')\n",
    "page1['e.phone_number'] = get_query_answer('E_PHONE')\n",
    "page1['i.501_c_3'] = get_query_answer('I_501_CX3')\n",
    "page1['i.501_c'] = get_query_answer('I_501_C')\n",
    "page1['part1.1'] = get_query_answer('PART_1_1')\n",
    "page1['part1.3'] = get_query_answer('PART_1_3')\n",
    "page1['part1.8_pre_yr'] = get_query_answer('PART_1_8_PRIOR_YEAR')\n",
    "page1['part1.8_cur_yr'] = get_query_answer('PART_1_8_CURRENT_YEAR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each fields in the `page1` object contains 3 sub-fields:\n",
    "* *value*: The text value extracted by Textract\n",
    "* *confidence*: Textract confidence score - you can define custom business rule base on it.\n",
    "* *block*: The original Textract block section keeps Geometry metadata. We will need it for the custom A2I UI to plot the bounding box on top of the original text block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out page1\n",
    "page1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Define generic business rules <a id=\"step3\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, we defined 3 business rules for demo purposes:\n",
    "* The 1st rule is for the employer Id field. The rule will fail if the Textract confidence score is lower than 99%. In this demo rule, we set the confidence score threshold high, which will break by design. You could adjust the threshold to a more reasonable value to reduce unnecessary human effort in a real-world environment.\n",
    "\n",
    "* The 2nd rule is for the DLN field, the unique identifier of the Tax form, which is a must-have for the downstream processing logic. This rule will fail if the DLN field misses or with an empty value.\n",
    "\n",
    "* The 3rd rule is also for DLN field but with a different condition type “LengthCheck”. The rule will break if the DLN length is not 16 characters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rules = [\n",
    "    {\n",
    "        \"description\": \"Employee Id confidence score should greater than 99\",\n",
    "        \"field_name\": \"d.employer_id\",\n",
    "        \"field_name_regex\": None, # support Regex: \"_confidence$\",\n",
    "        \"condition_category\": \"Confidence\",\n",
    "        \"condition_type\": \"ConfidenceThreshold\",\n",
    "        \"condition_setting\": \"99\",\n",
    "    },\n",
    "    {\n",
    "        \"description\": \"dln is required\",\n",
    "        \"field_name\": \"dln\",\n",
    "        \"condition_category\": \"Required\",\n",
    "        \"condition_type\": \"Required\",\n",
    "        \"condition_setting\": None,\n",
    "    },\n",
    "    {\n",
    "        \"description\": \"dln length should be 16\",\n",
    "        \"field_name\": \"dln\",\n",
    "        \"condition_category\": \"LengthCheck\",\n",
    "        \"condition_type\": \"ValueRegex\",\n",
    "        \"condition_setting\": \"^[0-9a-zA-Z]{16}$\",\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information about the rule definition:\n",
    "* *description*: the description of the rule. \n",
    "* *field_name*: the field name defined in the Data Model JSON. (in the Define Data Model section)\n",
    "* *field_name_regex*: The regular expression applies to field_name when you want to apply the same rule to multiple fields. E.g., applying the rule to all fields with the prefix “part1” will need field_name_regex value “part1$” in a standard regular expression format. Note, filed_name will be ignored when “field_name_regex” is specified.\n",
    "* *condition_category*: The category of the condition for display and tracking purpose. In one of these values: \"Required\", \"Confidence\", \"LengthCheck\", \"ValueCheck\"\n",
    "* *condition_type*: The type of the condition in one of these values: \"Required\", \"ValueRegex\", \"ConfidenceThreshold\"\n",
    "* *condition_setting*: Regular Expression string when the condition_type is “ValueRegex”. None when the condition_type is “RequiredField”."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We have the data extracted from the document and the rules defined. Now let's evaluate the data against these rules.\n",
    "The `Condition` class is a generic `Rules Engine` that takes 2 parameters: the data (page1 object) and the conditions we defined above. It will return 2 lists due to met and failed conditions. We then can send the document to A2I for human review if any conditions fail.\n",
    "\n",
    "The `Condition` class source code locates in the a2idata folder `condition.py` file. It supports basic validation logic, such as validating a string's length, value range, and confidence score threshold. You can modify the code to support more condition types for complex validation logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from a2idata.condition import Condition\n",
    "\n",
    "# Validate business rules:\n",
    "con = Condition(page1, rules)\n",
    "rule_missed, rule_satisfied = con.check_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print out the list of failed business rules\n",
    "rule_missed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see 2 conditions that failed by design:\n",
    "- The 1st condition expects the DLN confidence score higher than 99%, but the acutal Textract confidence is 98%.\n",
    "- The 3rd condition does a length check of the DLN number and expects the length should be 16 exact, but the actual length is 15. \n",
    "\n",
    "In the next step, we will send this list of failed conditions to A2I for human review."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3.2: Setup customized A2I UI template and workforce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon A2I allows you to customize the reviewer’s web view by defining Work Task Template (https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-custom-templates.html). The template is a static web page in HTML and JavaScript. You can pass data to the customized reviewer page leveraging the Liquid (https://shopify.github.io/liquid/) syntax.\n",
    "In the below sample, the custom template shows the PDF page on the left and the unsatisfied conditions on the right. Reviewers can correct the extraction value along with their comments. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In the previous lab: notebook 04-idp-document-a2i.ipynb, we have set up the the below resources:\n",
    "* A **Work Team (WHO)** authenticates the group of workers you have selected to review the tasks.\n",
    "* A **Human Task UI (WHAT)** defines what the reviewer will see in the A2I console when reviewing the task, using the default Textract/A2I template.\n",
    "* A **Workflow Definition (WHEN)** wrapping the above information and defining the conditions when the human review should trigger, using the default Textract A2I condition template.\n",
    "\n",
    "In this lab, we will check if the account already has a work team. The below code will get the work team ARN and store it in a local variable. If you get an error from the above step. Follow this instruction to set up the workforce and run the below cell again: \n",
    "https://catalog.workshops.aws/intelligent-document-processing/en-US/02-getting-started/module-4-human-review#setup-an-a2i-human-review-workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the existing workforce arn\n",
    "work_team_arn = sagemaker.list_workteams()[\"Workteams\"][0][\"WorkteamArn\"]\n",
    "work_team_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new A2I Work Task Template - this is the Liquid HTML page you use to customize the reviewer UI. (The HTML template stores at a2idata/a2i-custom-ui.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the UI template from a2i-data directory\n",
    "template = \"\"\n",
    "with open('a2idata/a2i-custom-ui.html','r') as f:\n",
    "    template = f.read()\n",
    "\n",
    "resp = sagemaker.create_human_task_ui(\n",
    "        HumanTaskUiName=\"a2i-custom-ui-demo\",\n",
    "        UiTemplate={'Content': template})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep the new UI template ARN in a variable\n",
    "ui_template_arn = resp[\"HumanTaskUiArn\"]\n",
    "ui_template_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new human review workflow to wrap up all the information A2I needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = sagemaker.create_flow_definition(\n",
    "        FlowDefinitionName= \"a2i-custom-ui-demo-workflow\",\n",
    "        RoleArn= role,\n",
    "        HumanLoopConfig= {\n",
    "            \"WorkteamArn\": work_team_arn,\n",
    "            \"HumanTaskUiArn\": ui_template_arn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"A2I custom business rule and UI demo workflow\",\n",
    "            \"TaskTitle\": \"Custom rule sample task\"\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : f's3://{data_bucket}/a2i/output/'\n",
    "        }\n",
    "    )\n",
    "\n",
    "workflow_definition_arn = resp['FlowDefinitionArn']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new A2I UI template and the Workflow definition are in place. Let's send the missed conditions to the Workflow, so a reviewer can verify the result using A2I."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "human_loop_name = 'custom-loop-' + str(uuid.uuid4())\n",
    "\n",
    "# Construct the data send to the custom A2I human review task\n",
    "a2i_payload = {\n",
    "                \"InputContent\": json.dumps({\n",
    "                    \"Results\": {\n",
    "                        \"ConditionMissed\": rule_missed,\n",
    "                        \"ConditionSatisfied\": rule_satisfied\n",
    "                    },\n",
    "                    \"s3\":{\n",
    "                        \"bucket\":data_bucket,\n",
    "                        \"path\":s3_key,\n",
    "                        \"url\": f's3://{data_bucket}/{s3_key}',\n",
    "                        \"image_width\": image_width,\n",
    "                        \"image_height\": image_height\n",
    "                    },\n",
    "                    \"text\": \"990 Tax Form Page 1\",\n",
    "                })\n",
    "            }\n",
    "\n",
    "# Start the human loop task\n",
    "start_loop_response = a2i.start_human_loop(\n",
    "            HumanLoopName=human_loop_name,\n",
    "            FlowDefinitionArn=workflow_definition_arn,\n",
    "            HumanLoopInput=a2i_payload)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_loop_arn = start_loop_response[\"HumanLoopArn\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check status of the Human Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2i.describe_human_loop(HumanLoopName=human_loop_name)[\"HumanLoopStatus\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below cell will print out the A2I console URL, which you can use to log in using the credential received when setting up the Workforce to review the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "work_team_name = work_team_arn[work_team_arn.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker.describe_workteam(WorkteamName=work_team_name)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the A2I console, you should see a task in the list. Click on the \"Start Working\" button, and A2I will bring you to the customized UI page that looks like the below. \n",
    "\n",
    "Below is a screenshot of the customized A2I UI. It shows the original image document on the left and the 2 failed conditions on the right (We defined the conditions to fail on purpose):\n",
    "\n",
    "* The DLN numbers should be 16 characters long. The actual DLN has 15 characters.\n",
    "* The field employer_id’s confidence score is lower than 99%. The actual confidence score is around 98%.\n",
    "\n",
    "\n",
    "![A2I custom UI](./images/a2i-custom-ui.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you review the task and click on the Sumbit button. The human review task status will change to \"Completed\".\n",
    "\n",
    "A2I will store a JSON file in the S3 bucket once the review is submitted. The JSON will include the original data sent to A2I and the reviewer's input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2i_resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "print(\"Human Loop task status: \", a2i_resp[\"HumanLoopStatus\"])\n",
    "print(\"Human Loop output: \", a2i_resp[\"HumanLoopOutput\"][\"OutputS3Uri\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check A2I generated JSON\n",
    "Now, let's download the A2I output file and print it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3.download_file(data_bucket, a2i_resp[\"HumanLoopOutput\"][\"OutputS3Uri\"].replace(f's3://{data_bucket}/',''), 'a2i-output.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `humanAnswers` field in the JSON file contains the reviewer's input. The `inputContent` field contains the original data sent to A2I."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open('a2i-output.json','r') as f:\n",
    "    print(json.dumps(json.loads(f.read()), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Expand the solution to support more documents and business rules\n",
    "\n",
    "To expand the solution to support more document pages with corresponding business rules, you will need to make changes in the below 3 places:\n",
    "\n",
    "* Create a data model for the new page: in JSON structure representing all the values you want to extract out of the pages. \n",
    "* The extraction logic: leverage Amazon Textract to extract text out of the document and populate value to the data model.\n",
    "* Add business rules corresponding to the page in JSON format. \n",
    "\n",
    "The custom A2I UI in the solution is generic, which doesn’t require a change to support new business rules.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Build BI dashboard using QuickSight  <a id=\"step4\"></a>\n",
    "\n",
    "In the below section, we will build a BI dashboard using Amazon QuickSight to get insights into the IDP pipeline.  \n",
    "Below is a screenshot of the Amazon QuickSight dashboard. It includes widgets presenting numbers of the documents processed automatically or requiring human review. The primary reason caused the document to require human review and a histogram plot of the number of documents processed daily.\n",
    "\n",
    "You can expand the dashboard by including more data and visuals to get insights and support business decisions.\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-dashboard.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## QuickSight Initial Setup \n",
    "You will need author access to a QuickSight Enterprise Account for this workshop.\n",
    "\n",
    "If you don't have a QuickSight account already, steps to create one are given below.\n",
    "\n",
    "***Setup QuickSight***\n",
    "\n",
    "1. Launch AWS Console (https://console.aws.amazon.com ) in a new browser tab, search for QuickSight and launch it.\n",
    "2. On QuickSight page, click Sign up for QuickSight button.\n",
    "3. Keep the default Enterprise edition, scroll down and click Continue button.\n",
    "4. Enter QuickSight account name & Notification email address. Be sure to choose a name that is relevant and applicable to your entire user pool. Enter your official email as the notification email.\n",
    "5. Scroll down and click Finish button. (It can take 15-30 Secs to set up the account)\n",
    "6. Click Go to Amazon QuickSight button. You will now be taken to QuickSight console.\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-init.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, we have a CSV file under the a2idata folder called a2i-bi-sample-data.csv, which you can use to build the QuickSight dashboard. It is a sample dataset to start with. You can develop your ETL process to transform the A2I JSON data to your preferred format.\n",
    "\n",
    "**Add Dataset**\n",
    "\n",
    "1. Download the CSV file to your local drive from a2idata/a2i-bi-sample-data.csv\n",
    "2. On the QuickSight page, click on \"Datasets\" then click on \"Add new dataset\" button on the top-right side\n",
    "3. Click on \"Upload a file\" then choose the csv file downloaded in step 1\n",
    "4. Click \"Next\" on the confirm file upload setting\" page.\n",
    "5. click on \"Visualize\" button then QuickSight will navigate to the Analyses page\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-dataset.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the dataset adds to QuickSight, you will navigate to the Analyses page managing visuals. Let's create a Pie chart to show the total number of documents processed automatically vs. with human review.\n",
    "\n",
    "**Create a Pie Chart**\n",
    "1. Select the Pie chart in the \"Visual Type\" section\n",
    "2. Drag the \"process_method\" field to the first drop-down list in Field wells\n",
    "3. Drag the \"doc_id\" field to the 2nd drop-down list in Field wells, then change the aggregation type from the default \"count\" to \"Count distinct\"\n",
    "4. Change the Visual display summary by double-clicking the visual to \"Numbers of documents processed automatically vs. human review\" then click \"Save\"\n",
    "\n",
    "You now get the first visual ready.\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-visual-pie.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create another visual that shows the field(s) that caused most of the human review. So you could optimize the workflow based on the insights.\n",
    "\n",
    "**Create a Word Cloud Visual**\n",
    "1. Click \"Add\" - > \"Add Visual\" on the top-left menu\n",
    "2. Select the Word Cloud in the \"Visual Type\" section\n",
    "3. Drag the \"field_name\" field to the first drop-down list in Field wells\n",
    "\n",
    "We need to filter the dataset for this Visual, so it only shows fields from the \"manu\" tasks\n",
    "\n",
    "4. Click \"Filter\" on the left menu, then click \"Create one...\" link\n",
    "5. Choose \"process_method\" field which you will apply the filter\n",
    "6. Click on \"include all\" to change the default filter setting\n",
    "7. Uncheck \"auto\" from the list, then click \"Apply\"\n",
    "\n",
    "Now the visual will ignore the \"auto\" task in the dataset\n",
    "\n",
    "8. Change the Visual display summary by double-clicking the visual to \"Fields caused most of the human review\" then click \"Save\"\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-visual-wordcloud.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add a 3rd visual, a histogram showing the numbers of documents processed daily by humans vs. automation. \n",
    "\n",
    "**Create a Histogram Visual**\n",
    "1. Click \"Add\" - > \"Add Visual\" on the top-left menu\n",
    "2. Select the Line Chart in the \"Visual Type\" section\n",
    "3. Drag the \"timestamp\" field to the first drop-down list in Field wells\n",
    "4. Drag the \"doc_id\" field to the second drop-down list in Field wells and change the aggregation type of Count distinct\n",
    "5. Drag the \"process_type\" field to the third drop-down list in Field wells\n",
    "6. Change the Visual display summary by double-clicking the visual to \"Numbers documents processed daily\" then click \"Save\"\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-visual-histogram.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have 3 visuals show insights of the IDP A2I workflow. You can publish them to a dashboard by following the below steps:\n",
    "1. Click \"Share\" on the top-right menu and choose \"Publish a dashboard\"\n",
    "2. For the first time publishing a dashboard, type a name in the textbox under \"Publish new dashboard as\" then click \"Publish Dashboard\"\n",
    "You can set up access control in QuickSight to share the dashboard with the other users.\n",
    "\n",
    "![A2I custom UI](./images/a2i-quicksight-publish.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Cleanup\n",
    "\n",
    "Cleanup is optional if you want to execute subsequent notebooks. \n",
    "\n",
    "Refer to the `05-idp-cleanup.ipynb` for cleanup and deletion of resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Conclusion\n",
    "\n",
    "In this notebook, we built a pipeline to extract data from the first page 990 Tax form using Textract, applied customized business rules against the extracted data, and then used the customized A2I UI to review the result. In the end, we also put together a QuickSight dashboard to get an insight into the overall workflow. \n",
    "\n",
    "Intelligent Document Processing is in high demand, and companies need a customized pipeline to support their unique business logic. Amazon A2I offers a built-in template integrated with Amazon Textract support for common human review use cases. It also allows you to customize the reviewer page to serve flexible requirements. "
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}