DISCO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b29ad0cb-6f63-4bfd-ae07-194450859e86",
   "metadata": {},
   "source": [
    "# MRP Outputs Review"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4319eb54-2740-4fbb-a67c-39f34ff39357",
   "metadata": {},
   "source": [
    "## What this notebook does"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d211be1-86cb-4e9a-bd79-b6ccad65faa5",
   "metadata": {},
   "source": [
    "This notebook has been set up to read in csvs, jpegs, and html files which have been sent to the review bucket, so that they can be checked for quality and disclosure. All items which are requested for export first needs to be Disclosure controlled by a seperate DisCO (Disclosure Control Officer), this cannot be the same person who ran the data and requested the export. Once disclosure controlled a Data Journey manager can be notified to move the data ready for export.\n",
    "\n",
    "For additional info see 'DISCO GUIDE IN POLESTAR PLUS' within the Polestar plus Disco resource area of sharepoint."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9121e9a-f15c-4672-bf2d-b776af87b350",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa588865-941a-4f45-bd9a-49a7a95cfbd6",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to set the script up to run <br>\n",
    "<em>(For example, by clicking on cell and pressing ctrl+enter)</em>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "458d82d1-15e4-4a08-a413-d68774b4be4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Authentification for GCS\n",
    "options(\"googleAuthR.httr_oauth_cache\" = \"gce.oauth\")\n",
    "googleAuthR::gar_gce_auth()\n",
    "devtools::install(\"../gcptools\", upgrade = FALSE, quiet = TRUE)\n",
    "install.packages(\"jpeg\", quiet = TRUE)\n",
    "library(googleCloudStorageR)\n",
    "library(readr)\n",
    "library(dplyr ,quietly = TRUE)\n",
    "# SECURITY FEATURE FOR PREVENTING .ipynb FILES FROM ACCIDENTALLY UPLOADING DATA TO GITHUB [DO NOT REMOVE]\n",
    "gcptools::commit_hooks_setup(\"/home/jupyter/CIS_disclosure_control\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4f860d-f241-411a-9a52-64bde57355ea",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Project Name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f0e9b2e-3973-400d-9972-7ad87209ecd5",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Insert the project name in quotation marks to the right of the assignment opperator <br>\n",
    "<em>(keep a '/' on the end of the folder name, check there's no spaces) <br> \n",
    "<strong> (e.g. project_name <- \"20221121_mrp\")</strong></em>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe1ba7c4-7adc-4839-809f-a5b855d63c8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "project_name <- \"20221205_agecontour_swabs/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1b5a4ca-2836-4c8c-8bfa-9584bf554e09",
   "metadata": {},
   "outputs": [],
   "source": [
    "#' @title get_filepaths_from_project_name\n",
    "#' \n",
    "#' @description function to search all of the available filepaths witihin a given project_name and return it as a dataframe\n",
    "#'\n",
    "#' @param project_name a string containing the exact project name (case-sensitive) which you want to get the filepaths from\n",
    "#' \n",
    "#' @return dataframe containing the filepaths of all of the files within that project\n",
    "#'\n",
    "#' @export \n",
    "\n",
    "get_filepaths_from_project_name <- function(project_name, file_type = c(\".csv\", \".jpeg|.jpg\", \".yaml|.yml\", \".html\")){\n",
    "    \n",
    "    file_type <- match.arg(file_type)\n",
    "    \n",
    "    project_pattern <- paste0(\"^\", project_name)\n",
    "    \n",
    "    project_files <- grep(x = googleCloudStorageR::gcs_list_objects(\"polestar-prod-review\")$name,\n",
    "                   pattern = project_pattern,\n",
    "                   value = TRUE)\n",
    "    \n",
    "    approved_file_patterns <- file_type\n",
    "    \n",
    "    project_files <- grep(x = project_files,\n",
    "                   pattern = approved_file_patterns,\n",
    "                   value = TRUE)\n",
    "    \n",
    "    as.data.frame(project_files)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd7457f0-c512-4bd4-bd7d-bfa1f518e113",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Checking .csv files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20485f5d-5bba-4e40-a667-4f34e824e35e",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to get the names of the .csv file types in the project_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "795a3830-e10f-4dc1-a28b-370f1ad94e34",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_filepaths_from_project_name(project_name, \".csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4985845e-9ba0-4097-870e-ccf7ad3eb7be",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Select the csv filepath you want to review by replacing the string below with a filepath to the csv <br> \n",
    "<em>(keep .csv at end, check there's no spaces between \"\") <br>\n",
    "<em><strong> (e.g. csv_path <- \"20221121_mrp/probs_over_time_mrp_20221115_DTS221122_1411UTC.csv\") </strong></em>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c6d2da4-a9d2-4515-bad4-b0c324662891",
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_path <- \"20221205_agecontour_swabs/input_age_contour_swabs_20221126_DTS221206_1028UTC.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "586c1803-3393-4f29-85ed-8e420645545a",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to retrieve the data from the bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9c5e7c8-32e9-4995-a456-90a3d4ff0f59",
   "metadata": {},
   "outputs": [],
   "source": [
    "data <- suppressMessages(gcptools::gcp_read_csv(csv_path, bucket=\"review_bucket\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46fc379e-41a2-470a-8c6d-cad899cc03a4",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Preview what types of data are in the csv, by printing data to the console <br>\n",
    "Look specifically for columns which could contain identifiable information, such as columns which contain <em><strong><u> counts</em></strong></u>  or other identifiable information like names, addresses, emails etc. <br> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa30dee2-3f3d-43ff-ac6e-f99dfa712ebf",
   "metadata": {},
   "outputs": [],
   "source": [
    "head(data) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e5051fa-5bdf-46f6-88fe-ea3dc82f944d",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> To have a closer look at values within the columns, insert the column name below and run the next code chunk to see what's inside. <br> \n",
    "<em><strong> e.g column_name <- \"probability_increase\"  </em></strong> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5987bf4e-c182-44f1-9605-798ab3ab8a67",
   "metadata": {},
   "outputs": [],
   "source": [
    "column_name <- \"visit_date\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18b44e5c-80c4-40db-b083-dad60301d2ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "#' @title explore_column\n",
    "#' \n",
    "#' @description function for use in disclosure control to explore the contents of a particular column\n",
    "#'\n",
    "#' @param data a dataframe containing the data you want to explore\n",
    "#' @param column_name a string containing the exact name (case-sensitive) of the column you want to explore\n",
    "#' \n",
    "#' @return either unique values or a percentage of low counts under a given threshold (default 10) \n",
    "#'\n",
    "#' @export \n",
    "\n",
    "explore_column <- function(data, column_name){\n",
    "    \n",
    "    column_type <- typeof(data[[column_name]])\n",
    "\n",
    "    if(column_type == \"character\"){\n",
    "        get_unique_values(data, column_name)\n",
    "    } else if(column_type %in% c(\"double\", \"integer\")){\n",
    "        reveal_low_counts(data, column_name)\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "#' @title get_unique_values\n",
    "#' \n",
    "#' @description function for use in disclosure control to explore the contents of a character or factor type column\n",
    "#'\n",
    "#' @param data a dataframe containing the data you want to explore\n",
    "#' @param column_name a string containing the exact name (case-sensitive) of the column you want to explore\n",
    "#' \n",
    "#' @return a pretty list containing the unique values within that column\n",
    "#'\n",
    "#' @export \n",
    "\n",
    "get_unique_values <- function(data, column_name){\n",
    "    \n",
    "    unique_values <- data%>%\n",
    "        dplyr::pull(column_name)%>%\n",
    "        unique()\n",
    "    \n",
    "    cat(c(\"Unique values:\", unique_values), sep = \"\\n   \")\n",
    "\n",
    "}\n",
    "\n",
    "#' @title reveal_low_counts\n",
    "#' \n",
    "#' @description function for use in disclosure control to explore the percentage of low counts within a given numeric column\n",
    "#'\n",
    "#' @param data a dataframe containing the data you want to explore\n",
    "#' @param column_name a string containing the exact name (case-sensitive) of the column you want to explore\n",
    "#' @param low_count_threshold the threshold at which counts are considered disclosive (default 10)\n",
    "#' \n",
    "#' @return a message to the console which states the percentage of low counts within the column. \n",
    "#'\n",
    "#' @export \n",
    "\n",
    "reveal_low_counts <- function(data, column_name, low_count_threshold = 10){\n",
    "    \n",
    "    perc_less_than_10 <- data%>%\n",
    "        dplyr::filter(!!sym(column_name) < 10 )%>%\n",
    "        dplyr::summarise(perc_less_than_10 = paste0(round(n()/nrow(data),2)*100,\"%\"))%>%\n",
    "        dplyr::pull()\n",
    "    \n",
    "    paste(perc_less_than_10, \"of rows contain values less than 10\")\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1840e36d-adcb-4011-9cb0-05c8f61eb302",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> The explore function below's output depends on the variable type: <br>\n",
    "<strong> If categorical  </strong> (e.g. region) it lists the unique categories (e.g. wales, south east) <br>\n",
    "<strong> If numeric </strong> (e.g. numbers of postives) it lists the % of data within which falls below the low count threshold <br>\n",
    "<strong> If an error  </strong> nothing or a non relevant dataset is produced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52b1c8eb-4b92-421f-bea9-2cdc1d7b847c",
   "metadata": {},
   "outputs": [],
   "source": [
    "explore_column(data, column_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "604abe88-37f0-4503-98d6-b8285ccef781",
   "metadata": {},
   "outputs": [],
   "source": [
    "#' @title gcp_read_yaml\n",
    "#' \n",
    "#' @description function to read yaml files in to the environment from a gcpbucket\n",
    "#'\n",
    "#' @param filepath a string containing file path (including folder name) to the yaml file which you want to read in\n",
    "#' @param bucket a string containing the name of the gcp bucket you want to read the yaml file from \n",
    "#' \n",
    "#' @return a list containing the contents of the yaml file \n",
    "#'\n",
    "#' @export \n",
    "\n",
    "gcp_read_yaml <- function(filepath, bucket = \"polestar-prod-review\"){\n",
    "    \n",
    "    tmp <- tempfile(fileext = \".yaml\")\n",
    "    \n",
    "    suppressMessages(gcs_get_object(filepath,  bucket = bucket,  saveToDisk = tmp))\n",
    "    \n",
    "    yaml <- yaml::read_yaml(tmp)\n",
    "    \n",
    "    unlink(tmp)\n",
    "    \n",
    "    return(yaml)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3414d6d4-b363-4698-9668-3619d6d5c5cb",
   "metadata": {},
   "source": [
    "## Checking .yml/.yaml files "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "414e6d56-5484-4617-9264-99f2650d9e61",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to get the names of the .yml or .yaml file types in the project_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c72c624e-f534-453d-9011-e08487679f94",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_filepaths_from_project_name(project_name, \".yaml|.yml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "424463bf-b0f9-48fd-b75e-1a5d0abdb5b1",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Insert the filepath of the yaml file you want to explore <br> \n",
    "<em>(keep .yaml or .yml at end, check there's no spaces between \"\") <br>\n",
    "<em><strong> e.g yaml_filepath <- \"20221121_mrp/configs_mrp_20221115_DTS221215_1814UTC.yaml\"  </em></strong> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70fa64e7-2ff4-4e7b-9450-1ab745becd5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "yaml_filepath <- \"20221205_agecontour_swabs/config_age_contour_swabs_20221126_DTS221206_1028UTC.yaml\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8cf3011-a6df-474a-b4e4-15bf9b887dcf",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to retrieve the contents of the yaml file from the review bucket <br>\n",
    "If a table with 0s is produced, there isn'that file type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68250963-89ab-4b2d-8035-961697d4bda8",
   "metadata": {},
   "outputs": [],
   "source": [
    "gcp_read_yaml(yaml_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aab37c02-bd86-44d0-931f-c298ac823b32",
   "metadata": {},
   "source": [
    "## Checking .jpg/.jpeg files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bceaf72-ba04-4233-90e3-cc55c8039bda",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to get the names of the .jpg or .jpeg file types in the project <br>\n",
    "<em>(keep .jpeg or .jpg at end, check there's no spaces between \"\") <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f38d854a-d504-4b32-a413-a894af53614d",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_filepaths_from_project_name(project_name, \".jpeg|.jpg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c14373c-4509-4cd2-9665-a8559f5704fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "jpeg_filepath <- \"20221205_agecontour_swabs/plot_age_contour_swabs_regions_20221126_DTS221206_1028UTC.jpeg\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b318dc2-10e0-4bd4-9db1-feff82c6cff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#' @title print_jpeg\n",
    "#' \n",
    "#' @description function to print jpegs to the console\n",
    "#'\n",
    "#' @param jpeg_filepath a string containing a jpeg file path (including folder name) that you want to print to the console\n",
    "#' \n",
    "#' @return an image of the jpeg within the console\n",
    "#'\n",
    "#' @export \n",
    "\n",
    "print_jpeg <- function(jpeg_filepath){\n",
    "    \n",
    "    jpeg <- suppressMessages(googleCloudStorageR::gcs_get_object(jpeg_filepath, bucket = \"polestar-prod-review\"))\n",
    "\n",
    "    options(repr.plot.width = 25, repr.plot.height = 25)\n",
    "\n",
    "    plot(0:3000, type = 'n')\n",
    "\n",
    "    rasterImage(image = jpeg, 0, 0, 3000, 3000)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb06778-760d-4308-bbff-89b506ebcd17",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to print the jpeg to the console"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a482a3a1-483e-42a7-9d88-9af5171f04ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "print_jpeg(jpeg_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e01242-631b-42bf-95e6-5418caa5d22d",
   "metadata": {},
   "source": [
    "## Checking .html Files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb29a167-2712-491c-8c85-a300833a5527",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Run this code chunk to get the names of the .html file types in the project_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e86a3be-24bc-49a8-9bc4-c1d132d8d6ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_filepaths_from_project_name(project_name, \".html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ccbf8cc-26f0-4114-a0ff-f46774d4a53b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#' @title retrieve_html_file\n",
    "#' \n",
    "#' @description function to retrieve html file from the review bucket and put it in the QA_reports folder\n",
    "#'\n",
    "#' @param html_filepath a string containing a html file path (including folder name) that you want to save to the QA_reports folder\n",
    "#' \n",
    "#' @return a message saying where to find the html file\n",
    "#'\n",
    "#' @export \n",
    "\n",
    "retrieve_html_file <- function(html_filepath){\n",
    "\n",
    "    main_directory <- \"/home/jupyter\"\n",
    "    sub_directory <- \"QA_reports\"\n",
    "\n",
    "    if (!file.exists(paste0(main_directory,\"/\",sub_directory))){\n",
    "        dir.create(file.path(main_directory, sub_directory))\n",
    "        setwd(file.path(main_directory, sub_directory))   \n",
    "    }\n",
    "\n",
    "    suppressMessages(gcptools::download_qa_report_to_notebook(file = html_filepath, \n",
    "                                            bucket = \"polestar-prod-review\" ))\n",
    "\n",
    "    print(paste0(\"The \", html_filepath, \" file has been saved in the '\", main_directory,\"/\", sub_directory, \"' folder\"))\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6229fae-99ff-4a0f-a367-e721c9cf8dbd",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Replace the string in the 'file' argument to select the correct file from the review bucket<br> \n",
    "<em>(keep .html at end, check there's no spaces between \"\") <br>\n",
    "<em><strong> e.g html_filepath <- \"20221121_mrp/MRP_QA_England_Datarun20221121_Co20221115_PrevCo20221108.html\"  </em></strong> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "476500e1-8dfd-4920-b158-deb619162960",
   "metadata": {},
   "outputs": [],
   "source": [
    "html_filepath <- \"20221121_mrp/MRP_QA_England_Datarun20221121_Co20221115_PrevCo20221108.html\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f12f83d3-99ff-42af-9cfc-434b082e232f",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieve_html_file(html_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b2ca1ab-5966-4d71-884e-788caef8a49b",
   "metadata": {},
   "source": [
    "<FONT COLOR=\"RED\"> **INSTRUCTION:**</FONT> Locate the QA_reports folder at the repo (/root) level of folders (i.e. where you would go to navigate to other repo's) to review the report"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "ir",
   "name": "r-cpu.4-1.m96",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/r-cpu.4-1:m96"
  },
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.2.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}