Email Connector: Send Email with Erasure Instructions [#1158]

CHANGELOG.md

@@ -31,6 +31,7 @@ The types of changes are:
 * Foundations for a new email connector type [#1142](https://github.com/ethyca/fidesops/pull/1142)
 * Have the new email connector cache action needed for each collection [#1168](https://github.com/ethyca/fidesops/pull/1168)
 * Added `execution_timeframe` to Policy model and schema [#1244](https://github.com/ethyca/fidesops/pull/1244)
+* Wrap up the email connector - it sends an email with erasure instructions as part of request execution [#1246](https://github.com/ethyca/fidesops/pull/1246)
 
 ### Docs
 

docs/fidesops/docs/guides/email_communications.md

@@ -6,7 +6,7 @@ Fidesops supports email server configurations for sending processing notices to
 Supported modes of use:
 
 - Subject Identity Verification - for more information on identity verification in subject requests, see the [Privacy Requests](privacy_requests.md#subject-identity-verification) guide.
-
+- Erasure Request Email Fulfillment - sends an email to configured third parties to process erasures for a given data subject.  See [Email Connectors](email_connectors.md) for more information.
 
 ## Prerequisites
 

docs/fidesops/docs/guides/email_connectors.md

@@ -0,0 +1,106 @@
+# Setting up an Email Connector
+
+
+## What is the purpose of the Email Connector?
+
+The Email Connector is a ConnectionConfig type that emails a third party to ask them to complete
+a Privacy Request when their service cannot be accessed automatically. 
+
+Fidesops will gather details about each collection described on the third party service as part of request execution and 
+wait to send a single email to the service after all collections have been visited.  Fidesops does not 
+collect confirmation that the erasure was completed by the third party; the EmailConnector is only responsible 
+for notifying them.
+
+Importantly, only *erasure* requests are supported at this time for EmailConnectors.  
+
+
+## What pieces should be configured for an Email Connector?
+
+In short, you will need to create an `email` ConnectionConfig to store minor details about the third party service, and a DatasetConfig
+to describe the data contained in their datasource.  You will also need to have configured a separate [EmailConfig](email_communications.md) which
+is used system-wide to carry out the actual email send.
+
+
+## Setting up the ConnectionConfig
+
+Create a ConnectionConfig object with
+
+```json title="<code>PATCH api/v1/connection</code>"
+[
+  { 
+    "name": "Email Connection Config",
+    "key": "third_party_email_connector",
+    "connection_type": "email",
+    "access": "write"
+  }
+]
+```
+EmailConnectors must be given "write" access in order to send an email.
+
+
+## Configuring who to notify
+
+Save a `to_email` on the ConnectionConfig secrets.  This is the user that will be notified via email to complete
+an erasure. Only one `to_email` is supported at this time.
+
+Optionally, configure a `test_email` to which you have access, to verify that your setup is working.  Provided your 
+EmailConfig is set up properly, you should receive an email similar to the one sent to third-party services, containing
+dummy data.
+
+```json title="<code>PUT api/v1/connection/<email_connection_config_key>/secret</code>" 
+{
+    "test_email": "[email protected]",
+    "to_email": "[email protected]
+}
+```
+
+## Configuring the dataset
+
+Describe the collections and fields on a third party source with a [DatasetConfig](datasets.md), the same way you'd describe attributes 
+on a database.  If you do not know the exact data structure of a third party, you might configure a single collection
+with the fields you'd like masked.
+
+As with all collections that support erasures, a primary key must be specified on each collection.
+
+
+```json title="<code>PUT api/v1/connection/<email_connection_config_key>/dataset" 
+[
+    {
+      "fides_key": "email_dataset",
+      "name": "Dataset not accessible automatically",
+      "description": "Third party data - will email to request erasure",
+      "collections": [
+        {
+          "name": "daycare_customer",
+          "fields": [
+            {
+              "name": "id",
+              "data_categories": [
+                "system.operations"
+              ],
+              "fidesops_meta": {
+                "primary_key": true
+              }
+            },
+            {
+              "name": "child_health_concerns",
+              "data_categories": [
+                "user.biometric_health"
+              ]
+            },
+            {
+              "name": "user_email",
+              "data_categories": [
+                "user.contact.email"
+              ],
+              "fidesops_meta": {
+                "identity": "email"
+              }
+            }
+          ]
+        }
+      ]
+    }
+]
+```
+

docs/fidesops/mkdocs.yml

@@ -28,6 +28,7 @@ nav:
       - Configure OneTrust Integration: guides/onetrust.md
       - Preview Query Execution: guides/query_execution.md
       - Data Rights Protocol: guides/data_rights_protocol.md
+      - Configure Email Connectors: guides/email_connectors.md
   - SaaS Connectors:
       - Connect to SaaS Applications: saas_connectors/saas_connectors.md
       - SaaS Configuration: saas_connectors/saas_config.md

src/fidesops/ops/api/v1/endpoints/privacy_request_endpoints.py

@@ -79,7 +79,7 @@
 from fidesops.ops.schemas.privacy_request import (
     BulkPostPrivacyRequests,
     BulkReviewResponse,
-    CollectionActionRequired,
+    CheckpointActionRequired,
     DenyPrivacyRequests,
     ExecutionLogDetailResponse,
     PrivacyRequestCreate,
@@ -492,33 +492,33 @@ def attach_resume_instructions(privacy_request: PrivacyRequest) -> None:
     about how to resume manually if applicable.
     """
     resume_endpoint: Optional[str] = None
-    stopped_collection_details: Optional[CollectionActionRequired] = None
+    action_required_details: Optional[CheckpointActionRequired] = None
 
     if privacy_request.status == PrivacyRequestStatus.paused:
-        stopped_collection_details = privacy_request.get_paused_collection_details()
+        action_required_details = privacy_request.get_paused_collection_details()
 
-        if stopped_collection_details:
+        if action_required_details:
             # Graph is paused on a specific collection
             resume_endpoint = (
                 PRIVACY_REQUEST_MANUAL_ERASURE
-                if stopped_collection_details.step == CurrentStep.erasure
+                if action_required_details.step == CurrentStep.erasure
                 else PRIVACY_REQUEST_MANUAL_INPUT
             )
         else:
             # Graph is paused on a pre-processing webhook
             resume_endpoint = PRIVACY_REQUEST_RESUME
 
     elif privacy_request.status == PrivacyRequestStatus.error:
-        stopped_collection_details = privacy_request.get_failed_collection_details()
+        action_required_details = privacy_request.get_failed_checkpoint_details()
         resume_endpoint = PRIVACY_REQUEST_RETRY
 
-    if stopped_collection_details:
-        stopped_collection_details.step = stopped_collection_details.step.value  # type: ignore
-        stopped_collection_details.collection = (
-            stopped_collection_details.collection.value  # type: ignore
+    if action_required_details:
+        action_required_details.step = action_required_details.step.value  # type: ignore
+        action_required_details.collection = (
+            action_required_details.collection.value if action_required_details.collection else None  # type: ignore
         )
 
-    privacy_request.stopped_collection_details = stopped_collection_details
+    privacy_request.action_required_details = action_required_details
     # replaces the placeholder in the url with the privacy request id
     privacy_request.resume_endpoint = (
         resume_endpoint.format(privacy_request_id=privacy_request.id)
@@ -784,7 +784,10 @@ async def resume_privacy_request_with_manual_input(
     manual_rows: List[Row] = [],
     manual_count: Optional[int] = None,
 ) -> PrivacyRequest:
-    """Resume privacy request after validating and caching manual data for an access or an erasure request."""
+    """Resume privacy request after validating and caching manual data for an access or an erasure request.
+
+    This assumes the privacy request is being resumed from a specific collection in the graph.
+    """
     privacy_request: PrivacyRequest = get_privacy_request_or_error(
         db, privacy_request_id
     )
@@ -796,7 +799,7 @@ async def resume_privacy_request_with_manual_input(
         )
 
     paused_details: Optional[
-        CollectionActionRequired
+        CheckpointActionRequired
     ] = privacy_request.get_paused_collection_details()
     if not paused_details:
         raise HTTPException(
@@ -805,7 +808,7 @@ async def resume_privacy_request_with_manual_input(
         )
 
     paused_step: CurrentStep = paused_details.step
-    paused_collection: CollectionAddress = paused_details.collection
+    paused_collection: Optional[CollectionAddress] = paused_details.collection
 
     if paused_step != expected_paused_step:
         raise HTTPException(
@@ -818,6 +821,12 @@ async def resume_privacy_request_with_manual_input(
     dataset_graphs = [dataset_config.get_graph() for dataset_config in datasets]
     dataset_graph = DatasetGraph(*dataset_graphs)
 
+    if not paused_collection:
+        raise HTTPException(
+            status_code=HTTP_422_UNPROCESSABLE_ENTITY,
+            detail="Cannot save manual data on paused collection. No paused collection saved'.",
+        )
+
     node: Optional[Node] = dataset_graph.nodes.get(paused_collection)
     if not node:
         raise HTTPException(
@@ -939,16 +948,16 @@ async def restart_privacy_request_from_failure(
         )
 
     failed_details: Optional[
-        CollectionActionRequired
-    ] = privacy_request.get_failed_collection_details()
+        CheckpointActionRequired
+    ] = privacy_request.get_failed_checkpoint_details()
     if not failed_details:
         raise HTTPException(
             status_code=HTTP_400_BAD_REQUEST,
             detail=f"Cannot restart privacy request from failure '{privacy_request.id}'; no failed step or collection.",
         )
 
     failed_step: CurrentStep = failed_details.step
-    failed_collection: CollectionAddress = failed_details.collection
+    failed_collection: Optional[CollectionAddress] = failed_details.collection
 
     logger.info(
         "Restarting failed privacy request '%s' from '%s step, 'collection '%s'",
@@ -964,7 +973,7 @@ async def restart_privacy_request_from_failure(
         from_step=failed_step.value,
     )
 
-    privacy_request.cache_failed_collection_details()  # Reset failed step and collection to None
+    privacy_request.cache_failed_checkpoint_details()  # Reset failed step and collection to None
 
     return privacy_request
 

src/fidesops/ops/common_exceptions.py

@@ -101,6 +101,10 @@ class PrivacyRequestPaused(BaseException):
     """Halt Instruction Received on Privacy Request"""
 
 
+class PrivacyRequestErasureEmailSendRequired(BaseException):
+    """Erasure requests will need to be fulfilled by email send.  Exception is raised to change ExecutionLog details"""
+
+
 class SaaSConfigNotFoundException(FidesopsException):
     """Custom Exception - SaaS Config Not Found"""
 

src/fidesops/ops/email_templates/get_email_template.py

@@ -5,6 +5,7 @@
 
 from fidesops.ops.common_exceptions import EmailTemplateUnhandledActionType
 from fidesops.ops.email_templates.template_names import (
+    EMAIL_ERASURE_REQUEST_FULFILLMENT,
     SUBJECT_IDENTITY_VERIFICATION_TEMPLATE,
 )
 from fidesops.ops.schemas.email.email import EmailActionType
@@ -22,6 +23,8 @@
 def get_email_template(action_type: EmailActionType) -> Template:
     if action_type == EmailActionType.SUBJECT_IDENTITY_VERIFICATION:
         return template_env.get_template(SUBJECT_IDENTITY_VERIFICATION_TEMPLATE)
+    if action_type == EmailActionType.EMAIL_ERASURE_REQUEST_FULFILLMENT:
+        return template_env.get_template(EMAIL_ERASURE_REQUEST_FULFILLMENT)
 
     logger.error("No corresponding template linked to the %s", action_type)
     raise EmailTemplateUnhandledActionType(

src/fidesops/ops/email_templates/template_names.py

@@ -1 +1,2 @@
 SUBJECT_IDENTITY_VERIFICATION_TEMPLATE = "subject_identity_verification.html"
+EMAIL_ERASURE_REQUEST_FULFILLMENT = "erasure_request_email_fulfillment.html"

src/fidesops/ops/email_templates/templates/erasure_request_email_fulfillment.html

@@ -0,0 +1,30 @@
+<!DOCTYPE html>
+<html lang="en">
+   <head>
+      <meta charset="UTF-8">
+      <title>Erasure request</title>
+   </head>
+   <body>
+      <main>
+         <p>Please locate and erase the associated records from the following collections:</p>
+         {% for collection, action_required in dataset_collection_action_required.items() -%}
+         <p><b>{{ collection }}</b></p>
+         {% for action in action_required.action_needed -%}
+         <p>Locate the relevant records with:</p>
+         <ul>
+            {% for field, values in action.locators.items() -%}
+            <li> Field: <i>{{ field }}</i>, Values: {{ values|join(', ') }} </li>
+            {%- endfor %}
+         </ul>
+         {% if action.update -%}
+         <p>Erase the following fields:</p>
+         <ul>
+            {% for field_name, masking_strategy in action.update.items() -%}
+            <li><i>{{field_name}}</i></li>
+            {%- endfor %}
+         </ul>
+         {%- endif %}
+          {%- endfor %} {%- endfor %}
+      </main>
+   </body>
+</html>

src/fidesops/ops/migrations/versions/912d801f06c0_audit_log_email_send.py

@@ -0,0 +1,32 @@
+"""audit log email send
+
+Revision ID: 912d801f06c0
+Revises: bde646a6f51e
+Create Date: 2022-09-01 16:23:10.905356
+
+"""
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "912d801f06c0"
+down_revision = "bde646a6f51e"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.execute("alter type auditlogaction add value 'email_sent'")
+
+
+def downgrade():
+    op.execute("delete from auditlog where action in ('email_sent')")
+    op.execute("alter type auditlogaction rename to auditlogaction_old")
+    op.execute("create type auditlogaction as enum('approved', 'denied', 'finished')")
+    op.execute(
+        (
+            "alter table auditlog alter column action type auditlogaction using "
+            "action::text::auditlogaction"
+        )
+    )
+    op.execute("drop type auditlogaction_old")

src/fidesops/ops/models/policy.py

@@ -27,8 +27,11 @@
 
 
 class CurrentStep(EnumType):
+    pre_webhooks = "pre_webhooks"
     access = "access"
     erasure = "erasure"
+    erasure_email_post_send = "erasure_email_post_send"
+    post_webhooks = "post_webhooks"
 
 
 class ActionType(str, EnumType):