Additional PIR opt out confirmation pixels (#3119)

Task/Issue URL: https://app.asana.com/0/0/1207733131726022/f
Tech Design URL:
CC:

**Description**:
I've linked to the implementation task above, but more useful is the
proposal task here: https://app.asana.com/0/0/1207999607996551/f
Adds new pixels that indicate how long it takes us to confirm successful
opt outs

This involves adding new fields to the DB, so the migration also needs
to be tested

Also moves monitoring pixel firing up the stack to the agent manager
where IMO it belongs, so special attention needs to be made that nothing
is broken there.

**Steps to test this PR**:
1. With DBP already setup, confirm via the DB view that OptOutJobs have
the new fields (createdDate, submittedSuccessfullyDate,
sevenDaysConfirmationPixelFired, fourteenDaysConfirmationPixelFired,
twentyOneDaysConfirmationPixelFired). Only submittedSuccessfullyDate
should be nil.
1. From a fresh start, set up DBP and confirm via the DB viewer again
that everything looks correct
1. Let DBP run for a bit (you could change some background activity
values to speed it up) and check that opt out created dates are set
correctly and that confirmed opt outs get the correct
submittedSuccessfullyDate.
1. You probably don't want to wait 7/14/21 days to test the actual pixel
firing, so in DataBrokerProtectionStatsPixels, change one of these to
just be true "let hasEnoughTimePassedToFirePixel =
submittedSuccessfullyDate.hasBeenExceededByNumberOfDays(14)" (e.g. to
"let hasEnoughTimePassedToFirePixel = true")

<!--
Tagging instructions
If this PR isn't ready to be merged for whatever reason it should be
marked with the `DO NOT MERGE` label (particularly if it's a draft)
If it's pending Product Review/PFR, please add the `Pending Product
Review` label.

If at any point it isn't actively being worked on/ready for
review/otherwise moving forward (besides the above PR/PFR exception)
strongly consider closing it (or not opening it in the first place). If
you decide not to close it, make sure it's labelled to make it clear the
PRs state and comment with more information.
-->

**Definition of Done**:

* [ ] Does this PR satisfy our [Definition of
Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)?

---
###### Internal references:
[Pull Request Review
Checklist](https://app.asana.com/0/1202500774821704/1203764234894239/f)
[Software Engineering
Expectations](https://app.asana.com/0/59792373528535/199064865822552)
[Technical Design
Template](https://app.asana.com/0/59792373528535/184709971311943)
[Pull Request
Documentation](https://app.asana.com/0/1202500774821704/1204012835277482/f)

DuckDuckGo/DBP/DataBrokerProtectionPixelsHandler.swift

@@ -82,6 +82,12 @@ public class DataBrokerProtectionPixelsHandler: EventMapping<DataBrokerProtectio
                     .monthlyActiveUser,
                     .weeklyReportScanning,
                     .weeklyReportRemovals,
+                    .optOutJobAt7DaysConfirmed,
+                    .optOutJobAt7DaysUnconfirmed,
+                    .optOutJobAt14DaysConfirmed,
+                    .optOutJobAt14DaysUnconfirmed,
+                    .optOutJobAt21DaysConfirmed,
+                    .optOutJobAt21DaysUnconfirmed,
                     .scanningEventNewMatch,
                     .scanningEventReAppearance,
                     .webUILoadingFailed,

LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/Database/DataBrokerProtectionDatabase.swift

@@ -37,6 +37,22 @@ protocol DataBrokerProtectionRepository {
     func updatePreferredRunDate(_ date: Date?, brokerId: Int64, profileQueryId: Int64, extractedProfileId: Int64) throws
     func updateLastRunDate(_ date: Date?, brokerId: Int64, profileQueryId: Int64) throws
     func updateLastRunDate(_ date: Date?, brokerId: Int64, profileQueryId: Int64, extractedProfileId: Int64) throws
+    func updateSubmittedSuccessfullyDate(_ date: Date?,
+                                         forBrokerId brokerId: Int64,
+                                         profileQueryId: Int64,
+                                         extractedProfileId: Int64) throws
+    func updateSevenDaysConfirmationPixelFired(_ pixelFired: Bool,
+                                               forBrokerId brokerId: Int64,
+                                               profileQueryId: Int64,
+                                               extractedProfileId: Int64) throws
+    func updateFourteenDaysConfirmationPixelFired(_ pixelFired: Bool,
+                                                  forBrokerId brokerId: Int64,
+                                                  profileQueryId: Int64,
+                                                  extractedProfileId: Int64) throws
+    func updateTwentyOneDaysConfirmationPixelFired(_ pixelFired: Bool,
+                                                   forBrokerId brokerId: Int64,
+                                                   profileQueryId: Int64,
+                                                   extractedProfileId: Int64) throws
     func updateRemovedDate(_ date: Date?, on extractedProfileId: Int64) throws
 
     func add(_ historyEvent: HistoryEvent) throws
@@ -221,6 +237,86 @@ final class DataBrokerProtectionDatabase: DataBrokerProtectionRepository {
         }
     }
 
+    func updateSubmittedSuccessfullyDate(_ date: Date?,
+                                         forBrokerId brokerId: Int64,
+                                         profileQueryId: Int64,
+                                         extractedProfileId: Int64) throws {
+        do {
+            let vault = try self.vault ?? DataBrokerProtectionSecureVaultFactory.makeVault(reporter: secureVaultErrorReporter)
+
+            try vault.updateSubmittedSuccessfullyDate(
+                date,
+                forBrokerId: brokerId,
+                profileQueryId: profileQueryId,
+                extractedProfileId: extractedProfileId
+            )
+        } catch {
+            os_log("Database error: updateSubmittedSuccessfullyDate, error: %{public}@", log: .error, error.localizedDescription)
+            pixelHandler.fire(.generalError(error: error, functionOccurredIn: "DataBrokerProtectionDatabase.updateSubmittedSuccessfullyDate date forBrokerId profileQueryId extractedProfileId"))
+            throw error
+        }
+    }
+
+    func updateSevenDaysConfirmationPixelFired(_ pixelFired: Bool,
+                                               forBrokerId brokerId: Int64,
+                                               profileQueryId: Int64,
+                                               extractedProfileId: Int64) throws {
+        do {
+            let vault = try self.vault ?? DataBrokerProtectionSecureVaultFactory.makeVault(reporter: secureVaultErrorReporter)
+
+            try vault.updateSevenDaysConfirmationPixelFired(
+                pixelFired,
+                forBrokerId: brokerId,
+                profileQueryId: profileQueryId,
+                extractedProfileId: extractedProfileId
+            )
+        } catch {
+            os_log("Database error: updateSevenDaysConfirmationPixelFired, error: %{public}@", log: .error, error.localizedDescription)
+            pixelHandler.fire(.generalError(error: error, functionOccurredIn: "DataBrokerProtectionDatabase.updateSevenDaysConfirmationPixelFired pixelFired forBrokerId profileQueryId extractedProfileId"))
+            throw error
+        }
+    }
+
+    func updateFourteenDaysConfirmationPixelFired(_ pixelFired: Bool,
+                                                  forBrokerId brokerId: Int64,
+                                                  profileQueryId: Int64,
+                                                  extractedProfileId: Int64) throws {
+        do {
+            let vault = try self.vault ?? DataBrokerProtectionSecureVaultFactory.makeVault(reporter: secureVaultErrorReporter)
+
+            try vault.updateFourteenDaysConfirmationPixelFired(
+                pixelFired,
+                forBrokerId: brokerId,
+                profileQueryId: profileQueryId,
+                extractedProfileId: extractedProfileId
+            )
+        } catch {
+            os_log("Database error: updateFourteenDaysConfirmationPixelFired, error: %{public}@", log: .error, error.localizedDescription)
+            pixelHandler.fire(.generalError(error: error, functionOccurredIn: "DataBrokerProtectionDatabase.updateFourteenDaysConfirmationPixelFired pixelFired forBrokerId profileQueryId extractedProfileId"))
+            throw error
+        }
+    }
+
+    func updateTwentyOneDaysConfirmationPixelFired(_ pixelFired: Bool,
+                                                   forBrokerId brokerId: Int64,
+                                                   profileQueryId: Int64,
+                                                   extractedProfileId: Int64) throws {
+        do {
+            let vault = try self.vault ?? DataBrokerProtectionSecureVaultFactory.makeVault(reporter: secureVaultErrorReporter)
+
+            try vault.updateTwentyOneDaysConfirmationPixelFired(
+                pixelFired,
+                forBrokerId: brokerId,
+                profileQueryId: profileQueryId,
+                extractedProfileId: extractedProfileId
+            )
+        } catch {
+            os_log("Database error: updateTwentyOneDaysConfirmationPixelFired, error: %{public}@", log: .error, error.localizedDescription)
+            pixelHandler.fire(.generalError(error: error, functionOccurredIn: "DataBrokerProtectionDatabase.updateTwentyOneDaysConfirmationPixelFired pixelFired forBrokerId profileQueryId extractedProfileId"))
+            throw error
+        }
+    }
+
     func updateRemovedDate(_ date: Date?, on extractedProfileId: Int64) throws {
         do {
             let vault = try self.vault ?? DataBrokerProtectionSecureVaultFactory.makeVault(reporter: secureVaultErrorReporter)
@@ -288,8 +384,13 @@ final class DataBrokerProtectionDatabase: DataBrokerProtectionRepository {
             try vault.save(brokerId: optOut.brokerId,
                            profileQueryId: optOut.profileQueryId,
                            extractedProfile: extractedProfile,
+                           createdDate: optOut.createdDate,
                            lastRunDate: optOut.lastRunDate,
-                           preferredRunDate: optOut.preferredRunDate)
+                           preferredRunDate: optOut.preferredRunDate,
+                           submittedSuccessfullyDate: optOut.submittedSuccessfullyDate,
+                           sevenDaysConfirmationPixelFired: optOut.sevenDaysConfirmationPixelFired,
+                           fourteenDaysConfirmationPixelFired: optOut.fourteenDaysConfirmationPixelFired,
+                           twentyOneDaysConfirmationPixelFired: optOut.twentyOneDaysConfirmationPixelFired)
         } catch {
             os_log("Database error: saveOptOutOperation, error: %{public}@", log: .error, error.localizedDescription)
             pixelHandler.fire(.generalError(error: error, functionOccurredIn: "DataBrokerProtectionDatabase.saveOptOutOperation optOut extractedProfile"))

LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/Model/BrokerJobData.swift

@@ -70,22 +70,40 @@ struct ScanJobData: BrokerJobData, Sendable {
 struct OptOutJobData: BrokerJobData, Sendable {
     let brokerId: Int64
     let profileQueryId: Int64
+    let createdDate: Date
     let preferredRunDate: Date?
     let historyEvents: [HistoryEvent]
     let lastRunDate: Date?
+
+    // This was added in a later DB migration (V4), so will be nil for older entries submitted before the migration
+    let submittedSuccessfullyDate: Date?
+
     let extractedProfile: ExtractedProfile
+    let sevenDaysConfirmationPixelFired: Bool
+    let fourteenDaysConfirmationPixelFired: Bool
+    let twentyOneDaysConfirmationPixelFired: Bool
 
     init(brokerId: Int64,
          profileQueryId: Int64,
+         createdDate: Date,
          preferredRunDate: Date? = nil,
          historyEvents: [HistoryEvent],
          lastRunDate: Date? = nil,
-         extractedProfile: ExtractedProfile) {
+         submittedSuccessfullyDate: Date? = nil,
+         extractedProfile: ExtractedProfile,
+         sevenDaysConfirmationPixelFired: Bool = false,
+         fourteenDaysConfirmationPixelFired: Bool = false,
+         twentyOneDaysConfirmationPixelFired: Bool = false) {
         self.brokerId = brokerId
         self.profileQueryId = profileQueryId
+        self.createdDate = createdDate
         self.preferredRunDate = preferredRunDate
         self.historyEvents = historyEvents
         self.lastRunDate = lastRunDate
+        self.submittedSuccessfullyDate = submittedSuccessfullyDate
         self.extractedProfile = extractedProfile
+        self.sevenDaysConfirmationPixelFired = sevenDaysConfirmationPixelFired
+        self.fourteenDaysConfirmationPixelFired = fourteenDaysConfirmationPixelFired
+        self.twentyOneDaysConfirmationPixelFired = twentyOneDaysConfirmationPixelFired
     }
 }

LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/Operations/DataBrokerProfileQueryOperationManager.swift

@@ -171,10 +171,15 @@ struct DataBrokerProfileQueryOperationManager: OperationsManager {
                         // we insert the opt-out operation, we do not want to do things separately in case creating an opt-out fails
                         // causing the extracted profile to be orphan.
                         let optOutJobData = OptOutJobData(brokerId: brokerId,
-                                                                      profileQueryId: profileQueryId,
-                                                                      preferredRunDate: preferredRunOperation,
-                                                                      historyEvents: [HistoryEvent](),
-                                                                      extractedProfile: extractedProfile)
+                                                          profileQueryId: profileQueryId,
+                                                          createdDate: Date(),
+                                                          preferredRunDate: preferredRunOperation,
+                                                          historyEvents: [HistoryEvent](),
+                                                          submittedSuccessfullyDate: nil,
+                                                          extractedProfile: extractedProfile,
+                                                          sevenDaysConfirmationPixelFired: false,
+                                                          fourteenDaysConfirmationPixelFired: false,
+                                                          twentyOneDaysConfirmationPixelFired: false)
 
                         try database.saveOptOutJob(optOut: optOutJobData, extractedProfile: extractedProfile)
 

LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/Operations/OperationPreferredDateUpdater.swift

@@ -139,6 +139,15 @@ struct OperationPreferredDateUpdaterUseCase: OperationPreferredDateUpdater {
                                        profileQueryId: profileQueryId,
                                        extractedProfileId: extractedProfileId)
         }
+
+        if let extractedProfileId = extractedProfileId,
+           let optOutJob = optOutJob,
+           let lastEvent = brokerProfileQuery.events.last,
+           lastEvent.type == .optOutRequested && optOutJob.submittedSuccessfullyDate == nil
+        {
+            let submittedSuccessfullyDate = SystemDate().now
+            try database.updateSubmittedSuccessfullyDate(submittedSuccessfullyDate, forBrokerId: brokerId, profileQueryId: profileQueryId, extractedProfileId: extractedProfileId)
+        }
     }
 
     private func returnMostRecentDate(_ date1: Date?, _ date2: Date?) -> Date? {

LocalPackages/DataBrokerProtection/Sources/DataBrokerProtection/Pixels/DataBrokerProtectionPixels.swift

@@ -152,6 +152,14 @@ public enum DataBrokerProtectionPixels {
     case scanningEventNewMatch
     case scanningEventReAppearance
 
+    // Additional opt out metrics
+    case optOutJobAt7DaysConfirmed(dataBroker: String)
+    case optOutJobAt7DaysUnconfirmed(dataBroker: String)
+    case optOutJobAt14DaysConfirmed(dataBroker: String)
+    case optOutJobAt14DaysUnconfirmed(dataBroker: String)
+    case optOutJobAt21DaysConfirmed(dataBroker: String)
+    case optOutJobAt21DaysUnconfirmed(dataBroker: String)
+
     // Web UI - loading errors
     case webUILoadingStarted(environment: String)
     case webUILoadingFailed(errorCategory: String)
@@ -279,6 +287,14 @@ extension DataBrokerProtectionPixels: PixelKitEvent {
         case .webUILoadingSuccess: return "m_mac_dbp_web_ui_loading_success"
         case .webUILoadingFailed: return "m_mac_dbp_web_ui_loading_failed"
 
+            // Additional opt out metrics
+        case .optOutJobAt7DaysConfirmed: return "m_mac_dbp_optoutjob_at-7-days_confirmed"
+        case .optOutJobAt7DaysUnconfirmed: return "m_mac_dbp_optoutjob_at-7-days_unconfirmed"
+        case .optOutJobAt14DaysConfirmed: return "m_mac_dbp_optoutjob_at-14-days_confirmed"
+        case .optOutJobAt14DaysUnconfirmed: return "m_mac_dbp_optoutjob_at-14-days_unconfirmed"
+        case .optOutJobAt21DaysConfirmed: return "m_mac_dbp_optoutjob_at-21-days_confirmed"
+        case .optOutJobAt21DaysUnconfirmed: return "m_mac_dbp_optoutjob_at-21-days_unconfirmed"
+
             // Backend service errors
         case .generateEmailHTTPErrorDaily: return "m_mac_dbp_service_email-generate-http-error"
         case .emptyAccessTokenDaily: return "m_mac_dbp_service_empty-auth-token"
@@ -376,6 +392,13 @@ extension DataBrokerProtectionPixels: PixelKitEvent {
             return [Consts.hadNewMatch: hadNewMatch ? "1" : "0", Consts.hadReAppereance: hadReAppereance ? "1" : "0", Consts.scanCoverage: scanCoverage.description]
         case .weeklyReportRemovals(let removals):
             return [Consts.removals: String(removals)]
+        case .optOutJobAt7DaysConfirmed(let dataBroker),
+                .optOutJobAt7DaysUnconfirmed(let dataBroker),
+                .optOutJobAt14DaysConfirmed(let dataBroker),
+                .optOutJobAt14DaysUnconfirmed(let dataBroker),
+                .optOutJobAt21DaysConfirmed(let dataBroker),
+                .optOutJobAt21DaysUnconfirmed(let dataBroker):
+            return [Consts.dataBrokerParamKey: dataBroker]
         case .webUILoadingStarted(let environment):
             return [Consts.environmentKey: environment]
         case .webUILoadingSuccess(let environment):
@@ -537,6 +560,12 @@ public class DataBrokerProtectionPixelsHandler: EventMapping<DataBrokerProtectio
                     .monthlyActiveUser,
                     .weeklyReportScanning,
                     .weeklyReportRemovals,
+                    .optOutJobAt7DaysConfirmed,
+                    .optOutJobAt7DaysUnconfirmed,
+                    .optOutJobAt14DaysConfirmed,
+                    .optOutJobAt14DaysUnconfirmed,
+                    .optOutJobAt21DaysConfirmed,
+                    .optOutJobAt21DaysUnconfirmed,
                     .scanningEventNewMatch,
                     .scanningEventReAppearance,
                     .webUILoadingFailed,