Skip to content

Latest commit

 

History

History
446 lines (342 loc) · 39.5 KB

README.md

File metadata and controls

446 lines (342 loc) · 39.5 KB
Raw

Table of Contents

Overview

A command line interface that allows you to define your Data Map in code and sync that configuration back to https://app.transcend.io.

Installation

Step 1) Install NPM, Node, and Yarn

This package is distributed through npm and github package registries and assumes an installation of npm and node. You will also want to install yarn.

Step 2) Install @transcend-io/vz-cli and @transcend-io/cli

If your codebase is typescript or javascript based, you can add this package as a dev dependency:

# install the vz cli
yarn add -D @transcend-io/vz-cli

# install the transcend cli as well
yarn add -D @transcend-io/cli

Step 3) Create a .env file to store secrets and common environment configuration

Create a ./.env file with variables like this:

TRANSCEND_API_KEY=FILL_ME
TRANSCEND_API_URL=https://api.us.transcend.io

The TRANSCEND_API_KEY is a secret, and this should not be committed to git or pasted into your terminal history. You should add your ./.env to the .gitignore if you are running these commands from a git repository. If you do commit an API key or paste it in the terminal, the API key can be cycled in the Transcend Dashboard.

Step 4) Run commands

# load in .env file
source .env

# vz cli commands available within @transcend-io/vz-cli
yarn vz-combine-legal-csv-data-flows \
   --legalCsv=/Users/test/Desktop/legal.csv \
   --dataFlowYml=/Users/test/Desktop/transcend.yml \
   --output=/Users/test/Desktop/transcend-output.yml
yarn vz-transcend-from-parent-for-children --file=./transcend.yml

# transcend cli commands available within @transcend-io/cli
yarn tr-pull --auth=$TRANSCEND_API_KEY --resources=consentManager,dataFlows,cookies
yarn tr-push --auth=$TRANSCEND_API_KEY
yarn tr-update-consent-manager --auth=$TRANSCEND_API_KEY
yarn tr-pull-consent-metrics --auth=$TRANSCEND_API_KEY --start=01/01/2023
yarn tr-consent-managers-to-business-entities --consentManagerYmlFolder=./working/consent-managers/ --output=./custom.yml
yarn tr-upload-data-flows-from-csv --auth=$TRANSCEND_API_KEY --file=./approved-flows.csv --trackerStatus=LIVE
yarn tr-upload-cookies-from-csv --auth=$TRANSCEND_API_KEY --file=./approved-flows.csv --trackerStatus=LIVE
yarn tr-generate-api-keys --auth=$TRANSCEND_API_KEY [email protected] --password=$TRANSCEND_PASSWORD \
   --scopes="View Email Templates,View Data Map" --apiKeyTitle="CLI Usage Cross Instance Sync" -file=./working/auth.json
yarn tr-build-xdi-sync-endpoint --auth=$TRANSCEND_API_KEY --xdiLocation=https://cdn.your-site.com/xdi.js
yarn tr-derive-data-silos-from-data-flows --auth=$TRANSCEND_API_KEY --dataFlowsYmlFolder=./working/data-flows/ \
   --dataSilosYmlFolder=./working/data-silos/ --ignoreYmls="0 - Data Mapping.yml"
yarn tr-derive-data-silos-from-data-flows-cross-instance --auth=$TRANSCEND_API_KEY
   --dataFlowsYmlFolder=./working/data-flows/ --output=./transcend.yml

The cli-commands default to using the EU Transcend backend. To use these commands with the US backend, you will need to use the flag You can also set the environment variable TRANSCEND_API_URL=https://api.us.transcend.io

Transcend CLI

The Transcend CLI comes with documentation on each of the standard cli commands. This repository holds cli commands that are custom to this implementation, such as custom CSV file formats.

Custom Commands

vz-combine-legal-csv-data-flows

This command allows for combining information in a CSV from a legal team into a CSV of data flow data

Authentication

No authentication is required to run this cli command, it comes CSV files that are expected to be on disk.

Arguments

Argument Description Type Default Required
legalCsv Custom Legal CSV file. File format defined by LegalTrackerCsvCodec string - file-path ./legalMaster.csv false
dataFlowYml Export of data flows from the Transcend dashboard. File format defined by TranscendInput string - file-path ./transcend.yml false
output Output file format that can be re-imported into Transcend dashboard. File format defined by TranscendInput string - file-path ./transcend.yml false

Note: You trackerStatus can be specified on a per data flow basis by adding a column named "Status" to the CSV. The values should be of type ConsentTrackerStatus - which is LIVE or NEEDS_REVIEW.

Usage

Combine two files on disk

yarn vz-combine-legal-csv-data-flows \
   --legalCsv=/Users/test/Desktop/legal.csv \
   --dataFlowYml=/Users/test/Desktop/transcend.yml \
   --output=/Users/test/Desktop/transcend-output.yml

vz-transform-from-parent-for-children

Remove data from the 0 - Data Mapping transcend.yml output that should not be synced to the other child Transcend instances.

Authentication

No authentication is required to run this cli command, it operates on a YML file on disk

Arguments

Argument Description Type Default Required
file Path to transcend.yml file string - file-path ./transcend.yml false

Usage

yarn vz-transcend-from-parent-for-children --file=./transcend.yml

Useful Commands

A) Combine Legal CSV with Transcend Data Flows

Combine legal's categorizations of vendors, combine those categorizations with what was in Transcend, and push those categorizations back into Transcend.

Step 1) Run CLI commands:

export TRANSCEND_API_URL=https://api.us.transcend.io
export LEGAL_FILE=/Users/test/Desktop/legal.csv
export TRANSCEND_DATA_FLOWS_FILE=/Users/test/Desktop/data-flows.yml
export COMBINED_TRANSCEND_DATA_FLOWS_FILE=/Users/test/Desktop/data-flows-combined.yml
export TRANSCEND_API_KEY=SECRET_FILL_ME
yarn tr-pull --auth=$TRANSCEND_API_KEY --file=$TRANSCEND_DATA_FLOWS_FILE --resources=dataFlows -trackerStatuses=NEEDS_REVIEW,LIVE
yarn vz-combine-legal-csv-data-flows --legalCsv=$LEGAL_FILE --dataFlowYml=$TRANSCEND_DATA_FLOWS_FILE --output=$COMBINED_TRANSCEND_DATA_FLOWS_FILE
yarn tr-push --auth=$TRANSCEND_API_KEY --file=$COMBINED_TRANSCEND_DATA_FLOWS_FILE

Required Environment Variables

Argument Description Type Is Secret
LEGAL_FILE Custom Legal CSV file. File format defined by LegalTrackerCsvCodec string - file-path false
TRANSCEND_DATA_FLOWS_FILE Export of data flows from the Transcend dashboard. File format defined by DataFlowCsvInput string - file-path false
COMBINED_TRANSCEND_DATA_FLOWS_FILE Output file format that can be re-imported into Transcend dashboard. File format defined by DataFlowCsvInput string - file-path false
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEY Transcend API key with scopes Manage Cookies string - api-key true

B) Generate API keys to Synchronize Cross-Account Data

This command will generate a JSON file (at ./working/api-keys.json) containing a list of API Keys for each Transcend instance. The API keys have the minimal set of permissions required to synchronize data across all Transcend instances for your account. Each API key will have the same title. These API keys will be available under Infrastructure -> API Keys.

export TRANSCEND_API_URL=https://api.us.transcend.io
export [email protected]
export TRANSCEND_PASSWORD=SECRET_FILL_ME
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_PARENT_ORGANIZATION_ID=53f5008e-163a-4208-bd0a-aa5ce2208185
yarn tr-generate-api-keys --email=$TRANSCEND_EMAIL --password=$TRANSCEND_PASSWORD --scopes=""Manage Consent Manager Developer Settings,Manage Data Map,Connect Data Silos,Manage Data Subject Request Settings,Manage Global Attributes,Manage Request Identity Verification,View API Keys,View Data Subject Request Settings,View Email Templates,View Identity Verification Settings,Manage Data Inventory,Manage Data Flows" --apiKeyTitle="[cli][$TRANSCEND_EMAIL] Cross Instance Sync" --file=$TRANSCEND_API_KEYS_PATH --parentOrganizationId=$TRANSCEND_PARENT_ORGANIZATION_ID

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEYS_PATH The path to a JSON file where API keys will be written to. This folder should be git-ignored and secured as the contents of this file are a secret. string - file-path false - file contents are a secret
TRANSCEND_PARENT_ORGANIZATION_ID The ID of the parent organization that has child organizations that need API keys generated for. string - uuid false
TRANSCEND_EMAIL Your email address that you use to log into Transcend string - email false
TRANSCEND_PASSWORD The password you use to log into Transcend string - password true

C) Delete API Keys Cross-Account

If you need to delete the API keys generated by the above command, this command can be run to delete API keys across all instances that have a specific title. If you decide to run any cli commands on a CI, you can use these commands to establish a process to cycle API keys.

export TRANSCEND_API_URL=https://api.us.transcend.io
export [email protected]
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_PARENT_ORGANIZATION_ID=53f5008e-163a-4208-bd0a-aa5ce2208185
export TRANSCEND_PASSWORD=SECRET_FILL_ME
yarn tr-generate-api-keys --email=$TRANSCEND_EMAIL --password=$TRANSCEND_PASSWORD --scopes="View API Keys" --apiKeyTitle="[cli][$TRANSCEND_EMAIL] Cross Instance Sync" --file=$TRANSCEND_API_KEYS_PATH--createNewApiKey=false --parentOrganizationId=$TRANSCEND_PARENT_ORGANIZATION_ID

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_EMAIL Your email address that you use to log into Transcend string - email false
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)
TRANSCEND_PARENT_ORGANIZATION_ID The ID of the parent organization that has child organizations that need API keys generated for. string - uuid false
TRANSCEND_PASSWORD The password you use to log into Transcend string - password true

D) Sync Shared Configuration from 0 - Data Mapping Across All Accounts

There are some consistent settings that we will apply across all Transcend instances. The set of resource settings that can be pulled can be found here. The resources listed in the command below can be pulled and synced into other instances to enforce a consistent configuration across the privacy request in order to:

  • attributes - any custom table columns are defined across the data flows, cookies, privacy requests and data inventory tables
  • identifiers - the detected identifiers used to querying users for data deletion
  • consentManager - developer settings for the consent manager

A post-processing step is done to remove some data that should not be synced, before that data is pushed back up into other Transcend instances

export TRANSCEND_API_URL=https://api.us.transcend.io
export TRANSCEND_API_KEY=SECRET_FILL_ME
export TRANSCEND_YAML=./transcend.yml
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
yarn tr-pull --auth=$TRANSCEND_API_KEY --resources=attributes,identifiers,consentManager --file=$TRANSCEND_YAML
yarn vz-transform-from-parent-for-children --file=$TRANSCEND_YAML
yarn tr-push --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_YAML

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_YAML Transcend yml file format, defined by Transcend Input string - file-path false
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEY Transcend API key with scopes Manage Data Flows string - api-key true
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)

E) Pull a transcend.yml for Each Instance

If you need to pull the configuration files for all associated Transcend instances, you can do so using the following command:

export TRANSCEND_API_URL=https://api.us.transcend.io
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_YAML_FOLDER=./working/output/
yarn tr-pull --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_YAML_FOLDER --resources=consentManager,dataFlows,cookies

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_YAML_FOLDER A folder where the transcend.yml files will be written. Each file will be named <instance name>,.yml string - file-path false
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)

F) Combining Configurations Cross-Instances into the 0 - Data Mapping -> Data Inventory -> Business Entities table

Take the output of each Transcend instance and combine them into a single YML file to sync into the 0 - Data Mapping Business Entities table in the Transcend dashboard. This view allows for a summarization of various information across all Transcend instances.

export TRANSCEND_API_URL=https://api.us.transcend.io
export COMBINED_TRANSCEND_BUSINESS_ENTITIES=/Users/test/Desktop/business-entities-combined.yml
export TRANSCEND_API_KEY=SECRET_FILL_ME
export TRANSCEND_YAML_FOLDER=./working/output/
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
rm -rf $TRANSCEND_YAML_FOLDER
yarn tr-pull --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_YAML_FOLDER --resources=consentManager
yarn tr-consent-managers-to-business-entities --transcendYmlFolder=$TRANSCEND_YAML_FOLDER --output=$COMBINED_TRANSCEND_BUSINESS_ENTITIES
yarn tr-push --auth=$TRANSCEND_API_KEY --file=$COMBINED_TRANSCEND_BUSINESS_ENTITIES

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_YAML_FOLDER A folder where the transcend.yml files will be written. Each file will be named <instance name>,.yml string - file-path false
COMBINED_TRANSCEND_DATA_FLOWS_FILE Output file format that can be re-imported into Transcend dashboard. File format defined by DataFlowCsvInput string - file-path false
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEY Transcend API key with scopes Manage Data Flows string - api-key true

G) Pull Down Approved Data Flows to YAML Cross-Account

Pull down list of all data flows that are approved

export TRANSCEND_API_URL=https://api.us.transcend.io
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_YAML_FOLDER=./working/output/
yarn tr-pull --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_YAML_FOLDER --resources=dataFlows,cookies --trackerStatuses=LIVE

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_YAML_FOLDER A folder where the transcend.yml files will be written. Each file will be named <instance name>,.yml string - file-path false
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)

H) Generate Cross-Instance List of Ad Tech & Site Tech

Create a single export of ad tech and site tech across all Transcend instances.

  1. Pull down the data flows and cookies across all instances
  2. Determine the set of Data Silos by looking at the services tied to each data flow and cookie
  3. Produce a single transcend.yml file with the full set of all Data Silos found from each data flow and cookie
  4. Load the transcend.yml into the 0 - Data Mapping Transcend instance.
export TRANSCEND_API_URL=https://api.us.transcend.io
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_DATA_FLOWS_YAML_FOLDER=./working/data-flows/
export TRANSCEND_DATA_SILOS_YAML_FILE=./working/data-silos.yml
export TRANSCEND_IGNORE_YMLS="0 - Data Mapping.yml"
export TRANSCEND_API_KEY=SECRET_FILL_ME
yarn tr-pull --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_DATA_FLOWS_YAML_FOLDER --resources=dataFlows --trackerStatuses=LIVE
yarn tr-derive-data-silos-from-data-flows-cross-instance --auth=$TRANSCEND_API_KEY --dataFlowsYmlFolder=$TRANSCEND_DATA_FLOWS_YAML_FOLDER --output=$TRANSCEND_DATA_SILOS_YAML_FILE --ignoreYmls=$TRANSCEND_IGNORE_YMLS
yarn tr-push --auth=$TRANSCEND_API_KEY --file=$TRANSCEND_DATA_SILOS_YAML_FOLDER

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)
TRANSCEND_DATA_FLOWS_YAML_FOLDER The folder to write the set of data flow configurations string - folder-path false
TRANSCEND_DATA_SILOS_YAML_FILE The file to write the set of data silo configurations to string - file-path false
TRANSCEND_IGNORE_YMLS The yml files that should not be synced string[] false
TRANSCEND_API_KEY Transcend API key with Manage Data Map permission for the instance to upload the TRANSCEND_DATA_SILOS_YAML_FILE file to string - api-key true

I) Generate Per-Instance List of Ad Tech Data Silos

  1. Pull down the data flows and cookies across all instances
  2. Determine the set of Data Silos for that instance by looking at the services tied to each data flow and cookie
  3. Produce a new transcend.yml file containing the full set of all Data Silos found from in each instance
  4. Load each transcend.yml into each respective Transcend instance
export TRANSCEND_API_URL=https://api.us.transcend.io
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_DATA_FLOWS_YAML_FOLDER=./working/data-flows/
export TRANSCEND_DATA_SILOS_YAML_FOLDER=./working/data-silos/
export TRANSCEND_IGNORE_YMLS="0 - Data Mapping.yml"
export TRANSCEND_API_KEY=SECRET_FILL_ME
yarn tr-pull --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_DATA_FLOWS_YAML_FOLDER --resources=dataFlows --trackerStatuses=LIVE
yarn tr-derive-data-silos-from-data-flows --auth=$TRANSCEND_API_KEY --dataFlowsYmlFolder=$TRANSCEND_DATA_FLOWS_YAML_FOLDER --dataSilosYmlFolder=$TRANSCEND_DATA_SILOS_YAML_FOLDER --ignoreYmls=$TRANSCEND_IGNORE_YMLS
yarn tr-push --auth=$TRANSCEND_API_KEYS_PATH --file=$TRANSCEND_DATA_SILOS_YAML_FOLDER

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)
TRANSCEND_DATA_FLOWS_YAML_FOLDER The folder to write the set of data flow configurations string - folder-path false
TRANSCEND_DATA_SILOS_YAML_FOLDER The folder to write the set of data silo configurations string - folder-path false
TRANSCEND_IGNORE_YMLS The yml files that should not be synced string[] false
TRANSCEND_API_KEY Any Transcend API key across any instance. No scope required string - api-key true

J) Update the Consent Manager to Latest Cross-Instance

Use this command to update each consent manager to the latest version, and deploy all active changes. Please be careful using this command, as it will deploy any outstanding changes since the last deploy.

export TRANSCEND_API_URL=https://api.us.transcend.io
export [email protected]
export TRANSCEND_PASSWORD=SECRET_FILL_ME
export TRANSCEND_API_KEYS_PATH=./working/api-keys.json
export TRANSCEND_PARENT_ORGANIZATION_ID=53f5008e-163a-4208-bd0a-aa5ce2208185
yarn tr-generate-api-keys  --email=$TRANSCEND_EMAIL --password=$TRANSCEND_PASSWORD --scopes="Manage Consent Manager" --apiKeyTitle="[cli][$TRANSCEND_EMAIL] Cross Instance Consent Deploy" --file=$TRANSCEND_API_KEYS_PATH --parentOrganizationId=$TRANSCEND_PARENT_ORGANIZATION_ID
yarn tr-update-consent-manager  --auth=$TRANSCEND_API_KEYS_PATH --deploy=true
yarn tr-generate-api-keys  --email=$TRANSCEND_EMAIL --password=$TRANSCEND_PASSWORD --scopes="Manage Consent Manager" --apiKeyTitle="[cli][$TRANSCEND_EMAIL] Cross Instance Consent Deploy" --file=$TRANSCEND_API_KEYS_PATH --createNewApiKey=false --parentOrganizationId=$TRANSCEND_PARENT_ORGANIZATION_ID

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_EMAIL Your email address that you use to log into Transcend string - email false
TRANSCEND_PASSWORD The password you use to log into Transcend string - password true
TRANSCEND_API_KEYS_PATH Path to the JSON file holding the API keys for each instance. This file can be created in section "Generate API keys to Synchronize Cross-Account Data" string - file-path false - file contents are secret (should be git-ignored)
TRANSCEND_PARENT_ORGANIZATION_ID The ID of the parent organization that has child organizations that need API keys generated for. string - uuid false

H) Download Transcend Cookies and Re-Upload After Edits

Step 1) Download the CSV of cookies that you want to edit from the Admin Dashboard under Consent Manager -> Cookies. You can download data flows from both the "Triage" and "Approved" tabs.

Step 2) Edit the file

Step 3) Run CLI commands:

export TRANSCEND_API_URL=https://api.us.transcend.io
export TRANSCEND_COOKIES_FILE=/Users/test/Desktop/cookies.csv
export COMBINED_TRANSCEND_COOKIES_FILE=/Users/test/Desktop/cookies.csv
export TRANSCEND_API_KEY=SECRET_FILL_ME
yarn tr-upload-cookies-from-csv --auth=$TRANSCEND_API_KEY --file=$COMBINED_TRANSCEND_COOKIES_FILE --trackerStatus=LIVE

Required Environment Variables

Argument Description Type Is Secret
TRANSCEND_COOKIES_FILE Export of cookies from the Transcend dashboard. File format defined byCookieCsvInput string - file-path false
COMBINED_TRANSCEND_COOKIES_FILE Output file format that can be re-imported into Transcend dashboard. File format defined by CookieCsvInput string - file-path false
TRANSCEND_API_URL Transcend backend URL string - url false
TRANSCEND_API_KEY Transcend API key with scopes Manage Cookies string - api-key true