diff --git a/broker/iam/README.md b/broker/iam/README.md new file mode 100644 index 00000000..58e4e49e --- /dev/null +++ b/broker/iam/README.md @@ -0,0 +1,211 @@ +# IAM Resources + + + +- [IAM Resources](#iam-resources) + - [Setup](#setup) + - [Roles](#roles) + - [Create the roles](#create-the-roles) + - [Update a role](#update-a-role) + - [Policies](#policies) + - [Set the policy on the project](#set-the-policy-on-the-project) + - [Set a policy on a specific resource](#set-a-policy-on-a-specific-resource) + - [Onboard a new developer](#onboard-a-new-developer) + - [Developer instructions](#developer-instructions) + - [Project manager instructions](#project-manager-instructions) + - [Offboard a developer](#offboard-a-developer) + + + +Basic idea: + +We need to connect three things: resource(s), permission(s), and user(s). + +1. Create a Role - this is a collection of permissions and is a registered resource in GCP. +1. Create a Policy - this is a local file that defines a collection of bindings (bindings attach roles to members) +1. Attach the Policy to the Project, or to a specific Resource + +Helpful links: + +- [IAM overview](https://cloud.google.com/iam/docs/overview) +- [IAM predefined roles reference](https://cloud.google.com/iam/docs/understanding-roles#predefined) +- [IAM permissions reference](https://cloud.google.com/iam/docs/permissions-reference) + +## Setup + +Set GCP environment variables and authenticate yourself +For reference, our current projects are: + +- production project: `ardent-cycling-243415` +- testing project: `avid-heading-329016` + +```bash +# Set environment variables +# fill these in with your values; they are used throughout +export GOOGLE_CLOUD_PROJECT= +export GOOGLE_APPLICATION_CREDENTIALS= + +# Authenticate to use gcloud tools in this project +gcloud auth activate-service-account \ + --project="${GOOGLE_CLOUD_PROJECT}" \ + --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" +``` + +## Roles + +### Create the roles + +Here is an example for creating the developer role. + +```bash +role_id=developer +role_yaml=roles/developer.yaml +gcloud iam roles create $role_id --project=$GOOGLE_CLOUD_PROJECT --file=$role_yaml +``` + +### Update a role + +(simpler in python so you don't have to mess with the etag fingerprint) + +```python +# this is an untested example from the following url +# https://cloud.google.com/iam/docs/creating-custom-roles +def edit_role(name, project, title, description, permissions, stage): + """Creates a role.""" + + # pylint: disable=no-member + role = service.projects().roles().patch( + name='projects/' + project + '/roles/' + name, + body={ + 'title': title, + 'description': description, + 'includedPermissions': permissions, + 'stage': stage + }).execute() + + print('Updated role: ' + role['name']) + return role +``` + +## Policies + +We should set the project policy when the project is created. +Afterwards, add or remove bindings to the policy individually (see onboarding section below for an example) rather than using the instructions in this section (which set the policy as a whole). + +### Set the policy on the project + +This binds member-role pairs to every resource in the project. (See below to bind to a specific resource.) + +We must download the current policy, update the file and use it to set a new policy. +Setting a policy overrides the current policy. + +```bash +policy_file="current_policy.yaml" +gcloud projects get-iam-policy $GOOGLE_CLOUD_PROJECT >> $policy_yaml +# update the bindings in the policy_file as needed, then set a new policy +gcloud projects set-iam-policy $GOOGLE_CLOUD_PROJECT $policy_yaml +``` + +### Set a policy on a specific resource + +(simpler in python so you don't have to mess with the etag fingerprint) + +```python +# this is an example from +# https://cloud.google.com/pubsub/docs/access-control#python_2 +from google.cloud import pubsub_v1 + +# TODO(developer): Choose an existing subscription. +# project_id = "your-project-id" +# subscription_id = "your-subscription-id" + +client = pubsub_v1.SubscriberClient() +subscription_path = client.subscription_path(project_id, subscription_id) + +policy = client.get_iam_policy(request={"resource": subscription_path}) + +# Add all users as viewers. +policy.bindings.add(role="roles/pubsub.viewer", members=["domain:google.com"]) + +# Add a group as an editor. +policy.bindings.add(role="roles/editor", members=["group:cloud-logs@google.com"]) + +# Set the policy +policy = client.set_iam_policy( + request={"resource": subscription_path, "policy": policy} +) + +print("IAM policy for subscription {} set: {}".format(subscription_id, policy)) + +client.close() +``` + +## Onboard a new developer + +NOTE: The following instructions are left for reference, but these are in the docs (currently at: docs/source/broker/initial-setup/manager-instructions.rst) and should be kept up-to-date there. + +### Developer instructions + +Complete the instructions in the initial setup tutorial +(docs/source/broker/initial-setup/initial-setup.rst). + +You will need to do this in conjunction with a Pitt-Google project manager so that they can grant you the necessary permissions. + +You will need to provide the manager with both your Google account email address (e.g., Gmail address) and your service account name (which you will choose during setup). + +### Project manager instructions + +Make sure you've authenticated with the GCP project that the developer needs access to (see [Setup](#setup)). + +Note this needs to be done in conjunction with the developer's setup. +The developer needs permissions (a role) bound to their Google account in order to create a service account, and the service account must exist before we can bind a role to it. + +Setup: + +```bash +# fill in the user's Google account email address (e.g., Gmail address): +user_email= + +# fill in the user's service account name and set the email address +# service_account_name= +service_account_email="${service_account_name}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com" + +# choose a role_id (one example is commented out below), and set the role +role_id= +# role_id="developer" +role="projects/${GOOGLE_CLOUD_PROJECT}/roles/${role_id}" +# the above syntax is for a custom role that we've defined in the project +# you can also use a predefined role. see the reference link given above for all options +# role="role/viewer" +``` + +Add two policy bindings; one for the user's Google account (e.g., Gmail address, used for console access, etc.) and one for the user's service account (used to make API calls). + +```bash +gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="user:${user_email}" \ + --role="${role}" + +# the service account needs to exist before +# we can run this command to bind the policy +gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="serviceAccount:${service_account_email}" \ + --role="${role}" + +# required to deploy Cloud Functions. only needs to be granted to the user account. +gcloud iam service-accounts add-iam-policy-binding avid-heading-329016@appspot.gserviceaccount.com --member=user:amp322@pitt.edu --role=roles/iam.serviceAccountUser +``` + +### Offboard a developer + +Follow the setup in [Project manager instructions](#project-manager-instructions), then remove both policy bindings: + +```bash +gcloud projects remove-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="user:${user_email}" \ + --role="${role}" + +gcloud projects remove-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="serviceAccount:${service_account_email}" \ + --role="${role}" +``` diff --git a/broker/iam/policy_examples/README.md b/broker/iam/policy_examples/README.md new file mode 100644 index 00000000..1e15120c --- /dev/null +++ b/broker/iam/policy_examples/README.md @@ -0,0 +1,3 @@ +# Policies + +The files in this directory are example policies. They are not currently set on our projects. diff --git a/broker/iam/policy_examples/production_project.yaml b/broker/iam/policy_examples/production_project.yaml new file mode 100644 index 00000000..8fd81c09 --- /dev/null +++ b/broker/iam/policy_examples/production_project.yaml @@ -0,0 +1,8 @@ +# etag must be obtained from the current policy and filled in below +etag: +bindings: +- role: "projects/ardent-cycling-243415/roles/userPublic" + members: + # allUsers is anyone on the internet, no authentication required. + # Note that access to some GCP services does require authentication. + - user:allUsers diff --git a/broker/iam/policy_examples/testing_project.yaml b/broker/iam/policy_examples/testing_project.yaml new file mode 100644 index 00000000..a4c39479 --- /dev/null +++ b/broker/iam/policy_examples/testing_project.yaml @@ -0,0 +1,8 @@ +# etag must be obtained from the current policy and filled in below +etag: +bindings: +- role: "projects/avid-heading-329016/roles/developerStudent" + members: + # add users or service accounts. + # syntax for service accounts: + # - serviceAccount:@.iam.gserviceaccount.com diff --git a/broker/iam/roles/developer.yaml b/broker/iam/roles/developer.yaml new file mode 100644 index 00000000..ca84e0e4 --- /dev/null +++ b/broker/iam/roles/developer.yaml @@ -0,0 +1,10 @@ +title: "Pitt-Google Developer" +# intended role-id is (set this at deployment): +# "developer" +description: "Role granting permissions for Pitt-Google developers that are not already bundled in a suitable, predefined role." +stage: "ALPHA" +includedPermissions: +- storage.buckets.get +- storage.buckets.list +- storage.objects.list +- storage.objects.get diff --git a/broker/iam/roles/user_public.yaml b/broker/iam/roles/user_public.yaml new file mode 100644 index 00000000..976dfd72 --- /dev/null +++ b/broker/iam/roles/user_public.yaml @@ -0,0 +1,7 @@ +title: "Public User of Pitt-Google Broker" +# intended role-id is (set this at deployment): +# "userPublic" +description: "Role graning permissions necessary for accessing Pitt-Google's public data resources. Intended to be bound to the `allUsers` member in Pitt-Google's production project. (allUsers is anyone on the internet, no authentication required. Note that access to some GCP services does require authentication." +stage: "ALPHA" +includedPermissions: +- pubsub.topics.attachSubscription diff --git a/docs/source/broker/initial-setup.rst b/docs/source/broker/initial-setup.rst new file mode 100644 index 00000000..88716c48 --- /dev/null +++ b/docs/source/broker/initial-setup.rst @@ -0,0 +1,8 @@ +Initial Setup +======================== + +.. toctree:: + :maxdepth: 1 + + initial-setup/initial-setup + initial-setup/manager-instructions diff --git a/docs/source/broker/run-a-broker-instance/initial-setup.rst b/docs/source/broker/initial-setup/initial-setup.rst similarity index 71% rename from docs/source/broker/run-a-broker-instance/initial-setup.rst rename to docs/source/broker/initial-setup/initial-setup.rst index 94e7024b..ce5167b6 100644 --- a/docs/source/broker/run-a-broker-instance/initial-setup.rst +++ b/docs/source/broker/initial-setup/initial-setup.rst @@ -17,6 +17,10 @@ project. Create a GCP Project -------------------- +.. note:: + + You do not need to complete this section if you are a new developer for Pitt-Google broker. You will use our GCP projects; you do not need to create your own. Skip to the Setup Local Environment section. + 1. Create a new Google Cloud Platform (GCP) project. - A. Go to the `Cloud Resource @@ -44,8 +48,13 @@ Setup Local Environment Broker instances *run* 100% in the Google Cloud, as determined by the code in the ``broker`` package. You can *develop* the code and/or -*deploy* an instance to the Cloud from your local machine. Setup your -environment as follows: +*deploy* an instance to the Cloud from your local machine. + +.. note:: + + If you are a new developer for Pitt-Google, You will need to complete this in conjunction with a Pitt-Google manager so they can give you appropriate permissions. + +Setup your environment as follows: 1. **Install Google Cloud SDK command-line tools** using one of the following options. Included tools: gcloud, gsutil, and @@ -61,10 +70,11 @@ may be asked to re-authenticate occasionally in the future. .. code:: bash - PROJECT_ID=my-pgb-project # replace with your GCP Project ID + # fill in the ID of the GCP project you created above (or Pitt-Google's project ID) + PROJECT_ID= - gcloud auth login # follow the instructions to login to GCP - gcloud config set project $PROJECT_ID # set your project ID + gcloud auth login # follow the instructions to login to GCP with a Google account + gcloud config set project $PROJECT_ID # set gcloud to use this project by default 2. **Install Python libraries** for `GCP services `__ and @@ -80,8 +90,8 @@ may be asked to re-authenticate occasionally in the future. conda create -n pgb python=3.7 conda activate pgb - # install the requirements. assumes txt file is in current directory - pip3 install -r requirements.txt + # install the pgb-broker-utils library, which also installs several google.cloud libraries + pip3 install pgb-broker-utils **Note**: On an M1 Mac, first use Conda to install Astropy (``conda install astropy=3.2.1``), then comment the related line out of @@ -94,35 +104,39 @@ the requirements file before doing ``pip install``. .. code:: bash - PROJECT_ID=my-pgb-project # replace with your GCP Project ID - NAME=my-service-account # replace with desired account name - KEY_PATH=local/path/GCP_auth_key.json # replace with desired path (ending in .json) + # choose a service account name (e.g., your name) and fill it in + SA_NAME= + # choose a local path to store your authentication file and fill it in (file name must end with .json) + KEY_PATH= - gcloud iam service-accounts create $NAME - gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$NAME@$PROJECT_ID.iam.gserviceaccount.com" --role="roles/owner" - gcloud iam service-accounts keys create $KEY_PATH --iam-account=$NAME@$PROJECT_ID.iam.gserviceaccount.com + # create the service account + gcloud iam service-accounts create $SA_NAME -4. **Set environment variables** + # If this is a Pitt-Google project, send your service account name (SA_NAME) + # to a project manager to and as them to grant you a "developer" role on the project + # Otherwise, assign a role to your service account. + # This example below assigns a predifined role called "editor" + # gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + # --member="serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \ + # --role="role/editor" -.. code:: bash + # download the authentication file + gcloud iam service-accounts keys create $KEY_PATH --iam-account="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" - PROJECT_ID=my-pgb-project # replace with your GCP Project ID - KEY_PATH=local/path/GCP_auth_key.json # same path as in step 3 +4. **Set environment variables** +.. code:: bash export GOOGLE_CLOUD_PROJECT="$PROJECT_ID" export GOOGLE_APPLICATION_CREDENTIALS="$KEY_PATH" # export CLOUDSDK_COMPUTE_ZONE= -If you are using a Conda environment, you can configure the environment -variables as follows: +If you are using a Conda environment, you can configure it to automatically set these environment +variables when you activate the environment as follows: .. code:: bash - PROJECT_ID=my-pgb-project # replace with your GCP Project ID - KEY_PATH=local/path/for/key/file.json # same path as in step 3 - - # log into the environment and create de/activate files + # log into the environment and create activate and deactivate files conda activate pgb cd $CONDA_PREFIX mkdir -p ./etc/conda/activate.d @@ -130,14 +144,18 @@ variables as follows: touch ./etc/conda/activate.d/env_vars.sh touch ./etc/conda/deactivate.d/env_vars.sh - # add environment variables + # add commands to automatically set these variables when the environment is activated echo "export GOOGLE_CLOUD_PROJECT='$PROJECT_ID'" >> ./etc/conda/activate.d/env_vars.sh echo "export GOOGLE_APPLICATION_CREDENTIALS='$KEY_PATH'" >> ./etc/conda/activate.d/env_vars.sh + + # add commands to automatically unset these variables when the environment is deactivated echo 'unset GOOGLE_CLOUD_PROJECT' >> ./etc/conda/deactivate.d/env_vars.sh echo 'unset GOOGLE_APPLICATION_CREDENTIALS' >> ./etc/conda/deactivate.d/env_vars.sh 5. **Check that your authentication works** by making an API request. - Here we request a list of Cloud Storage buckets (in Python): + The example below requests a list of Cloud Storage buckets (in Python): + +(This will not work until your service account is assigned to a role, per instructions in step 3) .. code:: python diff --git a/docs/source/broker/initial-setup/manager-instructions.rst b/docs/source/broker/initial-setup/manager-instructions.rst new file mode 100644 index 00000000..d5bc788f --- /dev/null +++ b/docs/source/broker/initial-setup/manager-instructions.rst @@ -0,0 +1,94 @@ +Onboard a new developer +======================= + +..contents:: title + :depth: 2 + +Developer instructions +---------------------- + +Complete the instructions in the initial setup tutorial +(ref:`docs/source/broker/initial-setup/initial-setup`). + +You will need to do this in conjunction with a Pitt-Google project manager so that they can grant you the necessary permissions. + +You will need to provide the manager with both your Google account email address (e.g., Gmail address) and your service account name (which you will choose during setup). + +Project manager instructions +---------------------------- + +Note this needs to be done in conjunction with the developer's setup. +The developer needs permissions (a role) bound to their Google account in order to create a service account, and the service account must exist before we can bind a role to it. + +Setup +^^^^^ + +Set GCP environment variables and authenticate yourself to the GCP project that the developer needs access to (probably the testing project). +For reference, our current projects are: + + +* production project: ``ardent-cycling-243415`` +* testing project: ``avid-heading-329016`` + +.. code-block:: bash + + # Set environment variables + # fill these in with your values; they are used throughout + export GOOGLE_CLOUD_PROJECT= + export GOOGLE_APPLICATION_CREDENTIALS= + + # Authenticate to use gcloud tools in this project + gcloud auth activate-service-account \ + --project="${GOOGLE_CLOUD_PROJECT}" \ + --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" + +Set some variables defining the user account(s) and role. + +.. code-block:: bash + + # fill in the user's Google account email address (e.g., Gmail address): + user_email= + + # fill in the user's service account name and set the email address + service_account_name= + service_account_email="${service_account_name}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com" + + # choose a role_id (one eample is commented out below), and set the role + role_id= + # role_id="developerStudent" + role="projects/${GOOGLE_CLOUD_PROJECT}/roles/${role_id}" + # the above syntax is for a custom role that we've defined in the project + # you can also use a predefined role. see the reference link given above for all options + # role="role/viewer" + +Onboard a developer +^^^^^^^^^^^^^^^^^^^ + +Add two policy bindings; one for the user's Google account (e.g., Gmail address, used for console access, etc.) and one for the user's service account (used to make API calls). + +.. code-block:: bash + + gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="user:${user_email}" \ + --role="${role}" + + # the service account needs to exist before + # we can run this command to bind the policy + gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="serviceAccount:${service_account_email}" \ + --role="${role}" + +Offboard a developer +^^^^^^^^^^^^^^^^^^^^ + +Follow the setup instructions above to set variables, then remove both policy bindings: + +.. code-block:: bash + + gcloud projects remove-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="user:${user_email}" \ + --role="${role}" + + gcloud projects remove-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \ + --member="serviceAccount:${service_account_email}" \ + --role="${role}" diff --git a/docs/source/broker/run-a-broker-instance.rst b/docs/source/broker/run-a-broker-instance.rst index 4c6c6f69..10451234 100644 --- a/docs/source/broker/run-a-broker-instance.rst +++ b/docs/source/broker/run-a-broker-instance.rst @@ -5,7 +5,6 @@ Run a Broker Instance :maxdepth: 1 run-a-broker-instance/test-an-instance - run-a-broker-instance/initial-setup run-a-broker-instance/setup-broker run-a-broker-instance/run-broker run-a-broker-instance/delete-broker diff --git a/docs/source/broker/run-a-broker-instance/setup-broker.rst b/docs/source/broker/run-a-broker-instance/setup-broker.rst index 3f3e6159..41f7c82f 100644 --- a/docs/source/broker/run-a-broker-instance/setup-broker.rst +++ b/docs/source/broker/run-a-broker-instance/setup-broker.rst @@ -13,7 +13,7 @@ Setup the Broker Prerequisites ------------- -First, complete the :doc:`initial-setup` to setup your +First, complete the :ref:`docs/source/broker/initial-setup/initial-setup` to setup your Google Cloud project and install the SDKs. Make sure your environment variables are set: diff --git a/docs/source/broker/run-a-broker-instance/test-an-instance.rst b/docs/source/broker/run-a-broker-instance/test-an-instance.rst index 3d0a65da..620732fa 100644 --- a/docs/source/broker/run-a-broker-instance/test-an-instance.rst +++ b/docs/source/broker/run-a-broker-instance/test-an-instance.rst @@ -5,7 +5,7 @@ Run, develop, and test a broker instance. **Prerequisites:** -1. Complete the :doc:`initial-setup` for GCP and your local environment. +1. Complete the :ref:`docs/source/broker/initial-setup/initial-setup` for GCP and your local environment. 2. Create a broker instance by following :doc:`setup-broker`. -------------- diff --git a/docs/source/index.rst b/docs/source/index.rst index 98368cab..af58bdb5 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -34,6 +34,7 @@ The Pitt-Google Broker runs on the `Google Cloud Platform