[DDO-3719] Role Propagation #minor #575

jack-r-warren · 2024-06-05T19:34:40Z

This PR implements role propagation for [dev|qa|prod]FirecloudGroup and [dev|prod]AzureGroup. It's been tested for the dev variants in sherlock-dev. This code will remain dark in production; it's disabled by default.

Note

You may skip the overview below. It's written in case some people find that easier to digest than spoken explanations or documentation in-line.

Overview

This PR adds three new concepts to Sherlock. These concepts are entirely internal and don't show up in the API at all. They are "propagator", "propagationEngine", and "intermediaryUser." We'll go in reverse order.

An "intermediaryUser" is "how is a user represented in the remote cloud provider." It has an "identifier" that identifies the user and "fields" for anything else. Here's some examples:

For a dev Firecloud group, the identifier would be the user's test.firecloud.org email address. There's no fields.
For a dev Azure group, the identifier would be the UUID of the user's Azure account correlating to their test.firecloud.org email address. There's no fields.
For a dev Firecloud account, the identifier would be the user's test.firecloud.org email address. Fields would include the user's first and last name, their "recovery" (broadinstitute.org) email, and whether their account is suspended or not.

An identifier is how we find a user, while the fields are just other information that Sherlock marionettes once we've found them.

A "propagationEngine" is a set of five primitives for combining some specific intermediaryUser with a specific cloud provider. It teaches Sherlock how to work with the cloud provider in an encapsulated way. The primitives are:

Who currently has the permission in the cloud provider
Who should have the permission
Add someone
Update someone
Remove someone

The idea -- and this is holding true so far -- is that these primitives just delegate to client libraries ("list users in group" for Google Groups, etc.). The idea of intermediaryUser means that we don't need to worry about a bunch of conversion logic or API calls. The Azure group engine "speaks" in UUIDs because that's just what's easiest for it.

A "propagator" combines a propagationEngine with configuration and points it at a specific field of a role. We have just a plain "Google Workspace Groups engine", but then we have a specific "dev Firecloud Group propagator" that looks at the dev Firecloud group field on a role. If we wanted to point Sherlock at prod Firecloud, we'd add a new propagator with another instance of the same engine.

Still To-Do

Slack notifications and logging

^ I'm pushing this to another PR, this one is big enough.

Testing

I think the testing that can be automated within Sherlock is all there. We could go to extensive effort to test the HTTP and gRPC behavior of the client libraries we're calling, but after talking to Mike I don't think that's worth it at this moment.

This has been fully deployed to dev and is confirmed to be working there.

Risk

Low.

…stitute/sherlock into DDO-3719-propagation

…-propagation

github-actions · 2024-06-05T19:35:42Z

What's Changed

`POST` /api/role-assignments/v3/{role-selector}/{user-selector}

`DELETE` /api/role-assignments/v3/{role-selector}/{user-selector}

`PATCH` /api/role-assignments/v3/{role-selector}/{user-selector}

`GET` /api/roles/v3

Parameters:

Added: grantsProdAzureGroup in query

Added: grantsProdFirecloudGroup in query

Added: grantsQaFirecloudGroup in query

Return Type:

Changed response : 200 OK

OK

Changed content type : application/json

Changed items (object):
- Added property grantsProdAzureGroup (string)
- Added property grantsProdFirecloudGroup (string)
- Added property grantsQaFirecloudGroup (string)

`POST` /api/roles/v3

Request:

Changed content type : application/json

Added property grantsProdAzureGroup (string)
Added property grantsProdFirecloudGroup (string)
Added property grantsQaFirecloudGroup (string)

Return Type:

Changed response : 201 Created

Created

Changed content type : application/json
- Added property grantsProdAzureGroup (string)
- Added property grantsProdFirecloudGroup (string)
- Added property grantsQaFirecloudGroup (string)

`GET` /api/roles/v3/{selector}

Return Type:

Changed response : 200 OK

OK

Changed content type : application/json
- Added property grantsProdAzureGroup (string)
- Added property grantsProdFirecloudGroup (string)
- Added property grantsQaFirecloudGroup (string)

`DELETE` /api/roles/v3/{selector}

Return Type:

Changed response : 200 OK

OK

Changed content type : application/json
- Added property grantsProdAzureGroup (string)
- Added property grantsProdFirecloudGroup (string)
- Added property grantsQaFirecloudGroup (string)

`PATCH` /api/roles/v3/{selector}

Request:

Changed content type : application/json

Added property grantsProdAzureGroup (string)
Added property grantsProdFirecloudGroup (string)
Added property grantsQaFirecloudGroup (string)

Return Type:

Changed response : 200 OK

OK

Changed content type : application/json
- Added property grantsProdAzureGroup (string)
- Added property grantsProdFirecloudGroup (string)
- Added property grantsQaFirecloudGroup (string)

github-actions · 2024-06-05T19:40:36Z

Published image from f4e3da7 (merge ad72084):

us-central1-docker.pkg.dev/dsp-artifact-registry/sherlock/sherlock:v1.4.11-ad72084
us-central1-docker.pkg.dev/dsp-devops-super-prod/sherlock/sherlock:v1.4.11-ad72084

codecov · 2024-06-05T19:45:16Z

Codecov Report

Attention: Patch coverage is 51.78082% with 176 lines in your changes missing coverage. Please review.

Project coverage is 77.53%. Comparing base (c6fbda9) to head (f4e3da7).
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #575      +/-   ##
==========================================
- Coverage   78.57%   77.53%   -1.05%     
==========================================
  Files         191      202      +11     
  Lines        9102     9459     +357     
==========================================
+ Hits         7152     7334     +182     
- Misses       1322     1483     +161     
- Partials      628      642      +14

Files	Coverage Δ
go-shared/pkg/utils/substitute_suffix.go	`100.00% <100.00%> (ø)`
...nternal/api/sherlock/role_assignments_v3_create.go	`58.53% <100.00%> (+1.03%)`	⬆️
...nternal/api/sherlock/role_assignments_v3_delete.go	`65.62% <100.00%> (+1.10%)`	⬆️
.../internal/api/sherlock/role_assignments_v3_edit.go	`70.27% <100.00%> (+0.82%)`	⬆️
sherlock/internal/api/sherlock/roles_v3.go	`94.28% <100.00%> (+0.53%)`	⬆️
sherlock/internal/api/sherlock/roles_v3_create.go	`70.58% <100.00%> (+1.83%)`	⬆️
sherlock/internal/api/sherlock/roles_v3_delete.go	`87.50% <ø> (ø)`
sherlock/internal/api/sherlock/roles_v3_edit.go	`90.90% <100.00%> (+0.43%)`	⬆️
sherlock/internal/models/role_assignment.go	`73.25% <100.00%> (+0.63%)`	⬆️
sherlock/internal/models/test_data.go	`98.19% <100.00%> (+<0.01%)`	⬆️
... and 13 more

sonarqubecloud · 2024-06-11T14:33:06Z

Quality Gate passed

Issues
52 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
5.7% Duplication on New Code

See analysis details on SonarCloud

jack-r-warren added 30 commits May 21, 2024 16:45

refactor all packages

0eb6295

rename to middleware

34159d1

Merge branch 'main' of github.com:broadinstitute/sherlock into working

0879a6d

imports

fb2dbff

imports

76fea03

imports

f1e8fbf

remove dead code

26f5605

rename

695cc83

mockery config

523f83f

whoops

e7a9c56

whoops electric boogalo

abf5ddb

utils helper function

681c03b

advisory locks enum

43c0e65

add propagated_at

33f9d38

store azure group in uuid

3fabf6f

add assignment.IsActive

d83b505

add role methods

319e512

add intermediary_user

2547a88

add intermediary_user_mocks

913ae1f

fix mockery config

bc2da09

Merge branch 'DDO-3719-propagation-refactoring' of github.com:broadin…

97540fa

…stitute/sherlock into DDO-3719-propagation

propagation_engine

bd8a68c

propagation_engine_mocks

6e3b1d5

google workspace engine

a403a74

azure group engine

69c83d5

propagation_engine readme

e2924d1

propagator

1ac1de8

role propagation and mocks

9c8718b

go mod

121321d

test configs

7bfb759

jack-r-warren added 4 commits June 5, 2024 15:28

initalize propagators

023164c

run propagators on-demand

42f0f99

Merge branch 'main' of github.com:broadinstitute/sherlock into DDO-3719…

5e6fed6

…-propagation

Merge branch 'main' of github.com:broadinstitute/sherlock into DDO-3719…

4a975ce

…-propagation

jack-r-warren added 15 commits June 5, 2024 16:04

avoid linter complaints

ec01674

linter eee

255e7ff

enabled -> enable

13a6c1f

fix tests

74a16bc

actually log

e2e2864

fix tests

b4d789b

change tenant id

db6ced4

more config

de18ff9

use mail

4758cfd

handle suffixes

04851a6

fixes

32a0754

whoops

5bf89cb

explicit scope

2db9232

back to nil scopes

5dc9a3c

more fields

f4e3da7

jack-r-warren marked this pull request as ready for review June 11, 2024 14:37

jack-r-warren requested a review from a team as a code owner June 11, 2024 14:37

em-may approved these changes Jun 11, 2024

View reviewed changes

katiewelch approved these changes Jun 11, 2024

View reviewed changes

jack-r-warren changed the title ~~[DDO-3719] Role Propagation~~ [DDO-3719] Role Propagation #minor Jun 11, 2024

jack-r-warren merged commit 1ed3827 into main Jun 11, 2024
21 checks passed

jack-r-warren deleted the DDO-3719-propagation branch June 11, 2024 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DDO-3719] Role Propagation #minor #575

[DDO-3719] Role Propagation #minor #575

jack-r-warren commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 5, 2024 •

edited

Loading

sonarqubecloud bot commented Jun 11, 2024

[DDO-3719] Role Propagation #minor #575

[DDO-3719] Role Propagation #minor #575

Conversation

jack-r-warren commented Jun 5, 2024 • edited Loading

Overview

Still To-Do

Testing

Risk

github-actions bot commented Jun 5, 2024 • edited Loading

What's Changed

POST /api/role-assignments/v3/{role-selector}/{user-selector}

DELETE /api/role-assignments/v3/{role-selector}/{user-selector}

PATCH /api/role-assignments/v3/{role-selector}/{user-selector}

GET /api/roles/v3

Parameters:

Return Type:

POST /api/roles/v3

Request:

Return Type:

GET /api/roles/v3/{selector}

Return Type:

DELETE /api/roles/v3/{selector}

Return Type:

PATCH /api/roles/v3/{selector}

Request:

Return Type:

github-actions bot commented Jun 5, 2024 • edited Loading

codecov bot commented Jun 5, 2024 • edited Loading

Codecov Report

sonarqubecloud bot commented Jun 11, 2024

Quality Gate passed

jack-r-warren commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 5, 2024 •

edited

Loading

`POST` /api/role-assignments/v3/{role-selector}/{user-selector}

`DELETE` /api/role-assignments/v3/{role-selector}/{user-selector}

`PATCH` /api/role-assignments/v3/{role-selector}/{user-selector}

`GET` /api/roles/v3

`POST` /api/roles/v3

`GET` /api/roles/v3/{selector}

`DELETE` /api/roles/v3/{selector}

`PATCH` /api/roles/v3/{selector}

github-actions bot commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 5, 2024 •

edited

Loading