Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AJ-1602: list-jobs API should aggregate Import Service and cWDS #1308

Merged
merged 6 commits into from
Mar 14, 2024

Conversation

davidangb
Copy link
Contributor

@davidangb davidangb commented Mar 13, 2024

AJ-1602

High-level changes:

  • add the cWDS client library
  • create a new CwdsDAO trait, with Http* and Mock* implementations
  • when listing jobs:
    • call Import Service to list its jobs
    • if cwds.enabled (see https://github.com/broadinstitute/terra-helmfile/pull/5261):
      • call Rawls to translate the workspace namespace/name into a workspace id
      • call cWDS, using the workspace id, to list its jobs
    • else:
      • skip Rawls and cWDS and return an empty list of jobs
    • concat the Import Service and cWDS results
  • unit tests

open questions:

  • do we need more error trapping/retries?
  • do we need cWDS in swatomation?

for a future PR:

  • similar Import Service/cWDS support for the create-job APIs

Have you read CONTRIBUTING.md lately? If not, do that first.

I, the developer opening this PR, do solemnly pinky swear that:

  • I've followed the instructions if I've made any changes to the API, especially if they're breaking changes
  • I've updated the RC_XXX release ticket with any manual steps required to release this change
  • I've updated the FISMA documentation if I've made any security-related changes, including auth, encryption, or auditing

In all cases:

  • Get two thumbsworth of review and PO signoff if necessary
  • Verify all tests go green
  • Squash and merge; you can delete your branch after this unless it's for a hotfix. In that case, don't delete it!
  • Test this change deployed correctly and works on dev environment after deployment

Copy link

codecov bot commented Mar 13, 2024

Codecov Report

Attention: Patch coverage is 57.89474% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 69.19%. Comparing base (c083d6b) to head (0071fba).

Files Patch % Lines
...titute/dsde/firecloud/dataaccess/HttpCwdsDAO.scala 57.69% 11 Missing ⚠️
...roadinstitute/dsde/firecloud/FireCloudConfig.scala 0.00% 3 Missing ⚠️
...scala/org/broadinstitute/dsde/firecloud/Boot.scala 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1308      +/-   ##
===========================================
- Coverage    69.31%   69.19%   -0.13%     
===========================================
  Files           97       98       +1     
  Lines         3406     3441      +35     
  Branches       369      377       +8     
===========================================
+ Hits          2361     2381      +20     
- Misses        1045     1060      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

val importServiceResponse = List(
ImportServiceListResponse("jobId1", "status1", "filetype1", None),
ImportServiceListResponse("jobId2", "status2", "filetype2", None)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It weirds me out that these two list have identical jobIds. Not clear how/if that will impact the test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy/paste error, fixed!

}

private def toImportServiceStatus(cwdsStatus: GenericJob.StatusEnum): String = {
STATUS_TRANSLATION.getOrElse(cwdsStatus, "blah")
Copy link
Contributor

@jladieu jladieu Mar 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this could throw an error if there's no translation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a code comment; I would prefer to not fail, but now this returns "Unknown" instead of the draft-for-early-review "blah"

// get jobs from cWDS
cwdsJobs = cwdsDAO.listJobsV1(workspace.workspace.workspaceId, runningOnly)(userInfo)
} yield {
// merge Import Service and cWDS results
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not for this PR, but maybe for the metrics theme: this might be a good spot to add some metric instrumentation to understand migration status:

  • measure latency of each dispatch for comparison
  • measure the # of jobs returned from each for comparison

Copy link
Contributor

@jladieu jladieu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, all comments are optional/future.

@davidangb davidangb marked this pull request as ready for review March 14, 2024 15:55
@davidangb davidangb merged commit a1b4e65 into develop Mar 14, 2024
13 of 15 checks passed
@davidangb davidangb deleted the da_AJ-1602_cwdsJobs branch March 14, 2024 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants