Feat: Updates GET workflows to v3

[dapineyro]

Overview

This PR updates GET workflows endpoint from v1 to v3.

Jira

https://lifebit.atlassian.net/browse/LIF-1346

Changes

• Refactors Cloudos.get_workflow_list method:
  • Updates to /api/v3/workflows endpoint
  • That's a paginated endpoint, with 10 entries per page by default and 1000 as maximum. The function is refactored to be able to retrieve all the workflows from the paginated responses. The function is parametrised to control many aspects of the call, but cloudos-cli always use it to retrieve all the workflows
  • The implementation was done using a for loop instead of recursively for efficiency
• Refactors Cloudos.get_curated_workflow_listmethod:
  • The new v3 version of the endpoint does not have the previous getByType. Now it uses search and we need to search the groups property.
  • The function still uses recursive implementation, as the expected number of curated pipelines is small
• Refactors Cloudos.process_workflow_list method:
  • This method process the endpoint response. They have deprecated several response values: isModule and parameters are no longer available. Instead, the function now captures group which will be used to identify module pipelines (system-tools, data-factory,...)
• Refactors Cloudos.is_module method:
  • isModule is not returned by the server, so the method is modified to use group value instead
• Refactors Job.fetch_cloudos_id method:
  • That method serves to fetch ID from projects and workflows. Previously, both endpoints were so similar that we could use almost the same API call. Now, the new GET workflows v3 is too different, so the method was refactored to work with it
• Updates the pytests that were related to workflows, to adapt to the new call
• Updates version number, changelog and the docker image to 2.15.0 (minor update due the amount of changes made)

Tests

Using docker image:

docker run --rm -it quay.io/lifebitaiorg/cloudos-cli:v2.15.0

Global variables:

CLOUDOS="https://cloudos.lifebit.ai"
MY_API_KEY="xxxx"
WORKSPACE_ID="xxxx"

Get workflows list (default: only some fields and in CSV format)

cloudos workflow list \
    --cloudos-url $CLOUDOS \
    --workspace-id $WORKSPACE_ID \
    --apikey $MY_API_KEY

Get workflows list (--all-fields: all available fields and in CSV format)

cloudos workflow list \
    --cloudos-url $CLOUDOS \
    --workspace-id $WORKSPACE_ID \
    --apikey $MY_API_KEY \
    --all-fields

Get workflows list (--output-format json: all fields and in JSON format)

cloudos workflow list \
    --cloudos-url $CLOUDOS \
    --workspace-id $WORKSPACE_ID \
    --apikey $MY_API_KEY \
    --output-format json

Get curated workflows list

cloudos workflow list \
    --cloudos-url $CLOUDOS \
    --workspace-id $WORKSPACE_ID \
    --apikey $MY_API_KEY \
    --curated

Run a normal job

PROJECT_NAME="API jobs"
WORKFLOW_NAME="rnatoy"
JOB_PARAMS="cloudos/examples/rnatoy.config"

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --job-name cloudos_cli_default

Run a normal job from a system-tool selecting latest Nextflow version and another queue (two warnings will appear as it is not allowed to change the Nextflow version or the queue for a system-tool)

WORKFLOW_NAME="GWAS"

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --nextflow-profile test_ci_staging_cloudos_phenofile \
    --job-name cloudos_cli_system_tool \
    --job-queue mountpoints_test \
    --nextflow-version latest

[danielboloc]

LGTM!

[s0rthak]

LGTM!

[dapineyro]

Thanks @s0rthak for your review comments! I've added the changes you suggested.

# Get curated workflow list
cloudos workflow list \
    --cloudos-url $CLOUDOS \
    --workspace-id $WORKSPACE_ID \
    --apikey $MY_API_KEY \
    --curated

# Test 'omics-metadata-validator' (intermediate pipeline) to see if the tool picks it up as a 'module'
WORKFLOW_NAME="omics-metadata-validator"
PROJECT_NAME="API jobs"

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --nextflow-profile test_ci_basic \
    --job-name cloudos_cli_system_tool \
    --job-queue mountpoints_test \
    --nextflow-version latest