Genpei (源平)

Japanese Document

Genpei (源平) is a standard implementation conforming to the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification. A simple and highly scalable REST API Server using Flask and cwltool that follows the philosophy of Microservice. It supports the execution and management of Workflow written in Common Workflow Language (CWL).

Install and Run

Genpei supports Python 3.6 or newer.

$ pip3 install genpei
$ genpei

Docker

We also expect to launch using Docker. Because of the compatibility of cwltool and Docker-in-Docker (DinD), you have to mount docker.sock, /tmp, etc. Please check the documentation in DockerHub - cwltool for more information.

# Launch
$ docker-compose up -d

# Launch confirmation
$ docker-compose logs

Usage

As API specifications, please check GitHub - GA4GH WES and SwaggerUI - GA4GH WES.

As the simplest example of a REST API Request, here is the result of a GET /service-info.

GET /service-info
{
  "auth_instructions_url": "https://github.com/suecharo/genpei",
  "contact_info_url": "https://github.com/suecharo/genpei",
  "default_workflow_engine_parameters": [],
  "supported_filesystem_protocols": [
    "http",
    "https",
    "file"
  ],
  "supported_wes_versions": [
    "1.0.0"
  ],
  "system_state_counts": {},
  "tags": {
    "wes_name": "genpei"
  },
  "workflow_engine_versions": {
    "cwltool": "3.0.20200324120055"
  },
  "workflow_type_versions": {
    "CWL": {
      "workflow_type_version": [
        "v1.0",
        "v1.1",
        "v1.1.0-dev1",
        "v1.2.0-dev1",
        "v1.2.0-dev2"
      ]
    }
  }
}

The host and port used by the application can be changed by specifying the startup arguments (--host and --port). And environment variables corresponding to these arguments are GENPEI_HOST and GENPEI_PORT.

$ genpei --help
usage: genpei [-h] [--host] [-p] [--debug] [-r] [--service-info]

An implementation of GA4GH Workflow Execution Service Standard as a microservice

optional arguments:
  -h, --help       show this help message and exit
  --host           Host address of Flask. (default: 127.0.0.1)
  -p , --port      Port of Flask. (default: 8080)
  --debug          Enable debug mode of Flask.
  -r , --run-dir   Specify the run dir. (default: ./run)
  --service-info   Specify `service-info.json`. The workflow_engine_versions, workflow_type_versions
                   and system_state_counts are overwritten in the application.

$ genpei --host 0.0.0.0 --port 5000

Genpei manages the submitted workflows, workflow parameters, output files, etc. on the file system. The location of run dir can be overridden by the startup argument --run-dir or the environment variable GENPEI_RUN_DIR.

The run dir structure is as follows. Initialization and deletion of each run can be done by physical deletion with rm.

$ tree run
.
├── 11
│   └── 11a23a68-a914-427a-80cd-9ad6f7cfd256
│      ├── cmd.txt
│      ├── end_time.txt
│      ├── exe
│      │   └── workflow_params.json
│      ├── exit_code.txt
│      ├── outputs
│      │   ├── ERR034597_1.small_fastqc.html
│      │   ├── ERR034597_1.small.fq.trimmed.1P.fq
│      │   ├── ERR034597_1.small.fq.trimmed.1U.fq
│      │   ├── ERR034597_1.small.fq.trimmed.2P.fq
│      │   ├── ERR034597_1.small.fq.trimmed.2U.fq
│      │   └── ERR034597_2.small_fastqc.html
│      ├── run.pid
│      ├── run_request.json
│      ├── start_time.txt
│      ├── state.txt
│      ├── stderr.log
│      └── stdout.log
├── 14
│   └── ...
├── 2d
│   └── ...
└── 6b
    └── ...

The execution of POST /runs is very complex. Examples using Python's requests are provided by GitHub - genpei/tests/post_runs_examples. Please use this as a reference

Development

The development environment starts with the following.

$ docker-compose -f docker-compose.dev.yml up -d --build
$ docker-compose -f docker-compose.dev.yml exec app bash

We use flake8, isort, and mypy as the linter.

$ bash ./tests/lint_and_style_check/flake8.sh
$ bash ./tests/lint_and_style_check/isort.sh
$ bash ./tests/lint_and_style_check/mypy.sh

We use pytest as a Test Tool.

$ pytest .

License

Apache-2.0. See the LICENSE.

• The Developer and Maintainer: @suecharo
• The Godfather of this library: @inutano