Skip to content
This repository has been archived by the owner on May 31, 2024. It is now read-only.

Commit

Permalink
Add: encapsulate process (#111)
Browse files Browse the repository at this point in the history
* feat(mic) component init

* fix: create a dummy config

* add jinja2 as dependecies

* fix: init

* fix: develop docs

* add develop docs

* fix: typos

* typo

* renamed some steps. Changed step 0 to glossary

* improve docs

* fix: reorder

* skeleton git push

* adding new structure

* fix: ideas

* fix: typo

* step3: create skeleton

* test skeleton

* add: test skeleton

* Add: Step4 create yaml (#78)

* add skel command

* adding test

* create yaml

* fix: step_4 without test

* adding tests

* adding tests

* add test with directories

* fix: improve the usability

* fix: add pyyaml as dependencies

* fix: cleaning filenames

* Step4: Add the YAML variable in the run (#79)

* fix: step 6 - generate run.sh

* add: step3

* add: test step4

* fix: add help text

* Add: Step5 local execution

* add: local execution

* fix: ignore windows

* fix: add zip format

* Execution docker + outputs (#85)

* add docker build

* add: replace files

* fix: cleaning the code

* fix: replace in execution

* step7: running using docker

* add: status steps

* fix: improve docs on cli

* Add integration with GitHub (#80)

* modelconfiguration publish will make repo if it does not already exist

* Added begining of GitHub publish functionality

C:\Users\Admin\Desktop\USC\Internships\ISI\mic\mic.zip will now

* Fixed quotation marks typo

* publish will check parameters before uploading zip file

Checks if file exists before uploading. If file exists it will check if anything has changed before updating

* Publish now uploads model's READE.md to GitHub

Looks for README in given directory. If it cant fine one it will make a new README.md file in the repository

* added error if github credentials (Token) are incorrect

* Can now configure GitHub credentials in mic configure

* add --force (-f) flag

* Add a check to make sure README wont be overwridden accidentally

Also made -d option actually work

* Added documentation on GitHub credentials

* Added functionality for github releases

* Lower case J, maybe this was failing the test

* Added tag and message options for publish command

* Better error handeling + tag and message options

* Better documentation for generating GitHub token

* Added some better details

* Updated method documentation

Co-authored-by: Maximiliano Osorio <[email protected]>

* Publising (#87)

* Revert "Add integration with GitHub (#80)"

This reverts commit bf9c54e.

* add: publish docker and git

* Publish (#96)

* Revert "Add integration with GitHub (#80)"

This reverts commit bf9c54e.

* add: publish docker and git

* fix: better message

* add: handle outputs

* Improve documentation and software dependencies

* Fix: Added pygit2 to setup.py

* Fix: credentials swaps users email and name

* Docs: Updated documentation for configure

* Docs: Fix typo

* Fix: Add PyGithub to setup.py

* remove email duplicate

Co-authored-by: Maximiliano Osorio <[email protected]>

* Mic ch (#99)

* Fix: Added pygit2 to setup.py

* Fix: credentials swaps users email and name

* Docs: Updated documentation for configure

* Docs: Fix typo

* Fix: Add PyGithub to setup.py

* Add: step1 creates .gitignore

* Model catalog push (#100)

* push to model catalog

* fix: push to model catalog

* fix: extract zip

* fix: use basedir pwd

* fixing: step1

* fix: bug step3

* fix: bug step4 render output

* fix: verify if outputs is none

* fix: default value is 0

* fix: removing the local execution

* fix: removing the option add a new mc to a model

* Add version mdodel (#104)

* fix: add imports

* fix: select existing model versions

* Mic ch (#102)

* Fix: Added pygit2 to setup.py

* Fix: credentials swaps users email and name

* Docs: Updated documentation for configure

* Docs: Fix typo

* Fix: Add PyGithub to setup.py

* Add: step1 creates .gitignore

* add: list_credentials

mic list_credentials will show the user their different profile configurations. This will aslo help with debugging issues with credentials. This command will not show password or full github token for configuration

* Add: --short flag for list-credentials

Short parameter allows user to clearly see a list of the profiles they have configured

* fix: typos and minor changes

* Fix: import typo

* Fix: step4 and pytest typo

* Fix: test_executor no longer crashes

* Add: append comment to default parameter

* Fix: write_step() deletes yaml comments

* Fix: Replace yaml.dump with write_to_yaml()

write_to_yaml() makes sure that the comments on the yaml file persist when using yaml.dump since yaml.dump will not save any comments made

* Add: description field to input, parameter and output fields in mic.yaml

* change: config.yaml -> mic.yaml and mic configure -> credentials

Co-authored-by: Maximiliano Osorio <[email protected]>

* Add version model (#110)

* fix: add imports

* fix: select existing model versions

Co-authored-by: Daniel Garijo <[email protected]>
Co-authored-by: Christopher Heidelberg <[email protected]>
  • Loading branch information
3 people authored Jun 2, 2020
1 parent d57c472 commit 9fb8c43
Show file tree
Hide file tree
Showing 52 changed files with 2,702 additions and 44 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
mydir/
test
Scripts/
*.json
pyvenv.cfg
Expand Down
62 changes: 57 additions & 5 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,61 @@
dist: xenial
language: python
python:
- '3.7'
install: pip install tox-travis
script: tox
matrix:
include:
- os: linux
python: '3.6'
- os: linux
python: '3.7'
- os: linux
python: '3.8'
- os: osx
language: generic
python: '3.6'
before_install:
- brew upgrade pyenv
- brew install pyenv-virtualenv
- export CFLAGS="-I$(brew --prefix openssl)/include"
- export LDFLAGS="-L$(brew --prefix openssl)/lib"
- pyenv install 3.6.7
- eval "$(pyenv init -)"
- pyenv virtualenv 3.6.7 venv
- pyenv activate venv
- os: osx
language: generic
python: '3.7'
before_install:
- brew upgrade pyenv
- brew install pyenv-virtualenv
- export CFLAGS="-I$(brew --prefix openssl)/include"
- export LDFLAGS="-L$(brew --prefix openssl)/lib"
- pyenv install 3.7.2
- eval "$(pyenv init -)"
- pyenv virtualenv 3.7.2 venv
- pyenv activate venv
- os: windows
language: sh
python: '3.6'
before_install:
- choco install python --version 3.6.7
- export PATH="/c/Python36:/c/Python36/Scripts:$PATH"
- os: windows
language: sh
python: '3.7'
before_install:
- choco install python --version 3.7.2
- export PATH="/c/Python37:/c/Python37/Scripts:$PATH"
allow_failures:
- os: windows
install:
- pip install coverage
- pip install codecov
- pip install -U pytest
- pip install -U pytest-cov
- pip install --upgrade setuptools wheel
script:
- python setup.py install
- pytest
after_success:
- codecov
deploy:
provider: pypi
user: __token__
Expand Down
78 changes: 75 additions & 3 deletions docs/configure.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,85 @@
## Login and credentials

The MINT Model Catalog requires credentials for modifying the contents in the catalog. Use this command to configure username and password for the [Model Catalog API](https://model-catalog-python-api-client.readthedocs.io/en/latest/endpoints/).
## Overview
```
mic configure [-p | --profile] [--server] [--username] [--password] [--name]
[--email] [--git_username] [--git_token] [--dockerhub_username]
```

## Description

mic uses several APIs to upload models. The MINT Model Catalog requires credentials for modifying the contents in the catalog. Use this command to configure username and password for the [Model Catalog API](https://model-catalog-python-api-client.readthedocs.io/en/latest/endpoints/). This command can also be used with no parameters, it will prompt the user to enter any required field not given.

## Options

`-p, --profile <profile-name>`

Credentials can be set up with multiple configuration profiles. This option lets the user choose which profile they are editing. If the profile does not already exist it will generate a new one

`--server <server url> `

The Model Catalog API - [required]

`--username <username>`

Email for the Model Catalog API - [required]

`--password <password>`

Password for Model Catalog - [required]

`--name <name>`

Full name of the author - [required]

`--git_username <GitHub Username>`

Author's Github username - [required]

`--git_token <GitHub API Token>`

Authors's GitHub API Token. More information can be found in the [setting up GitHub credentials](#GitHubCreds) section below - [required]

`--dockerhub_username <Username>`

Username for dockerhub


## <a name="GitHubCreds">Setting up GitHub credentials</a>

GitHub credentials are also required for mic's GitHub features

The `GitHub Username` field is the users GitHub username. If unknown the username can be found at [GitHub.com](https://github.com/). Once logged, in at the top right dropdown menu there will be a "signed in as **[username]**"

The `GitHub Token` is the user's [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line). To create a personal access token click [here](https://github.com/settings/tokens/new), or go to GitHub.com -> Settings -> Developer settings -> personal access token. Click **Generate new token** this will open the new personal access token page. The following options must be checked:

- [x] repo: | Full control of private repositories
- [x] write:packages | Upload packages to github package registry
- [x] read:packages | Download packages from github package registry

Writing "mic access token" under notes is also recommended

When done click **Generate token** at the bottom of the page. Once the token is generated be sure to copy and save it in a secure location. Enter this key in the `GitHub API token` field when prompted.

!!! warning
If this token is lost there is no way to recover it without generating a new one.

### Setting up DockerHub credentials

### Example usage:

```
$ mic configure
Model Catalog API [https://api.models.mint.isi.edu/v1.4.0]:
Username [[email protected]]:
Password:
Name:
Email:
GitHub Username:
GitHub API token:
Docker Username:
```



!!! info
[Contact the MINT team](mailto:[email protected]) to create a new user/password if you want to edit your own models.
[Contact the MINT team](mailto:[email protected]) to create a new user/password if you want to edit your own models.
Binary file added docs/figures/tutorial/01.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/glossary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
## Concepts

- **Software Image**: Computational infrastructure needed to carry out a run of a model.
- **Code repository**: Location where the code of a model resides (e.g., GitHub)
- **Model configuration executable**: Executable definition of the . The executable also has a code repository, which may be different from the target model. For example, I may have a model in a GitHub repository and maintain executables in another repository.
- **Model configuration metadata**: Metadata associated to a model configuration: input and output variables, creator, contributor, license, etc.
37 changes: 37 additions & 0 deletions docs/model_configuration/02-pre-steps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
In order to use MIC effectively, your should create a GitHub and DockerHub account. We will use these accounts in MIC to help you publish your component.

!!! warning
MIC **will not** store your credentials.

## GitHub

### Create account

GitHub is a website and cloud-based service that helps developers store and manage their code, as well as track and control changes to their code.

`MIC` creates a GitHub repository and push your code.

### Obtain GitHub Token

To push your code, you must generate a GitHub Token. Instructions for this can be found [here](../configure.md#GitHubCreds)

!!! note
A documentation

## DockerHub

Docker Hub is a hosted repository service provided by Docker for sharing Docker images
If you don't have a Docker ID, head over to https://hub.docker.com to create one.

`MIC` creates and pushes Docker images for you.

### Login

To push the image, you must login

```bash
docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: frink
Password:
```
12 changes: 12 additions & 0 deletions docs/model_configuration/03-create-skeleton.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
### Creating the directory skeleton for your first configuration

You must create the directories and sub directories

```bash
$ mic encapsulate step1 <model_name>
```

MIC has created a directory `model_name`.
In this directory, there two subdirectories:
- data: Contains your data/inputs.
- src: Contains the invocation script.
77 changes: 77 additions & 0 deletions docs/model_configuration/04-copy-your-data.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
## Identify your inputs and copy them

You must copy your inputs into the directory `data`.

!!! warning
Your code is not an input.

An `input` can be:

- A file in the directory `data` is one input.
- A directory in the directory `data` is one input (MIC is going create a zip file).

Let's suppose that you have copied the following directory and file
- GLDAS_NOAH025_M.2.1/ - This is a directory
- prepicipitation_rates.txt - This is a file


## Identify your parameters

Analysts may want to explore indicators values under different initial conditions. These are expressed as adjustable parameters of models.


Let's suppose that you have identified two parameters:
- start_year:
- end_year


## Creating `config.yaml` file

Then, you must run the command:

```bash
$ mic encapsulate step2 <model_dir> --inputs_dir data/ --number-parameters 2
or
$ mic encapsulate step2 <model_dir> --number-parameters 2
```

This command generates `config.yaml` file. This YAML file with the information about your model configuration

```yaml
inputs:
gldas_noaho25_m.2.1:
path: data/GLDAS_NOAH025_M.2.1/
prepicipitation_rates:
path: data/prepicipitation_rates.txt
parameters:
parameter1:
default_value:
parameter2:
default_value:
```
You **must** add:
- A *default_value* for each parameter
You **can** edit
- The name of the parameters and inputs (Spaces are not admitted)
### Creating the invocation code
Then, we must generate the MINT wrapper to run your model
You must pass the `MIC_CONFIG_FILE` (`config.yaml`) using the option (`-f`).


```bash
$ mic encapsulate step3 -f config.yaml
The invocation has been created.
```

!!! warning
If you edit the inputs or the parameters section in the `config.yaml` file, you must re-run ` mic encapsulate step3 -f config.yaml`


In the next step, you are going to learn how to run your models using the MINT Wrapper
61 changes: 61 additions & 0 deletions docs/model_configuration/05-write-invocation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
## MINT wrapper

You must add the invocation line of your model,

### Types

An invocation can be one line.

For example, FloodSeverityIndex model. [Example](https://github.com/mintproject/MINT-WorkflowDomain/blob/master/WINGSWorkflowComponents/fsi-1.0.0/src/run#L19)

```bash
python FloodSeverityIndex.py ./ GloFAS_FloodThreshold.nc [23,48,3,15] [2016,2017] True
```

Or multiple lines as the HAND model. [Example](https://github.com/mintproject/HAND-TauDEM/blob/master/hand_v2_mint_component/src/run#L27)


```bash
...
pitremove -z $1 -fel demfel.tif
dinfflowdir -fel demfel.tif -ang demang.tif -slp demslp.tif
d8flowdir -fel demfel.tif -p demp.tif -sd8 demsd8.tif
aread8 -p demp.tif -ad8 demad8.tif -nc
areadinf -ang demang.tif -sca demsca.tif -nc

## Skeleton
slopearea -slp demslp.tif -sca demsca.tif -sa demsa.tif
d8flowpathextremeup -p demp.tif -sa demsa.tif -ssa demssa.tif -nc
python3 hand-thresh.py --resolution demfel.tif --output demthresh.txt
threshold -ssa demssa.tif -src demsrc.tif -thresh 500

streamnet -fel demfel.tif -p demp.tif -ad8 demad8.tif -src demsrc.tif -ord demord.tif -tree demtree.dat -coord demcoord.dat -net demnet.shp -w demw.tif -sw

connectdown -p demp.tif -ad8 demad8.tif -w demw.tif -o outlets.shp -od movedoutlets.shp

python3 hand-heads.py --network demnet.shp --output dangles.shp
python3 hand-weights.py --shapefile dangles.shp --template demfel.tif --output demwg.tif
```

What is the best option? That's is your decision. We provide flexibility.




### Adding your invocation line

!!! info
The language of the run file is bash.


1. Open the file `src/run`
2. Add the invocation line(s) after the comment `# WRITE THE COMMAND LINE INVOCATION HERE.`


```
# WRITE THE COMMAND LINE INVOCATION HERE
python FloodSeverityIndex.py ./ GloFAS_FloodThreshold.nc [23,48,3,15] [2016,2017] True
```


On the next page, we are going to learn how to pass the parameters to your model.
Loading

0 comments on commit 9fb8c43

Please sign in to comment.