This repository has been archived by the owner on May 31, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* feat(mic) component init * fix: create a dummy config * add jinja2 as dependecies * fix: init * fix: develop docs * add develop docs * fix: typos * typo * renamed some steps. Changed step 0 to glossary * improve docs * fix: reorder * skeleton git push * adding new structure * fix: ideas * fix: typo * step3: create skeleton * test skeleton * add: test skeleton * Add: Step4 create yaml (#78) * add skel command * adding test * create yaml * fix: step_4 without test * adding tests * adding tests * add test with directories * fix: improve the usability * fix: add pyyaml as dependencies * fix: cleaning filenames * Step4: Add the YAML variable in the run (#79) * fix: step 6 - generate run.sh * add: step3 * add: test step4 * fix: add help text * Add: Step5 local execution * add: local execution * fix: ignore windows * fix: add zip format * Execution docker + outputs (#85) * add docker build * add: replace files * fix: cleaning the code * fix: replace in execution * step7: running using docker * add: status steps * fix: improve docs on cli * Add integration with GitHub (#80) * modelconfiguration publish will make repo if it does not already exist * Added begining of GitHub publish functionality C:\Users\Admin\Desktop\USC\Internships\ISI\mic\mic.zip will now * Fixed quotation marks typo * publish will check parameters before uploading zip file Checks if file exists before uploading. If file exists it will check if anything has changed before updating * Publish now uploads model's READE.md to GitHub Looks for README in given directory. If it cant fine one it will make a new README.md file in the repository * added error if github credentials (Token) are incorrect * Can now configure GitHub credentials in mic configure * add --force (-f) flag * Add a check to make sure README wont be overwridden accidentally Also made -d option actually work * Added documentation on GitHub credentials * Added functionality for github releases * Lower case J, maybe this was failing the test * Added tag and message options for publish command * Better error handeling + tag and message options * Better documentation for generating GitHub token * Added some better details * Updated method documentation Co-authored-by: Maximiliano Osorio <[email protected]> * Publising (#87) * Revert "Add integration with GitHub (#80)" This reverts commit bf9c54e. * add: publish docker and git * Publish (#96) * Revert "Add integration with GitHub (#80)" This reverts commit bf9c54e. * add: publish docker and git * fix: better message * add: handle outputs * Improve documentation and software dependencies * Fix: Added pygit2 to setup.py * Fix: credentials swaps users email and name * Docs: Updated documentation for configure * Docs: Fix typo * Fix: Add PyGithub to setup.py * remove email duplicate Co-authored-by: Maximiliano Osorio <[email protected]> * Mic ch (#99) * Fix: Added pygit2 to setup.py * Fix: credentials swaps users email and name * Docs: Updated documentation for configure * Docs: Fix typo * Fix: Add PyGithub to setup.py * Add: step1 creates .gitignore * Model catalog push (#100) * push to model catalog * fix: push to model catalog * fix: extract zip * fix: use basedir pwd * fixing: step1 * fix: bug step3 * fix: bug step4 render output * fix: verify if outputs is none * fix: default value is 0 * fix: removing the local execution * fix: removing the option add a new mc to a model * Add version mdodel (#104) * fix: add imports * fix: select existing model versions * Mic ch (#102) * Fix: Added pygit2 to setup.py * Fix: credentials swaps users email and name * Docs: Updated documentation for configure * Docs: Fix typo * Fix: Add PyGithub to setup.py * Add: step1 creates .gitignore * add: list_credentials mic list_credentials will show the user their different profile configurations. This will aslo help with debugging issues with credentials. This command will not show password or full github token for configuration * Add: --short flag for list-credentials Short parameter allows user to clearly see a list of the profiles they have configured * fix: typos and minor changes * Fix: import typo * Fix: step4 and pytest typo * Fix: test_executor no longer crashes * Add: append comment to default parameter * Fix: write_step() deletes yaml comments * Fix: Replace yaml.dump with write_to_yaml() write_to_yaml() makes sure that the comments on the yaml file persist when using yaml.dump since yaml.dump will not save any comments made * Add: description field to input, parameter and output fields in mic.yaml * change: config.yaml -> mic.yaml and mic configure -> credentials Co-authored-by: Maximiliano Osorio <[email protected]> * Add version model (#110) * fix: add imports * fix: select existing model versions Co-authored-by: Daniel Garijo <[email protected]> Co-authored-by: Christopher Heidelberg <[email protected]>
- Loading branch information
1 parent
d57c472
commit 9fb8c43
Showing
52 changed files
with
2,702 additions
and
44 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,5 @@ | ||
mydir/ | ||
test | ||
Scripts/ | ||
*.json | ||
pyvenv.cfg | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,85 @@ | ||
## Login and credentials | ||
|
||
The MINT Model Catalog requires credentials for modifying the contents in the catalog. Use this command to configure username and password for the [Model Catalog API](https://model-catalog-python-api-client.readthedocs.io/en/latest/endpoints/). | ||
## Overview | ||
``` | ||
mic configure [-p | --profile] [--server] [--username] [--password] [--name] | ||
[--email] [--git_username] [--git_token] [--dockerhub_username] | ||
``` | ||
|
||
## Description | ||
|
||
mic uses several APIs to upload models. The MINT Model Catalog requires credentials for modifying the contents in the catalog. Use this command to configure username and password for the [Model Catalog API](https://model-catalog-python-api-client.readthedocs.io/en/latest/endpoints/). This command can also be used with no parameters, it will prompt the user to enter any required field not given. | ||
|
||
## Options | ||
|
||
`-p, --profile <profile-name>` | ||
|
||
Credentials can be set up with multiple configuration profiles. This option lets the user choose which profile they are editing. If the profile does not already exist it will generate a new one | ||
|
||
`--server <server url> ` | ||
|
||
The Model Catalog API - [required] | ||
|
||
`--username <username>` | ||
|
||
Email for the Model Catalog API - [required] | ||
|
||
`--password <password>` | ||
|
||
Password for Model Catalog - [required] | ||
|
||
`--name <name>` | ||
|
||
Full name of the author - [required] | ||
|
||
`--git_username <GitHub Username>` | ||
|
||
Author's Github username - [required] | ||
|
||
`--git_token <GitHub API Token>` | ||
|
||
Authors's GitHub API Token. More information can be found in the [setting up GitHub credentials](#GitHubCreds) section below - [required] | ||
|
||
`--dockerhub_username <Username>` | ||
|
||
Username for dockerhub | ||
|
||
|
||
## <a name="GitHubCreds">Setting up GitHub credentials</a> | ||
|
||
GitHub credentials are also required for mic's GitHub features | ||
|
||
The `GitHub Username` field is the users GitHub username. If unknown the username can be found at [GitHub.com](https://github.com/). Once logged, in at the top right dropdown menu there will be a "signed in as **[username]**" | ||
|
||
The `GitHub Token` is the user's [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line). To create a personal access token click [here](https://github.com/settings/tokens/new), or go to GitHub.com -> Settings -> Developer settings -> personal access token. Click **Generate new token** this will open the new personal access token page. The following options must be checked: | ||
|
||
- [x] repo: | Full control of private repositories | ||
- [x] write:packages | Upload packages to github package registry | ||
- [x] read:packages | Download packages from github package registry | ||
|
||
Writing "mic access token" under notes is also recommended | ||
|
||
When done click **Generate token** at the bottom of the page. Once the token is generated be sure to copy and save it in a secure location. Enter this key in the `GitHub API token` field when prompted. | ||
|
||
!!! warning | ||
If this token is lost there is no way to recover it without generating a new one. | ||
|
||
### Setting up DockerHub credentials | ||
|
||
### Example usage: | ||
|
||
``` | ||
$ mic configure | ||
Model Catalog API [https://api.models.mint.isi.edu/v1.4.0]: | ||
Username [[email protected]]: | ||
Password: | ||
Name: | ||
Email: | ||
GitHub Username: | ||
GitHub API token: | ||
Docker Username: | ||
``` | ||
|
||
|
||
|
||
!!! info | ||
[Contact the MINT team](mailto:[email protected]) to create a new user/password if you want to edit your own models. | ||
[Contact the MINT team](mailto:[email protected]) to create a new user/password if you want to edit your own models. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
## Concepts | ||
|
||
- **Software Image**: Computational infrastructure needed to carry out a run of a model. | ||
- **Code repository**: Location where the code of a model resides (e.g., GitHub) | ||
- **Model configuration executable**: Executable definition of the . The executable also has a code repository, which may be different from the target model. For example, I may have a model in a GitHub repository and maintain executables in another repository. | ||
- **Model configuration metadata**: Metadata associated to a model configuration: input and output variables, creator, contributor, license, etc. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
In order to use MIC effectively, your should create a GitHub and DockerHub account. We will use these accounts in MIC to help you publish your component. | ||
|
||
!!! warning | ||
MIC **will not** store your credentials. | ||
|
||
## GitHub | ||
|
||
### Create account | ||
|
||
GitHub is a website and cloud-based service that helps developers store and manage their code, as well as track and control changes to their code. | ||
|
||
`MIC` creates a GitHub repository and push your code. | ||
|
||
### Obtain GitHub Token | ||
|
||
To push your code, you must generate a GitHub Token. Instructions for this can be found [here](../configure.md#GitHubCreds) | ||
|
||
!!! note | ||
A documentation | ||
|
||
## DockerHub | ||
|
||
Docker Hub is a hosted repository service provided by Docker for sharing Docker images | ||
If you don't have a Docker ID, head over to https://hub.docker.com to create one. | ||
|
||
`MIC` creates and pushes Docker images for you. | ||
|
||
### Login | ||
|
||
To push the image, you must login | ||
|
||
```bash | ||
docker login | ||
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. | ||
Username: frink | ||
Password: | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
### Creating the directory skeleton for your first configuration | ||
|
||
You must create the directories and sub directories | ||
|
||
```bash | ||
$ mic encapsulate step1 <model_name> | ||
``` | ||
|
||
MIC has created a directory `model_name`. | ||
In this directory, there two subdirectories: | ||
- data: Contains your data/inputs. | ||
- src: Contains the invocation script. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
## Identify your inputs and copy them | ||
|
||
You must copy your inputs into the directory `data`. | ||
|
||
!!! warning | ||
Your code is not an input. | ||
|
||
An `input` can be: | ||
|
||
- A file in the directory `data` is one input. | ||
- A directory in the directory `data` is one input (MIC is going create a zip file). | ||
|
||
Let's suppose that you have copied the following directory and file | ||
- GLDAS_NOAH025_M.2.1/ - This is a directory | ||
- prepicipitation_rates.txt - This is a file | ||
|
||
|
||
## Identify your parameters | ||
|
||
Analysts may want to explore indicators values under different initial conditions. These are expressed as adjustable parameters of models. | ||
|
||
|
||
Let's suppose that you have identified two parameters: | ||
- start_year: | ||
- end_year | ||
|
||
|
||
## Creating `config.yaml` file | ||
|
||
Then, you must run the command: | ||
|
||
```bash | ||
$ mic encapsulate step2 <model_dir> --inputs_dir data/ --number-parameters 2 | ||
or | ||
$ mic encapsulate step2 <model_dir> --number-parameters 2 | ||
``` | ||
|
||
This command generates `config.yaml` file. This YAML file with the information about your model configuration | ||
|
||
```yaml | ||
inputs: | ||
gldas_noaho25_m.2.1: | ||
path: data/GLDAS_NOAH025_M.2.1/ | ||
prepicipitation_rates: | ||
path: data/prepicipitation_rates.txt | ||
parameters: | ||
parameter1: | ||
default_value: | ||
parameter2: | ||
default_value: | ||
``` | ||
You **must** add: | ||
- A *default_value* for each parameter | ||
You **can** edit | ||
- The name of the parameters and inputs (Spaces are not admitted) | ||
### Creating the invocation code | ||
Then, we must generate the MINT wrapper to run your model | ||
You must pass the `MIC_CONFIG_FILE` (`config.yaml`) using the option (`-f`). | ||
|
||
|
||
```bash | ||
$ mic encapsulate step3 -f config.yaml | ||
The invocation has been created. | ||
``` | ||
|
||
!!! warning | ||
If you edit the inputs or the parameters section in the `config.yaml` file, you must re-run ` mic encapsulate step3 -f config.yaml` | ||
|
||
|
||
In the next step, you are going to learn how to run your models using the MINT Wrapper |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
## MINT wrapper | ||
|
||
You must add the invocation line of your model, | ||
|
||
### Types | ||
|
||
An invocation can be one line. | ||
|
||
For example, FloodSeverityIndex model. [Example](https://github.com/mintproject/MINT-WorkflowDomain/blob/master/WINGSWorkflowComponents/fsi-1.0.0/src/run#L19) | ||
|
||
```bash | ||
python FloodSeverityIndex.py ./ GloFAS_FloodThreshold.nc [23,48,3,15] [2016,2017] True | ||
``` | ||
|
||
Or multiple lines as the HAND model. [Example](https://github.com/mintproject/HAND-TauDEM/blob/master/hand_v2_mint_component/src/run#L27) | ||
|
||
|
||
```bash | ||
... | ||
pitremove -z $1 -fel demfel.tif | ||
dinfflowdir -fel demfel.tif -ang demang.tif -slp demslp.tif | ||
d8flowdir -fel demfel.tif -p demp.tif -sd8 demsd8.tif | ||
aread8 -p demp.tif -ad8 demad8.tif -nc | ||
areadinf -ang demang.tif -sca demsca.tif -nc | ||
|
||
## Skeleton | ||
slopearea -slp demslp.tif -sca demsca.tif -sa demsa.tif | ||
d8flowpathextremeup -p demp.tif -sa demsa.tif -ssa demssa.tif -nc | ||
python3 hand-thresh.py --resolution demfel.tif --output demthresh.txt | ||
threshold -ssa demssa.tif -src demsrc.tif -thresh 500 | ||
|
||
streamnet -fel demfel.tif -p demp.tif -ad8 demad8.tif -src demsrc.tif -ord demord.tif -tree demtree.dat -coord demcoord.dat -net demnet.shp -w demw.tif -sw | ||
|
||
connectdown -p demp.tif -ad8 demad8.tif -w demw.tif -o outlets.shp -od movedoutlets.shp | ||
|
||
python3 hand-heads.py --network demnet.shp --output dangles.shp | ||
python3 hand-weights.py --shapefile dangles.shp --template demfel.tif --output demwg.tif | ||
``` | ||
|
||
What is the best option? That's is your decision. We provide flexibility. | ||
|
||
|
||
|
||
|
||
### Adding your invocation line | ||
|
||
!!! info | ||
The language of the run file is bash. | ||
|
||
|
||
1. Open the file `src/run` | ||
2. Add the invocation line(s) after the comment `# WRITE THE COMMAND LINE INVOCATION HERE.` | ||
|
||
|
||
``` | ||
# WRITE THE COMMAND LINE INVOCATION HERE | ||
python FloodSeverityIndex.py ./ GloFAS_FloodThreshold.nc [23,48,3,15] [2016,2017] True | ||
``` | ||
|
||
|
||
On the next page, we are going to learn how to pass the parameters to your model. |
Oops, something went wrong.