diff --git a/docs/index.md b/docs/index.md index ad89796..ff8c011 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,89 +1,35 @@ -Model Insertion command-line interface (MIC) is an application to assist modelers for adding through the steps required for encapsulating your model component and exposing a set of inputs and parameters of interest so they can be added on a Model Catalog Service. +## Introduction -In addition, MIC also allows describing basic model metadata such as model version, model configuration, parameters, inputs, outputs, authors and contributors. - -MIC has been tested in OSX, Linux and Windows. It is installed through a simple pip command. +The Model Insertion command-line interface (MIC) is an application to assist scientists in encapsulating their softwares (e.g., models, pre-processing and post-processing codes, visualizations) in a virtual environment to ensure their execution independently of the evolution of the libraries used to support them. In addition, MIC allows scientists to expose a set of inputs and parameters of interest and to describe basic metadata information to ensure maximum reuse. !!! info MIC is an ALPHA version, which we are still testing and developing continuously. If you find an error or experience any issue, please report them [here](https://github.com/mintproject/mic/issues/new/choose). -## What is a model components - -Encapsulating software into components allows other users to easily access and run software in their own environments. Following well-established component-based software engineering principles, we want to create self-contained software components that only reveal functionality that is of interest to third parties. This is important because models are often implemented in large software packages or libraries that contain many tools and functions to run the model in many different ways, to prepare data, to visualize data, etc. It is hard to navigate all the possible functions, especially for those who are interested in sophisticated functionality that may be hard to achieve. Other models have graphical user interfaces that are convenient to use, but cannot be used for invoking the model from another program. A user interface button to run a model would call a specific function of the model software, and that function (sometimes called a *command line invocation*, or *invocation function*) is what we want to capture. That function is known as the *component interface*, and its inputs can be provided when invoking the component but all other data or parameters will be pre-set and internal to the component so none will be able to change them. Finally, for reproducibility reasons, we want to be able to record how a model execution was set up, which means having an explicit description of the specific function call that was used to run the model. These issues are addressed by encapsulating software. - -A **model component** corresponds to a single invocation function for model software. From a sophisticated model software package, a model component could be created to include only certain model processes and variables while excluding others. For example, from a hydrology model software package we could create a component for arid zones that includes infiltration processes but not snowmelt processes from the package. The invocation function for that configuration could have as input the recharge rates. - -## How MIC Works - -MIC guides you to create a model component and uploading it to the MINT Model Catalog so it is available to others in 9 simple steps. - -Below is an overview of the different steps in MIC. - -![Diagram](figures/overview_01.png) - -## Requirements - -MIC has the following requirements: - -1. Python >= 3.6 -2. Docker - - -### Getting Python 3 - -MIC uses Python. Please, follow the steps bellow to install it: - -- [Installation on Linux](https://realpython.com/installing-python/#linux) -- [Installation on Windows](https://realpython.com/installing-python/#windows) -- [Installation on Mac](https://realpython.com/installing-python/#macos-mac-os-x) - -### Docker - -MIC uses Docker test and run model components. - -- [Installation on Linux](https://docs.docker.com/engine/install/) -- [Installation on MacOS](https://docs.docker.com/docker-for-mac/install/) - - -## Installation - -To install MIC, open a terminal and run: - -```bash -$ pip install mic -``` - -You did it! If you want to verify the installation just type: - -```bash -$ mic version -``` - -You should see a message similar to: - -```bash -mic v1.0.1 -``` +### What is a software component? -## Limitations +Encapsulating software into components allows other users to easily access and run software on their own machine without worrying about the environment the software needs. Following well-established component-based software engineering principles, we want to create self-contained software components that only reveal functionality that is of interest to third party users. This is important because scientific softwares are often implemented in large packages or libraries that can be used for various steps such as data preparation and visualization in addition to writing software to model specific processes (such as atmospheric dynamics for climate models, runoff and infiltration for hydrology models, fuel density for fire modeling...). -Note that MIC has been designed to run Unix-based applications. Windows based applications (e.g., models that execute through an .exe) are not currently supported. +These packages can be quite overwhelming for users, even if they are associated with the scientific domain for which the package has been written. It becomes an impossible tasks for users outside of the domain but who could make use of specific results. -## Development version +Over the years, scientists have created various user interfaces to conveniently provide access to their software but these interfaces are often not reusable from another program, requiring significant movement of data from one platform to the next. However, the basic framework for these user interfaces involves a specific function to call the software from a button in the interface and a fillable form that sets specific parameters. That call (sometimes called a *command line invocation*) is reusable from one framework to the next, provided that the software itself is containerized in such as way that it can be executed from any local machine in its specific environment. The call function is known as the *component interface*, and its inputs can be provided when invoking the component (as is done in a user interface where the values for some input parameters are filled out.) Other inputs can be pre-set within the component (including data files) if there are no reasons for third party users to change them given a specific application. -If you want to install the latest development version, open a terminal and type: +A **software component** corresponds to a single invocation function for software. From a sophisticated software package, a software component could be created to include only certain processes and variables, a specific pre-processing step, or a specific visualization. For example, a hydrology model software could be pre-set to be working in hot arid region only and ignore the processes (and therefore inputs) describing snowmelt. -```bash -$ pip install git+https://github.com/mintproject/mic.git@develop -U -``` -Note that the development version may be unstable. +The purpose of MIC is to assist a scientist in creating a model component for various pieces of their workflows, including data preparation and pre-processing, modeling, and visualizations. -## Issues, Troubleshooting and Feature Requests +MIC is meant to work with the [MINT Model Catalog](https://github.com/mintproject/ModelCatalog). However, the container it creates in the process can be reused outside of this framework. -Known issues with MIC are listed [here](https://github.com/mintproject/mic/issues). +### Why should you create a software component? -If you experience any issues when using MIC, or if you would like us to support additional exciting features, please open an issue on our [GitHub repository](https://github.com/mintproject/mic/issues). +* Everyone will be able to execute your model in their own local environments and operating systems (Linux, MacOS or Windows). +* You can simplify the number of inputs needed for a particular application. +* You will be able to create a secure environment for executing the model. When others want to run your model, they will be reassured that the model will not disrupt or delete anything in their local environment. +* Everyone will be able to find your component in a model catalog supported by the [MINT model catalog ontology](https://github.com/mintproject/Mint-ModelCatalog-Ontology) such as [MINT model catalog](https://models.mint.isi.edu/) using [DAME](dame-cli.readthedocs.io/). -## Code Releases and Next Updates +## Getting Started -The [latest release of MIC is available in GitHub](https://github.com/mintproject/mic/releases/latest). You can check the issues and updates we are working on for the next releases [here](https://github.com/mintproject/mic/milestones). +* [Installing MIC](installation) +* [Overview of the MIC process](overview) +* [Available commands](usage) +* [Step-by-step instructions](/model_configuration/01-overview) for creating a software component +* Report bugs and problems with the code or documentation to our [GitHub repository](https://github.com/mintproject/mic/issues). diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..5c4bd76 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,60 @@ +# Installation + +MIC has been tested on Mac, Linux, and Windows Platforms + +## Requirements + +1. Python >= 3.6 +2. Docker + +### Getting Python 3 + +To install Python: + +- [Installation on Linux](https://realpython.com/installing-python/#linux) +- [Installation on Windows](https://realpython.com/installing-python/#windows) +- [Installation on Mac](https://realpython.com/installing-python/#macos-mac-os-x) + +You can also complete the installation using alternative distribution such as [Anaconda](https://www.anaconda.com), a popular platform for scientists. If you plan on using Anaconda, we recommend that you set up MIC in a new environment. + +```bash +$ conda create --name myenv +$ conda activate myenv +``` + +### Docker + +- [Installation on Linux](https://docs.docker.com/engine/install/) +- [Installation on MacOS](https://docs.docker.com/docker-for-mac/install/) + +## Limitations + +Note that MIC has been designed to run Unix-based applications. Windows based applications (e.g., models that execute through an .exe) are not currently supported. + +## Installing MIC + +MIC install through `pip`. No Anaconda distribution currently available. + +```bash +$ pip install mic +``` + +To check your installation: + +```bash +$ mic version +mic v1.3.7 +``` + +### Development version + +If you want to install the latest development version, open a terminal and type: + +```bash +$ pip install git+https://github.com/mintproject/mic.git@develop -U +``` +Note that the development version may be unstable. + +## Code Releases and Next Updates + +The [latest release of MIC is available from GitHub](https://github.com/mintproject/mic/releases/latest). You can also check current development from our [GitHub milestones](https://github.com/mintproject/mic/milestones). diff --git a/docs/model_configuration/02-pre-steps.md b/docs/model_configuration/01-DockerGitHub.md similarity index 77% rename from docs/model_configuration/02-pre-steps.md rename to docs/model_configuration/01-DockerGitHub.md index 3e40f13..30fc159 100644 --- a/docs/model_configuration/02-pre-steps.md +++ b/docs/model_configuration/01-DockerGitHub.md @@ -1,24 +1,25 @@ -In order to use MIC, you should create a GitHub and a DockerHub account. We will use these accounts in MIC to help you upload your component. You only need to perform these steps once. Even though we recommend you performing these steps before starting using MIC, **you only need them for performing the `upload` step**. +In order to use MIC, you need to create a GitHub and a DockerHub account. We will use these accounts in MIC to help you upload your component. Even though we recommend you performing these steps before starting using MIC, you only need them for performing the `upload` step. +**You only need to perform these steps once.** If you have already used MIC to encapsulate a component, skip to [Preparing your executable](/model_configuration/02-Preparing your executable). -## Create a GitHub account +## Creating a GitHub account GitHub is a website and cloud-based service that helps developers store and manage their code, as well as track and control changes to their code. MIC uses GitHub to store a snapshot of your code and invocation by creating a GitHub repository and pushing your local code files. The code is stored in your account, so you will be the owner. ### Obtain your GitHub access token -In order for MIC to push code in your repository, you must generate a GitHub Token. +In order for MIC to push code in your repository, you must generate a GitHub Token. This token can be generated on the GitHub website. Once logged, in at the top right dropdown menu there will be a "signed in as **[username]**" The `GitHub Token` is the user's [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line). To create a personal access token click [here](https://github.com/settings/tokens/new), or go to GitHub.com -> Settings -> Developer settings -> personal access token. Click **Generate new token** this will open the new personal access token page. The following options must be checked: - + - [x] repo: | Full control of private repositories - [x] write:packages | Upload packages to github package registry - [x] read:packages | Download packages from github package registry - + Writing "mic access token" under notes is also recommended - + When done click **Generate token** at the bottom of the page. Once the token is generated be sure to copy and save it in a secure location. Enter this key in the `GitHub API token` field when prompted by MIC. !!! warning @@ -42,7 +43,6 @@ Username: frink Password: ``` -### Configure MIC - -Type `mic credentials` and fill in the information requested. You only need to do this once. +## Configure MIC +You need to give MIC access to these credentials. The [`mic credentials` command](/commands/configure) will assist you with that step. diff --git a/docs/model_configuration/01-overview.md b/docs/model_configuration/01-overview.md deleted file mode 100644 index be2e0ef..0000000 --- a/docs/model_configuration/01-overview.md +++ /dev/null @@ -1,118 +0,0 @@ -# Creating a Model Component - -The `pkg` command in MIC guides you through nine steps to create a model component that encapsulates your model code and makes it available in MINT so others can easily run it in their own local environments. To accomplish this, MIC relies on state-of-the-art virtualization tools (including Docker), software engineering best practices, and semantic reasoning for validation. - -There are several reasons for creating a model component: - -* You will be able to expose only the inputs and parameters that are more important for a particular use of the model, pre-setting the rest and making them internal to the model component and inaccessible to other users. -* You will be able to create a secure environment for executing the model. When others want to run your model, they will be reassured that the model will not disrupt or delete anything in their local environment. -* Everyone will be able to find your component in a model catalog supported by the [MINT model catalog ontology](https://github.com/mintproject/Mint-ModelCatalog-Ontology) such as [MINT model catalog](https://models.mint.isi.edu/) using [DAME](dame-cli.readthedocs.io/). -* Everyone will be able to execute your model in their own local environments and operating systems (Linux, MacOS or Windows). - -## What is a Model Component - -In preparation for creating a model component, you should consider what set of processes you would like to represent and what variables should be preset vs the ones you would want other users to vary. - -## Prepare your executable - -1. Place your model code in a local directory so it can be invoked from a command line along with any input data and configuration files needed. MIC will test that the code can be executed and will do so three times during the encapsulation process. Therefore, we recommend that your execution can be completed in a manner of minutes, for example, by reducing the size of the dataset (e.g., only considering a few months instead of a year). Because of limitation on GitHub which will be used during the process, files should not exceed **100 MB** each. -2. The code should not contain any hardcoded paths or values for the input files/variables that you wish to expose. We recommend making them explicit in a configuration file or as parameters from the command line execution. - -## Before you start - -Before you start, you should have a GitHub and Docker account setup following [these guidelines](02-pre-steps.md). - -## Overview of the MIC process - -MIC has an `pkg` command that will guide you through those steps in the form of nine subcommands. Each subcommand represents one of the nine steps to encapsulate your model. The steps should be followed in order, and after each step you will have a chance to check what MIC is doing to help you create your model component. - -MIC creates: 1) a MIC Directory, 2) a MIC File, 3) a MIC Wrapper, 4) a component virtualization image, 5) archival versions of your model code, MIC directories and files, and image in GitHub and DockerHub, and 6) a model component entry in the MINT Model Catalog. - -Below is a summary of the steps, please click on each page for detailed instructions and troubleshooting. - -!!! warning -DO NOT CLOSE the terminal in-between each step. -After inspecting the `.yml` file in-between steps, CLOSE the file. - -### Step 1: Start a MIC encapsulation component -Select a working directory (i.e., the directory where you have your input datasets and model executables) and start a new execution environment to run your model. MIC will create a template *MIC file* that will be modified in subsequent steps with information about the model component. The execution environment is created from scratch based on the files in your working directory. For example, if you have python executables, MIC will prepare a python Docker image to execute your model. - -**Expected outcomes of this step**: A folder (`mic`) with a placeholder MIC file that shows up as `mic.yaml` file in that folder. - -### Step 2: Trace your model execution -Install any missing dependency needed to execute your model and trace the execution of your model. - -**Expected outcome of this step**: You will see that your model took a little longer to run and that a .reprozip trace file was successfully added to your folder. - -### Step 3: Select parameters to expose -Specify the input parameters that you would like users to provide for your model component. This information will be added to the MIC file. - -**Expected outcome of this step**: The MIC file will be automatically edited to include information about the input parameters that you specified for your model component. - -### Step 4: Pre-populate the model configuration files -You will edit your model configuration files to include any new parameters that you specified as component inputs. In this step MIC will guide you to edit these configuration files so those parameters are appropriately set. - -**Expected results after completing this step**: MIC will update the MIC file with pointers to the model configuration files that set the parameters that are input to your component. - -### Step 5: Specify inputs to expose -MIC detects automatically most of the inputs your component used for execution. However, we need to determine which of these inputs do you actually want to expose in your model component. This information will be added to the MIC file. - -**Expected outcome of this step**: The MIC file will be automatically edited to include information about the input datasets that you specified for your model component. - -### Step 6: Specify outputs to expose -MIC detects automatically most of the outputs your component produced after its execution. However, we need to determine which of these outputs do you actually want to expose in your model component. This information will be added to the MIC file. - -**Expected outcome of this step**: The MIC file will be automatically edited to include information about the output datasets that you specified for your model component. - -### Step 7: Prepare MIC wrapper -Create three subfolders (`data`, `src` and `docker`) in the mic folder and *MIC wrapper* that indicates how the component invocation from Step 2 corresponds to the functions implemented in your model software. - -**Expected outcome of this step**: The MIC Wrapper will automatically set up three files (run, io.sh, and output.sh) that capture how the component has been run. They can be found in the /src folder of the MIC directory. - -### Step 8: Run your model through the MIC wrapper -Use the wrapper generated in Step 7 to execute your model. - -**Expected outcome of this step**: A successful test run of your model. If the tests are successful, the encapsulation of your model component has been successful. - -### Step 9: Upload the model component software and image -MIC will upload the MIC Wrapper and your model software in GitHub. MIC will also upload the model component image in DockerHub and upload your model in the MINT model catalog, which will make it accessible by others through MINT services and interfaces to be run in their own local hosts and servers. This will give them unique identifiers that represent the snapshot of the model that you wanted to encapsulate so that any future updates to your model or your model component can be distinct from each other. - -**Expected results after completing this step**: Your model component wrapper will be uploaded to Github, and your model component image to DockerHub. Both will receive a tag and will be versioned, and will be archived so they are available to anyone anywhere. MINT will have an entry for your model component in the MINT Model Catalog, which will be accessible through your browser. Anyone using DAME can run your component with their own data. Anyone using MINT can run your component with any dataset in MINT. - - -## Using MIC to Create Model Components - -Once you have MIC installed, you can type `mic pkg --help` to see the following message that lists all the steps: - -```bash -Usage: mic pkg [OPTIONS] COMMAND [ARGS]... - - Command to pkg your model component - -Options: - --help Show this message and exit. - -Commands: - start Create a Linux environment to run your model. The working - directory selected must contain all the files required for the - execution of your model - - trace Trace any command line and extract the information about your - model execution - - parameters Expose parameters in the mic.yaml file - configs Select configuration file(s) for your model (if applicable) - inputs Expose model inputs into the mic.yaml file - outputs Expose model outputs in the mic.yaml file - wrapper Generate the directory structure and commands required to run - your model component using the information from the previous - steps - - run Run your model component with the MIC Wrapper generated in the - previous step - - upload Upload your code to GitHub, DockerHub and your - model component on the MINT Model Catalog. -``` - -You have to follow the commands in order to successfully upload your component. If you want to know more about a specific step, just do `mic pkg --help`, where `` represents one of the steps listed above (e.g., `trace`). diff --git a/docs/model_configuration/02-Preparing your executable.md b/docs/model_configuration/02-Preparing your executable.md new file mode 100644 index 0000000..9bb4019 --- /dev/null +++ b/docs/model_configuration/02-Preparing your executable.md @@ -0,0 +1,26 @@ +## Preparing your executable + +Ultimately, a user will want to run your code with whatever data is available. However, for the purpose of encapsulating your software, think of the MIC process as a test run. Therefore, we strongly advise that you are prepared to run with the minimum amount of data needed for an execution test. For instance, if your software is used to model hydrologic processes, can you test the software with just 3 hours worth of hourly data or do you really need to run the entire day? Remember that at this stage, you are testing whether the component is encapsulated successfully not that it will work for all data, regardless of whether they have been preprocessed successfully nor that the results are scientifically correct. + +Once you are ready to proceed: + +1. Place your software code in a local directory so it can be invoked from a command line along with any input data and configuration files if applicable. MIC will test that the code can be executed and will do so three times during the encapsulation process. Therefore, we recommend that your execution can be completed in a manner of minutes, for example, by reducing the size of the dataset (e.g., only considering a few months instead of a year). Because of GitHub limitations, files should not exceed **100 MB** each. +2. The code should not contain any hardcoded paths or values for the input files/variables that you wish to expose. We recommend making them explicit in a configuration file or as parameters from the command line execution. + +## Parameters + +From here you have two options to continue: + +* Option 1: The parameters to the model are passed using a command line invocation such as: + +```bash +$ python mysoftware.py p1 p2 p3 +``` + +* Option 2: The parameters are passed through configuration files, themselves inputs in the command line invocation: + +```bash +$ python mysoftware.py config.json +``` + +Depending on which options your sofware falls under, the encapsulation steps are slightly different. diff --git a/docs/model_configuration/03-step1.md b/docs/model_configuration/03a-step1.md similarity index 62% rename from docs/model_configuration/03-step1.md rename to docs/model_configuration/03a-step1.md index dc487d2..a80a809 100644 --- a/docs/model_configuration/03-step1.md +++ b/docs/model_configuration/03a-step1.md @@ -1,30 +1,31 @@ -This step is going to create a computational environment based in the executables of your working directory and template MIC file template (`mic.yaml`) which will be placed in a `/mic` directory. This file (which will be completed in subsequent steps) contains the executable information and metadata about your model component. +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +In this step, MIC will assist you in creating a computational environment based on the executables present in your working directory and a template MIC file (`mic.yaml`) which will be placed in a `/mic` directory. This file (which will be completed in subsequent steps) contains the executable information and metadata about your software component. -### How to perform this step? +## MIC command -To run this step, you must `cd` to the folder you usually use for executing your component. Please make sure that all the information needed for executing your model exists in this folder (including data, executable scripts, etc.). You don't need to create any special folder structure, just use the same file structure you would normally use to execute your component. +To run this step, you must `cd` to the folder in which the software can be executed. Please make sure that all the information needed for executing your software exists in this folder (including data, executable scripts, etc.). You don't need to create any special folder structure, just use the same file structure you would normally use to execute your component. For example, let us consider a simple java model that reads an input file. In order to test it with MIC, we have prepared a sample input file (input.txt), which we placed in the same folder as my executable: + ```bash $ ls input.txt test_192-1.0-SNAPSHOT-jar-with-dependencies.jar ``` - -You can proceed in two different ways: +From here, you can proceed in two different ways: 1. Use MIC to create a docker image for you 2. Use a customized docker image (you may need to follow this step if you require specialized environments such as when working with Anaconda/Miniconda or a specific Linux OS) -#### Option 1: Creating an image through MIC +### Creating an image through MIC -Then, in the folder, type: +If you want to use MIC to create an image for you automatically, use the following command: ```bash $ mic pkg start ``` -!!! warning - If you want MIC to start from your own Docker image, you should use the flag `mic pkg start --image `. Do not forget to install MIC in your image with `pip install mic` so you have all the commands available. - -MIC will ask for the component name you want to use (you may use any name you want even when creating a new version of the component), and will show you a message similar to this: +MIC will ask for the component name you want to use (you may use any name you want even when creating a new version of the component). You should see the following message: ```bash Model component name: test_192 @@ -47,20 +48,28 @@ You are in a Linux environment Debian distribution We detect the following dependencies. - If you install new dependencies using `apt` or `apt-get`, remember to add them in Dockerfile mic\docker\Dockerfile -- If you install new dependencies using python. Before the step `upload` run +- If you install new dependencies using python. Before the step `upload` run: pip freeze > mic/docker/requirements.txt Please, run your Model Component. ``` -As can be seen in the message above, MIC is creating an execution environment from scratch to make sure we capture the minimum set of dependencies needed to execute a model. Since we had a java executable, MIC already selected a Java environment. If we had python files, MIC would have promted us to select which version of Python to start from. +As can be seen in the message above, MIC is creating an execution environment to make sure we capture the minimum set of dependencies needed for execution. Since we had a java executable, MIC already selected a Java environment. If we had python files, MIC would have promted us to select which version of Python to start from. !!! warning This command must **NOT** be executed on a folder already tracked by GitHub. -#### Option 2: Create your own Image +### Creating your own image + +You may already have a Docker image for your software or you may need to create one if you have complex dependencies (for instance, mix of conda and pip). MIC allows you to use your own image. --> Link to tutorial on creating docker images +For tutorials on creating docker images, see the [Docker website](https://docs.docker.com/develop/develop-images/baseimages/). + +If you are using Anaconda, the easiest method is to [export your conda environment to a `.yml` file](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment) and use a DockerFile to build the image. Below is an example of a DockerFile to build an image from the base Miniconda image. The subsequent commands create a new environment requiring Python 3.8 and all other dependencies contained in the `environment.yml` file. The last command opens up the container for the image in the right environment. + +![Example DockerFile](/model_configuration/figures/DockerFile.png) + +Then use the following command to direct MIC towards your image: ```bash $ mic pkg start --name [yourname] --image [imagename] @@ -71,6 +80,8 @@ Example: $ mic pkg start --name drought --image dsiv1.2.0 ``` +You should get the following results: + ```bash MIC has initialized the component. [Created] data: /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/data @@ -79,9 +90,6 @@ MIC has initialized the component. [Created] mic.yaml: /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/mic.yaml /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/mic.yaml created Using default tag: latest -Error response from daemon: pull access denied for dsiv1.2.0, repository does not exist or may require 'docker login': denied: requested access to the resource is denied -Using default tag: latest -Error response from daemon: pull access denied for dsiv1.2.0, repository does not exist or may require 'docker login': denied: requested access to the resource is denied Downloading the base image and building your image Step 1/4 : FROM dsiv1.2.0 @@ -105,9 +113,12 @@ Successfully tagged drought:latest $ pip3 install mic reprozip ``` -#### Expected Results +!!! warning + Do not forget to install MIC in your image with `pip install mic` for the next steps. + +## Expected Results -After executing the previous command, MIC creates a `mic` directory with three sub-directories and a MIC file (mic.yaml): +After executing the previous command, MIC creates a `mic` directory with three sub-directories and a MIC file (`mic.yaml`): - data/: folder for the input data (should be empty). - src/: folder for code and MIC wrapper (i.e., the file that executes your code). This folder should be empty at this stage. In the next step, you are going to specify how to run your model in the command line. MIC will capture all the required information automatically. @@ -137,7 +148,7 @@ Note: If you are using a conda environment, the name of the environment will sho (climate) root@4914deed9c58:/tmp/mint ``` -#### Help command +## Help command ```bash Usage: mic pkg start [OPTIONS] USER_EXECUTION_DIRECTORY diff --git a/docs/model_configuration/03b-step1.md b/docs/model_configuration/03b-step1.md new file mode 100644 index 0000000..a80a809 --- /dev/null +++ b/docs/model_configuration/03b-step1.md @@ -0,0 +1,166 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +In this step, MIC will assist you in creating a computational environment based on the executables present in your working directory and a template MIC file (`mic.yaml`) which will be placed in a `/mic` directory. This file (which will be completed in subsequent steps) contains the executable information and metadata about your software component. + +## MIC command + +To run this step, you must `cd` to the folder in which the software can be executed. Please make sure that all the information needed for executing your software exists in this folder (including data, executable scripts, etc.). You don't need to create any special folder structure, just use the same file structure you would normally use to execute your component. + +For example, let us consider a simple java model that reads an input file. In order to test it with MIC, we have prepared a sample input file (input.txt), which we placed in the same folder as my executable: + +```bash +$ ls +input.txt test_192-1.0-SNAPSHOT-jar-with-dependencies.jar +``` +From here, you can proceed in two different ways: +1. Use MIC to create a docker image for you +2. Use a customized docker image (you may need to follow this step if you require specialized environments such as when working with Anaconda/Miniconda or a specific Linux OS) + +### Creating an image through MIC + +If you want to use MIC to create an image for you automatically, use the following command: + +```bash +$ mic pkg start +``` +MIC will ask for the component name you want to use (you may use any name you want even when creating a new version of the component). You should see the following message: + +```bash +Model component name: test_192 +MIC has initialized the component. +[Created] data: C:\Users\dgarijo\Desktop\192\java_model\mic\data +[Created] docker: C:\Users\dgarijo\Desktop\192\java_model\mic\docker +[Created] src: C:\Users\dgarijo\Desktop\192\java_model\mic\src +[Created] mic.yaml: C:\Users\dgarijo\Desktop\192\java_model\mic\mic.yaml +C:\Users\dgarijo\Desktop\192\java_model\mic\mic.yaml created +You can disable the detection of dependencies using the option --no-dependencies +Dockerfile has been created: C:\Users\dgarijo\Desktop\192\java_model\mic\docker\Dockerfile +Downloading the base image and building your image +Step 1/1 : FROM mintproject/java:8 + + ---> 4950fcaa2d0d +Successfully built 4950fcaa2d0d +Successfully tagged test_192:latest + +You are in a Linux environment Debian distribution +We detect the following dependencies. + +- If you install new dependencies using `apt` or `apt-get`, remember to add them in Dockerfile mic\docker\Dockerfile +- If you install new dependencies using python. Before the step `upload` run: + +pip freeze > mic/docker/requirements.txt + +Please, run your Model Component. +``` +As can be seen in the message above, MIC is creating an execution environment to make sure we capture the minimum set of dependencies needed for execution. Since we had a java executable, MIC already selected a Java environment. If we had python files, MIC would have promted us to select which version of Python to start from. + +!!! warning + This command must **NOT** be executed on a folder already tracked by GitHub. + +### Creating your own image + +You may already have a Docker image for your software or you may need to create one if you have complex dependencies (for instance, mix of conda and pip). MIC allows you to use your own image. + +For tutorials on creating docker images, see the [Docker website](https://docs.docker.com/develop/develop-images/baseimages/). + +If you are using Anaconda, the easiest method is to [export your conda environment to a `.yml` file](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment) and use a DockerFile to build the image. Below is an example of a DockerFile to build an image from the base Miniconda image. The subsequent commands create a new environment requiring Python 3.8 and all other dependencies contained in the `environment.yml` file. The last command opens up the container for the image in the right environment. + +![Example DockerFile](/model_configuration/figures/DockerFile.png) + +Then use the following command to direct MIC towards your image: + +```bash +$ mic pkg start --name [yourname] --image [imagename] +``` +Example: + +```bash +$ mic pkg start --name drought --image dsiv1.2.0 +``` + +You should get the following results: + +```bash +MIC has initialized the component. +[Created] data: /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/data +[Created] docker: /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/docker +[Created] src: /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/src +[Created] mic.yaml: /Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/mic.yaml +/Users/deborahkhider/Documents/MINT/Drought/ForMIC2/mic/mic.yaml created +Using default tag: latest +Downloading the base image and building your image +Step 1/4 : FROM dsiv1.2.0 + + ---> 091f544d11b8 +Step 2/4 : COPY entrypoint.sh /set_umask.sh + + ---> 42aa41d755cb +Step 3/4 : RUN chmod +x /set_umask.sh + + ---> Running in 376c2e4304d6 + ---> 4270eb9cf0f0 +Step 4/4 : ENTRYPOINT ["/set_umask.sh"] + + ---> Running in 74c8ef789627 + ---> 49d4a9636391 +Successfully built 49d4a9636391 +Successfully tagged drought:latest + + You are using a custom image + You must install mic and reprozip + $ pip3 install mic reprozip +``` + +!!! warning + Do not forget to install MIC in your image with `pip install mic` for the next steps. + +## Expected Results + +After executing the previous command, MIC creates a `mic` directory with three sub-directories and a MIC file (`mic.yaml`): + +- data/: folder for the input data (should be empty). +- src/: folder for code and MIC wrapper (i.e., the file that executes your code). This folder should be empty at this stage. In the next step, you are going to specify how to run your model in the command line. MIC will capture all the required information automatically. +- docker/: It contains the required files to create the Docker Image (should not require further modifications). In later steps, MIC will populate this directory with the files that are needed to capture your computational infrastructure. + +The MIC file will have a few lines at the moment, capturing the dependencies of the current environment: + +```yaml +step: 1 +name: test_192 +docker_image: dgarijo/test_192:20.6.1 +framework: + - java8 + - mintproject/java:8 + - .jar +``` + +MIC will also redirect you to the container. The command line path should look something like this: + +```bash +root@4914deed9c58:/tmp/mint# +``` + +Note: If you are using a conda environment, the name of the environment will show up before the path: + +```bash +(climate) root@4914deed9c58:/tmp/mint +``` + +## Help command + +```bash +Usage: mic pkg start [OPTIONS] USER_EXECUTION_DIRECTORY + + This step generates a mic.yaml file and the directories (data/, src/, + docker/). It also initializes a local GitHub repository + + The argument: `model_configuration_name` is the name of the model + component you are defining in MIC + +Options: + --name TEXT Name of the model component you want for your model + --image TEXT (Optional) If you have a DockerImage, you can use it + --help Show this message and exit. +``` diff --git a/docs/model_configuration/04-step2.md b/docs/model_configuration/04a-step2.md similarity index 76% rename from docs/model_configuration/04-step2.md rename to docs/model_configuration/04a-step2.md index 03ddcb3..cc9defa 100644 --- a/docs/model_configuration/04-step2.md +++ b/docs/model_configuration/04a-step2.md @@ -1,28 +1,30 @@ -[Skip background explanation](#how-to-perform-this-step) +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. -In this step, MIC will trace the execution of your component to make sure it captures all its implicit file dependencies and its invocation command. First, you will have to make sure your model is being executed in the environment prepared by MIC, and then you will have to trace its execution. +In this step, MIC will trace the execution of your component to make sure it captures all its implicit file dependencies and its invocation command. First, you will have to make sure your software is being executed in the environment prepared by MIC, and then you will have to trace its execution. -### Testing your model in the execution environment prepared by MIC +## Testing your model in the execution environment prepared by MIC -The `mic pkg start` command created a blank execution environment that is independent of the installed dependencies of your computer; and mounted the folder you chose. In our example, we have a folder with an executable (jar) and an input file (input.txt): +The `mic pkg start` command created a blank execution environment that is independent of the installed dependencies of your computer; and mounted the folder you chose. In our Java example, we have a folder with an executable (jar) and an input file (input.txt): ![Diagram](figures/folder.png) -The next step to perform is to ensure our model works in this execution environment, and otherwise install all missing dependencies. In our case, we run the JAR executable: +The next step is to ensure our software works in this execution environment, and otherwise install all missing dependencies. In our case, we run the JAR executable: ```bash $ root@32fee4e4d205:/tmp/mint# java -jar test_192-1.0-SNAPSHOT-jar-with-dependencies.jar -i input.txt -p 1500 -o output.txt Done ``` -And we can see that an output (`output.txt`) was produced, which was thee xpected output from the model: +And we can see that an output (`output.txt`) was produced, which is the expected output from the software: ![Diagram](figures/folder_out.png) In this case, MIC already detected and prepared a Java environment because of the JAR executable, but if new dependencies are needed, they can be installed via `apt-get install ` or you favorite package manager. For example, if you use Python you may use `pip`. -Once you have tested your model, you can trace its execution +Once you have tested your model, you can trace its execution. -### How to perform this step? +## How to perform this step? Type: @@ -48,7 +50,7 @@ Trace directory .reprozip-trace exists ``` '`a`' will add the command to the existing trace; '`d`' will delete previous traces and create a new one with the new command and '`s`' will stop the tracing process. -### Expected result: +## Expected result: Your model component will take longer to run; at at the end you should see a long log message on the screen. The ending of this message should be similar to: @@ -62,9 +64,12 @@ A new folder called `.reprozip-trace` will have been created in your directory: If you are curious, you may open the `config.yaml` file inside `.reprozip-trace` folder, but no manipulation is required and you may proceed to the next step. -Note that `.reprozip-trace` is a hidden folder and you may need to enable this view on your computer. +!!! warning + After inspecting the `mic.yml` file in-between steps, CLOSE the file. -#### Help command +Note that `.reprozip-trace` is a hidden folder and you may need to enable this view on your computer. + +## Help command ```bash Usage: mic pkg trace [OPTIONS] [COMMAND]... diff --git a/docs/model_configuration/04b-step2.md b/docs/model_configuration/04b-step2.md new file mode 100644 index 0000000..4511dd1 --- /dev/null +++ b/docs/model_configuration/04b-step2.md @@ -0,0 +1,93 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +In this step, MIC will trace the execution of your component to make sure it captures all its implicit file dependencies and its invocation command. First, you will have to make sure your software is being executed in the environment prepared by MIC, and then you will have to trace its execution. + +## Testing your model in the execution environment prepared by MIC + +The `mic pkg start` command created a blank execution environment that is independent of the installed dependencies of your computer; and mounted the folder you chose. In our Python example, we have a folder with an executable (WM_climate_indices.py), an input file (GLDAS2.1_TP_2000_2018.nc), and the config file used to set the parameters for the model: + +![Folder containing all files necessary to run the model](figures/folder_config.png) + +The next step is to ensure our software works in this execution environment, and otherwise install all missing dependencies. In our case, we run the Python executable: + +```bash +$ root@32fee4e4d205:/tmp/mint# python3 WM_climate_indices.py config.json +Done +``` +Two outputs folder were created as a result of the execution: `figures` and `results`, which are expected from this software. In this case, the `results` folder contains the relevant results for a user. + +![Folder showing the output of the model run](figures/folder_config_out.png) + +In this case, MIC used an already prepared image. Other dependencies can be added along the way through `pip` or `conda`. + +Once you have tested your software, you actual step is to trace its execution. + +## How to perform this step? + +Within the contained image, type: + +```bash +$ mic pkg trace +``` + +Where `` corresponds to the way you would invoke your model, exactly as you would like to expose it. + +In the case above: + +```bash +$ (climate) root@d8826beb2c59:/tmp/mint# mic pkg trace python3 WM_climate_indices.py config.json +``` + +**What does this do?** MIC uses [Reprozip](https://www.reprozip.org/), a program to track all the system calls of your model, to automatically track down all the file and dependencies being used. The result is saved in a file called `.reprozip-trace`, which we will call `trace` for simplicity. + +If multiple commands are required for your model execution, trace each of them separately (or create a script including them and trace the invocation of the script). If you are capturing multiple traces, you will see a message like this one: + +```bash +Trace directory .reprozip-trace exists +(a)ppend run to the trace, (d)elete it or (s)top? [a/d/s] +``` +'`a`' will add the command to the existing trace; '`d`' will delete previous traces and create a new one with the new command and '`s`' will stop the tracing process. + +## Expected result: + +Your model component will take longer to run; at at the end you should see a long log message on the screen. The ending of this message should be similar to: + +```bash +Configuration file written in .reprozip-trace/config.yml +Edit that file then run the packer -- use 'reprozip pack -h' for help +``` +A new folder called `.reprozip-trace` will have been created in your directory: + +![Diagram](figures/folder_config_trace.png) + +If you are curious, you may open the `config.yaml` file inside `.reprozip-trace` folder, but no manipulation is required and you may proceed to the next step. + +!!! warning + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + + +Note that `.reprozip-trace` is a hidden folder and you may need to enable this view on your computer. + +## Help command + +```bash +Usage: mic pkg trace [OPTIONS] [COMMAND]... + + Complete the mic.yaml file with the information of the parameters and + inputs you want to expose + + MIC is going to automatically detect: - All inputs (files and + directories) used by your component and add them in the mic.yaml file. - + All parameters used by your component and add them in the configuration + file + + Usage example: mic pkg trace python main.py + ./your_program + +Options: + --continue add to the previous trace, don't replace it + --overwrite overwrite the previous trace, don't add to it + --help Show this message and exit. +``` diff --git a/docs/model_configuration/05-step3.md b/docs/model_configuration/05a-step3.md similarity index 64% rename from docs/model_configuration/05-step3.md rename to docs/model_configuration/05a-step3.md index d256e37..5359884 100644 --- a/docs/model_configuration/05-step3.md +++ b/docs/model_configuration/05a-step3.md @@ -1,10 +1,14 @@ -Models may use many parameters and input files, but we may not need to expose all of them as part of our model component. If you know individual parameters that you would like to expose in your model component, you may expose them with the `parameters` command. +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +Models may use many parameters and input files, but we may not need to expose all of them as part of our component. The goal of this step is to only expose some parameters. -If your using a `config` file to set the input variables values and file path, you still need to complete this step, entering each of the parameters you wish to expose from the file one at a time. +## How to perform this step? -### How to perform this step? +In our Java executable, we only had one parameter to expose through the command line (the parameter `p`). If you need to expose more than one parameter, you need to do so one at a time. -In our Java executable, we only had one parameter to expose through the command line (the parameter `p`). Note that you should use the `-f` option with the path of the mic file, typically under the `mic/mic.yaml` path: +Note that you can use the `-f` option with the path of the mic file, typically under the `mic/mic.yaml` path: ```bash $ mic pkg parameters -f mic/mic.yaml -n p -v 1350 @@ -15,9 +19,9 @@ The `-n` flag stands for `parameter name` and the `-v` flag stands for the defau !!! warning Parameters are case sensitive: a parameter `start_year` is different from `START_YEAR`. -### Expected result +## Expected result -If you edit the mic.yaml file, you should see that the parameter has been added. We **encourage** adding a description so the role of the parameter becomes clear: +If you open the `mic.yaml` file, you should see that the parameter has been added. We **encourage** adding a description so the role of the parameter becomes clear: ```yaml parameters: @@ -29,9 +33,7 @@ parameters: You can also add the parameter yourself to the `mic.yaml` file directly, if you prefer. -**Note**: If your parameters are declared in a configuration file, you must also complete the next step `mic pkg configs`. - -### Help command +## Help command ```bash root@32fee4e4d205:/tmp/mint# mic pkg parameters --help diff --git a/docs/model_configuration/05b-step3.md b/docs/model_configuration/05b-step3.md new file mode 100644 index 0000000..6ec5b3b --- /dev/null +++ b/docs/model_configuration/05b-step3.md @@ -0,0 +1,110 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +Models may use many parameters and input files, but we may not need to expose all of them as part of our component. The goal of this step is to only expose some parameters. + +## How to perform this step? + +Our Python executable takes the parameters value from its `config.json` file as shown below. + +```json +{ +"data" : + { + "dataset_name" : "GLDAS2.1_TP_2000_2018.nc", + "dataset_type" : "GLDAS" +}, +"output" : + { + "dynamic_name" : "False", + "path" : "./", + "fig" : "True" +}, +"index" : + { + "name" : "SPI", + "distribution" : "gamma", + "periodicity" : "monthly", + "scales" : "6", + "data_start_year" : "2015", + "data_end_year" : "2016", + "calibration_start_year" : "2000", + "calibration_end_year" : "2010" +}, +"spatial" : + { + "global" : "False", + "bounding_box": "[23,48,3,15]" +}, +"debug" : "False" +} +``` + +In this particular example, we only want to expose the `data_end_year` and `data_start_year` parameters, which corresponds to the end and start year of the simulation respectively + +```bash +$ (climate) root@d8826beb2c59:/tmp/mint# mic pkg parameters -n sim_start_year -v 2015 +Found mic.yaml in /tmp/mint/mic/mic.yaml +Adding the parameter sim_start_year, value 2015 and type int +``` + +The `-n` flag stands for `parameter name` and the `-v` flag stands for the default value you would like the parameter to have. In the case above, we defined a parameter `sim_start_year` with value `2015`. MIC detected that it was an integer. + +To add the second paramter, type: + +```bash +$ (climate) root@d8826beb2c59:/tmp/mint# mic pkg parameters -n sim_end_year -v 2016 +Found mic.yaml in /tmp/mint/mic/mic.yaml +Adding the parameter sim_end_year, value 2016 and type int +``` + +!!! warning + Parameters are case sensitive: a parameter `sim_start_year` is different from `SIM_START_YEAR`. + +## Expected result + +If you open the `mic.yaml` file, you should see that the parameter has been added. We **encourage** adding a description so the role of the parameter becomes clear: + +```yaml +parameters: + sim_start_year: + name: sim_start_year + default_value: 2015 + type: int + description: 'Start year for the simulation' + sim_end_year: + name: sim_end_year + default_value: 2016 + type: int + description: 'End year for the simulation' +``` + +You can also add the parameter yourself to the `mic.yaml` file directly following the structure above, if you prefer. + +## Help command + +```bash +root@32fee4e4d205:/tmp/mint# mic pkg parameters --help +Usage: mic pkg parameters [OPTIONS] + + Add a parameter into the MIC file (mic.yaml). + + - You must pass the MIC file (mic.yaml) as an argument using the (-f) + option; or run the command from the same directory as the MIC file + (mic.yaml) + + Usage example: + + mic pkg parameters -f --name PARAMETER_NAME --value + PARAMETER_VALUE + +Options: + -n, --name TEXT Name of the parameter [required] + -v, --value ANY TYPE (FLOAT, INTEGER, BOOL, STRING) + Default value of the parameter [required] + -d, --description TEXT Description for parameter + -o, --overwrite Overwrite an existing parameter + -f, --mic_file FILE + --help Show this message and exit. +``` diff --git a/docs/model_configuration/06-step4.md b/docs/model_configuration/06-step4.md deleted file mode 100644 index 6ab2831..0000000 --- a/docs/model_configuration/06-step4.md +++ /dev/null @@ -1,130 +0,0 @@ -[Skip background explanation](#how-to-perform-this-step) - -Models tend to be complex, and it becomes impractical to configure their parameters through the command line. Instead, many models define their parameters in one or multiple configuration files. - -Let us consider an example of a complex hydrological model ([SWAT hydrology model](https://swat.tamu.edu/)), which uses a folder with multiple inputs, including a configuration file. We would like to expose 2 parameters from this file, related to the start year of the simulation and the number of years to run it. Our `mic.yaml` file looks as follows: - -```yaml -inputs: - txtinout: - path: data/TxtInOut -parameters: - start_year: - default_value: 1991 - years: - default_value: 10 -``` - -These parameters should change the values in one of the SWAT configuration files, namely `file.cio`, which defines these parameters among others. In our example, we can find this file in the TxtInOut directory we used as input: `data/TxtInOut/file.cio`. The file looks as follows: - -``` -Master Watershed File: file.cio -Project Description: -General Input/Output section (file.cio): -5/13/2020 12:00:00 AM ARCGIS-SWAT interface AV - -General Information/Watershed Configuration: -fig.fig - 1 | NBYR : Number of years simulated - 1993 | IYR : Beginning year of simulation - 1 | IDAF : Beginning julian day of simulation - 180 | IDAL : Ending julian day of simulation -``` -As shown above, the parameters NBYR and IYR correspond to `years` and `start_year` respectively. In the following steps we will cover how to map these values into . - - -### How to feed parameters in configuration files? - -The MINT wrapper can replace the values of the parameters in their right configuration files, but it needs to know where they are. Therefore, we need to define **placeholders** in the configuration files stating which value will be replaced. - -In the SWAT example, we opened the `file.cio` configuration file and replaces: - the value `2` with ${years} -the value `1991` with ${start_year} - -The result would look as follows: - -``` -Master Watershed File: file.cio -Project Description: -General Input/Output section (file.cio): -5/13/2020 12:00:00 AM ARCGIS-SWAT interface AV - -General Information/Watershed Configuration: -fig.fig - ${years} | NBYR : Number of years simulated - ${start_year} | IYR : Beginning year of simulation - 1 | IDAF : Beginning julian day of simulation - 180 | IDAL : Ending julian day of simulation -``` - -Make sure that the names of the parameters match with the names we described in step2! Otherwise mic will not be able to replace the values accordingly: - -![Diagram](figures/06_01.png) - - -### How to perform this step? - -Once the mapping has been completed, we just need to add the file as a configuration file of the model component: - -```bash -mic pkg configs [configuration_files]... -``` - -In the example, we must run -``` -$ mic pkg configs data/TxtInOut/file.cio -Added: data/TxtInOut/file.cio as a configuration file -``` - -!!! info - The `-a` option will automatically recognize any parameter under `${parameter_name}` in the configuration files and add it to the mic.yaml file automatically. This way you can change them directly without having to perform the mapping yourself - -### Expected results - -If everything goes well, the `mic.yaml` will have been updated with a new field named `configs`: - -```yaml -inputs: - txtinout: - path: data/TxtInOut -parameters: - start_year: - default_value: 1991 - years: - default_value: 10 -step: 4 -configs: -- data/TxtInOut/file.cio -``` - -### Help command - -```bash -Usage: mic pkg configs [OPTIONS] CONFIGURATION_FILES... - - Note: If your model does not use configuration files, you can skip this - step - - Specify which parameters of your model component you want to expose from - any configuration file. - - - You must pass the MIC_FILE (mic.yaml) as an argument using the (-f) - option or run the command from the same directory as mic.yaml - - - Pass your model configuration files as arguments - - mic pkg configs -f [configuration_files]... - - If you have manually changed some parameters, the -a option will attempt - to recognize the configuration files automatically - - Example: - - mic pkg configs -f mic.yaml data/example_dir/file1.txt - data/file2.txt - -Options: - -f, --mic_file FILE - -a, --auto_param Enable automatic detection of parameters - --help Show this message and exit. -``` \ No newline at end of file diff --git a/docs/model_configuration/06a-step4.md b/docs/model_configuration/06a-step4.md new file mode 100644 index 0000000..891b964 --- /dev/null +++ b/docs/model_configuration/06a-step4.md @@ -0,0 +1,5 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +If you are not using a `config` file, you may skip this step. diff --git a/docs/model_configuration/06b-step4.md b/docs/model_configuration/06b-step4.md new file mode 100644 index 0000000..f4dcbeb --- /dev/null +++ b/docs/model_configuration/06b-step4.md @@ -0,0 +1,159 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +## Preparing your configuration file + +The MINT wrapper can replace the values of the parameters in their right configuration files, but it needs to know where they are. Therefore, we need to define **placeholders** in the configuration files stating which value will be replaced. + +In the previous step, we defined two parameters that a user may change, which corresponds to the start and end of the simulation and named them `sim_start_year` and `sim_end_year` and we assigned them default values. Now we need to indicate to MIC where these parameters are located in the configuration file. + +Let's open `config.json`: + +```json +{ +"data" : + { + "dataset_name" : "GLDAS2.1_TP_2000_2018.nc", + "dataset_type" : "GLDAS" +}, +"output" : + { + "dynamic_name" : "False", + "path" : "./", + "fig" : "True" +}, +"index" : + { + "name" : "SPI", + "distribution" : "gamma", + "periodicity" : "monthly", + "scales" : "6", + "data_start_year" : "2015", + "data_end_year" : "2016", + "calibration_start_year" : "2000", + "calibration_end_year" : "2010" +}, +"spatial" : + { + "global" : "False", + "bounding_box": "[23,48,3,15]" +}, +"debug" : "False" +} +``` + +and replace: + +- the value `2015` with ${sim_start_year} +- the value `2016` with ${sim_end_year} + +```json +{ +"data" : + { + "dataset_name" : "GLDAS2.1_TP_2000_2018.nc", + "dataset_type" : "GLDAS" +}, +"output" : + { + "dynamic_name" : "False", + "path" : "./", + "fig" : "True" +}, +"index" : + { + "name" : "SPI", + "distribution" : "gamma", + "periodicity" : "monthly", + "scales" : "6", + "data_start_year" : "${sim_start_year}", + "data_end_year" : "${sim_end_year}", + "calibration_start_year" : "2000", + "calibration_end_year" : "2010" +}, +"spatial" : + { + "global" : "False", + "bounding_box": "[23,48,3,15]" +}, +"debug" : "False" +} +``` + +Make sure that the names of the parameters match with the names we described in step2! Otherwise MIC will not be able to replace the values accordingly: + +![Diagram](figures/config_comp.png) + + +## How to perform this step? + +Once the mapping has been completed, we just need to add the file as a configuration file of the model component: + +```bash +mic pkg configs [configuration_files]... +``` + +In the example, we must run +``` +$ (climate) root@d8826beb2c59:/tmp/mint# mic pkg configs config.json +Found mic.yaml in /tmp/mint/mic/mic.yaml +Added: /tmp/mint/config.json as a configuration file +``` + +!!! info + The `-a` option will automatically recognize any parameter under `${parameter_name}` in the configuration files and add it to the mic.yaml file automatically. This way you can change them directly without having to perform the mapping yourself in step 3. + +## Expected results + +The `mic.yaml` will have been updated with a new field named `configs`: + +```yaml +parameters: + sim_start_year: + name: sim_start_year + default_value: 2015 + type: int + description: Start year for the simulation + sim_end_year: + name: sim_end_year + default_value: 2016 + type: int + description: End year for the simulation +configs: + config_json: + path: config.json + format: json +``` + +## Help command + +```bash +Usage: mic pkg configs [OPTIONS] CONFIGURATION_FILES... + + Note: If your model does not use configuration files, you can skip this + step + + Specify which parameters of your model component you want to expose from + any configuration file. + + - You must pass the MIC_FILE (mic.yaml) as an argument using the (-f) + option or run the command from the same directory as mic.yaml + + - Pass your model configuration files as arguments + + mic pkg configs -f [configuration_files]... + + If you have manually changed some parameters, the -a option will attempt + to recognize the configuration files automatically + + Example: + + mic pkg configs -f mic.yaml data/example_dir/file1.txt + data/file2.txt + +Options: + -f, --mic_file FILE + -a, --auto_param Enable automatic detection of parameters + --help Show this message and exit. +``` diff --git a/docs/model_configuration/07-step5.md b/docs/model_configuration/07a-step5.md similarity index 78% rename from docs/model_configuration/07-step5.md rename to docs/model_configuration/07a-step5.md index c5468ef..2b95605 100644 --- a/docs/model_configuration/07-step5.md +++ b/docs/model_configuration/07a-step5.md @@ -1,12 +1,16 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + After tracing your execution, MIC has recorded all the input/output files that are needed and produced by your model component. However, we need to identify which of these input types you want to allow others to change. This step will help you describe the input data exposed in your model component. All inputs will be added by MIC in the MIC file. -Note that even though we are referring to concrete files in this step, these represent placeholders that will be expected as input for running your model component. For example, if your model uses a `precip.csv` file with precipitation data and you expose it, your model component will expect a `precip` file as input in order to run; independently of its name. +Note that even though we are referring to concrete files in this step, these represent placeholders that will be expected as inputs for running your model component. For example, if your model uses a `precip.csv` file with precipitation data and you expose it, your model component will expect a `precip` file as input in order to run; independently of its name. -### How to perform this step? -Just type the following command: `mic pkg inputs`. MIC will ask you about whether the inputs and outputs used and produced are code or not, and based on that MIC will add the appropriate inputs into the `mic.yaml` file. For our simple Java example, this is the result of the command execution: +## How to perform this step? +Just type the following command: `mic pkg inputs`. MIC will ask you about whether the inputs and outputs used and produced are code, and based on that MIC will add the appropriate inputs into the `mic.yaml` file. For our simple Java example, this is the result of the command execution: ```bash -$ mic pkg inputs +$ mic pkg inputs Automatically found mic.yaml in /tmp/mint/mic/mic.yaml Detecting the data of your model using the information obtained by the `trace` command. Creating the inputs. @@ -25,7 +29,7 @@ MIC is going to detect the outputs of your model using the information obtained For more information, you can type. mic pkg outputs --help ``` -### Expected results +## Expected results If we inspect the `mic.yaml` file, we now see that the inputs and executable files have been correctly annotated: ```yaml @@ -41,9 +45,7 @@ code_files: If you detect that an input is missing, you can always add it through the `inputs` command. For example, by doing `mic pkg inputs `, where represents the path to an input you would like to expose. Added files must exist, or the program will issue an error. -If your input is defined in a configuration file, you may use the same method as with parameters for MIC to replace its value in the configuration file. For example, if the model in the example above used a configuration file, the path to `input.txt` could have been changed with `${input.txt}` - -### Help command +## Help command ```bash Usage: mic pkg inputs [OPTIONS] [CUSTOM_INPUTS]... diff --git a/docs/model_configuration/07b-step5.md b/docs/model_configuration/07b-step5.md new file mode 100644 index 0000000..495584f --- /dev/null +++ b/docs/model_configuration/07b-step5.md @@ -0,0 +1,133 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +After tracing your execution, MIC has recorded all the input/output files that are needed and produced by your model component. However, we need to identify which of these input types you want to allow others to change. This step will help you describe the input data exposed in your model component. All inputs will be added by MIC in the MIC file. + +Note that even though we are referring to concrete files in this step, these represent placeholders that will be expected as inputs for running your model component. Our example uses `GLDAS2.1_TP_2000_2018.nc` file with precipitation and temperature data. To expose it, your model component will expect a `input_nc` file as input in order to run; independently of its name. + +## How to perform this step? + +First you need to modify your configuration file in a similar fashion as you did for the parameters for MIC to replace its values in the file. In this particular example, we replaced the `GLDAS2.1_TP_2000_2018.nc` value with `${input_nc}` as shown below: + +```json +{ +"data" : + { + "dataset_name" : "${input_nc}", + "dataset_type" : "GLDAS" +}, +"output" : + { + "dynamic_name" : "False", + "path" : "./", + "fig" : "True" +}, +"index" : + { + "name" : "SPI", + "distribution" : "gamma", + "periodicity" : "monthly", + "scales" : "6", + "data_start_year" : "${sim_start_year}", + "data_end_year" : "${sim_end_year}", + "calibration_start_year" : "2000", + "calibration_end_year" : "2010" +}, +"spatial" : + { + "global" : "False", + "bounding_box": "[23,48,3,15]" +}, +"debug" : "False" +} +``` +Then type the following command: `mic pkg inputs`. MIC will ask you about whether the inputs and outputs used and produced are code, and based on that MIC will add the appropriate inputs into the `mic.yaml` file. + +```bash +$ mic pkg inputs +``` + +For our example: + +```bash +$ (climate) root@d8826beb2c59:/tmp/mint# mic pkg inputs +Found mic.yaml in /tmp/mint/mic/mic.yaml +Detecting the data of your model using the information obtained by the `trace` command. +Creating the inputs. +If the data is a directory, MIC is going to compress in a zipfile. +.py +Adding WM_climate_indices.py as executable +Input results added +Input GLDAS2.1_TP_2000_2018.nc added +Input figures added +Success +The inputs of model component are available in the mic directory. +You model component has 3 inputs +The next step is `mic pkg outputs` +MIC is going to detect the outputs of your model using the information obtained by the `trace` command. +For more information, you can type. +mic pkg outputs --help +``` + + +## Expected results + +Note that in this case, MIC incorrectly detected the output folders as inputs. We needed to delete them from the `mic.yml` file and replace the name of the netcdf file with `input_nc`: + +```yaml +parameters: + sim_start_year: + name: sim_start_year + default_value: 2015 + type: int + description: Start year for the simulation + sim_end_year: + name: sim_end_year + default_value: 2016 + type: int + description: End year for the simulation +configs: + config_json: + path: config.json + format: json +inputs: + input_nc: + path: GLDAS2.1_TP_2000_2018.nc + format: nc +code_files: + wm_climate_indices_py: + path: WM_climate_indices.py + format: py +``` + +If you detect that an input is missing, you can always add it through the `inputs` command. For example, by doing `mic pkg inputs `, where represents the path to an input you would like to expose. Added files must exist, or the program will issue an error. + +## Help command + +```bash +Usage: mic pkg inputs [OPTIONS] [CUSTOM_INPUTS]... + + Describe the inputs of your model using the information obtained by the + `trace` command. To identify which inputs have been automatically + detected, execute `mic pkg inputs -f mic/mic.yaml` and then + inspect the mic.yaml file + + - You must pass the MIC_FILE (mic.yaml) as an argument using the (-f) + option or run the command from the same directory as mic.yaml + + - Identify undetected files in or directories in mic.yaml and add them as + arguments to the `inputs` command + + mic pkg inputs -f [undetected files]... + + Usage example: + + mic pkg inputs -f mic/mic.yaml input.txt inputs_directory + + + +Options: + -f, --mic_file FILE + --help Show this message and exit. +``` diff --git a/docs/model_configuration/08-step6.md b/docs/model_configuration/08a-step6.md similarity index 91% rename from docs/model_configuration/08-step6.md rename to docs/model_configuration/08a-step6.md index 7437e22..492f5f0 100644 --- a/docs/model_configuration/08-step6.md +++ b/docs/model_configuration/08a-step6.md @@ -1,5 +1,11 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + Similarly to what we did with inputs, we must identify which outputs to expose in our model component. MIC will detect some of them automatically based on what we entered in previous steps, avoiding redundant questions. +## How to perform this step? + For example, for our Java simple model, the command `mic pkg outputs` does most of the work for us: ```bash @@ -15,7 +21,7 @@ For more information, you can type. mic pkg wrapper --help ``` -### Expected results +## Expected results If we inspect the `mic.yaml` file, we see that the output has been added correctly: ```yaml @@ -27,9 +33,9 @@ outputs: If you detect that an output is missing from the `mic.yaml` file, you can always add it through the `outputs` command. For example, by doing `mic pkg outputs `, where represents the path to an output you would like to expose. Added files must exist, or the program will issue an error. MIC will use this information to confirm that the output files are generated when testing the component. -If your code generates intermediate outputs that do not need to be exposed to a user, you can remove them from the `mic.yaml` file. +If your code generates intermediate outputs that do not need to be exposed to a user, you can remove them from the `mic.yaml` file. -### Help command +## Help command ```bash Usage: mic pkg outputs [OPTIONS] [CUSTOM_OUTPUTS]... diff --git a/docs/model_configuration/08b-step6.md b/docs/model_configuration/08b-step6.md new file mode 100644 index 0000000..4f36aee --- /dev/null +++ b/docs/model_configuration/08b-step6.md @@ -0,0 +1,94 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +Similarly to what we did with inputs, we must identify which outputs to expose in our model component. MIC will detect some of them automatically based on what we entered in previous steps, avoiding redundant questions. + +## How to perform this step? + +```bash +$ mic pkg outputs +``` +For our Python example: + +```bash +$ (climate) root@d8826beb2c59:/tmp/mint# mic pkg outputs +Found mic.yaml in /tmp/mint/mic/mic.yaml +Detecting the output of your model using the information obtained by the `trace` command. +Output added: spi_t15.jpeg +Output added: spi_t8.jpeg +Output added: spi_t18.jpeg +Output added: spi_t9.jpeg +Output added: spi_t21.jpeg +Output added: spi_t0.jpeg +Output added: spi_t10.jpeg +Output added: results.nc +Output added: results.mp4 +Output added: spi_t4.jpeg +Output added: spi_t11.jpeg +Output added: spi_t13.jpeg +Output added: spi_t22.jpeg +Output added: spi_t7.jpeg +Output added: spi_t5.jpeg +Output added: spi_t19.jpeg +Output added: spi_t17.jpeg +Output added: spi_t1.jpeg +Output added: spi_t6.jpeg +Output added: spi_t14.jpeg +Output added: spi_t3.jpeg +Output added: spi_t12.jpeg +Output added: spi_t23.jpeg +Output added: spi_t16.jpeg +Output added: spi_t20.jpeg +Output added: spi_t2.jpeg +Success +You model component has 26 outputs +The next step is `mic pkg wrapper` +MIC is going to generate the directory structure and commands required to run your model. +For more information, you can type. +mic pkg wrapper --help +``` + +## Expected results +Outputs should be identified as followed in `mic.yaml`: + +```yaml +outputs: + results_nc: + path: results/results.nc + format: nc + results_mp4: + path: results/results.mp4 + format: mp4 +``` + +If you detect that an output is missing from the `mic.yaml` file, you can always add it through the `outputs` command. For example, by doing `mic pkg outputs `, where represents the path to an output you would like to expose. Added files must exist, or the program will issue an error. MIC will use this information to confirm that the output files are generated when testing the component. + +If your code generates intermediate outputs that do not need to be exposed to a user, you can remove them from the `mic.yaml` file. This is the case in our Python example where intermediate `jpeg` files are generated to produce the final `mp4` movie. + +## Help command + +```bash +Usage: mic pkg outputs [OPTIONS] [CUSTOM_OUTPUTS]... + + Describe the outputs of your model using the information obtained by the + `trace` command. To identify which inputs have been automatically + detected, execute `mic pkg outputs -f mic/mic.yaml` and then + inspect the mic.yaml file + + - You must pass the MIC_FILE (mic.yaml) as an argument using the (-f) + option; or run the command from the same directory as mic.yaml + + - Identify undetected files or directories in the mic.yaml file and pass + them as as arguments to the command + + mic pkg outputs -f [undetected files]... + + Example: + + mic pkg outputs -f mic/mic.yaml output.txt outputs_directory + +Options: + -f, --mic_file FILE + --help Show this message and exit. +``` diff --git a/docs/model_configuration/09-step7.md b/docs/model_configuration/09a-step7.md similarity index 77% rename from docs/model_configuration/09-step7.md rename to docs/model_configuration/09a-step7.md index bfb9bed..2745e4d 100644 --- a/docs/model_configuration/09-step7.md +++ b/docs/model_configuration/09a-step7.md @@ -1,16 +1,20 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + The MINT Wrapper is a plain text file that contains a series of commands needed for executing a model. The MINT Wrapper does the following tasks: - Copy and extract your inputs in the src directory -- Feed the parameter values to your models. +- Feed the parameter values to your software. - Detect errors on execution time. The commands in the MINT Wrapper are a mixture of commands we would normally type ourselves on the command line (such as ls or cp). If this sequence of commands is needed to execute your model, we need to preserve it in your model component. Remember that anything you can run normally on the command line can be put into a script with equivalent functionality. !!! info - Many models have graphical interfaces for data preparation purposes and set up. However, the scope of this effort is making your model available on any infrastructure. Cloud servers and supercomputers don’t usually provide graphical interfaces, and therefore we cannot assume a graphical interface to be available. It is a good engineering practice to deliver a component that can be used without a graphical interface. + Many models have graphical interfaces for data preparation purposes and set up. However, the scope of this effort is making your software available on any infrastructure. Cloud servers and supercomputers don’t usually provide graphical interfaces, and therefore we cannot assume a graphical interface to be available. It is a good engineering practice to deliver a component that can be used without a graphical interface. -### How to perform this step? +## How to perform this step? Just type `mic pkg wrapper` and MIC will attempt to generate the wrapper script automatically. For example, in our simple Java model: ```bash @@ -26,16 +30,13 @@ The command run is going to create a new directory (execution directory), and MI For more information, you can type. mic pkg run --help ``` -### Expected results +## Expected results If MIC has successfully created the wrapper and drafted an executable automatically, you should see the following: 1. The data has been moved to the data folder -2. Model code moved to src. If you are using a `config` file, this should also have been moved to the src folder. -3. `output.sh`, `input.sh` and `run` have been created. - -Next, you should ensure that the executable is correct. If using `config` file, you should not have to change anything in `run`. +2. `output.sh`, `input.sh` and `run` have been created. -Otherwise you need to update the command line execution to reflect the parameters. From our example, the executable at `/tmp/mint/mic/src/run` looks like follows: +Next, you should ensure that the executable is correct. Otherwise you need to update the command line execution to reflect the parameters. From our example, the executable at `/tmp/mint/mic/src/run` looks like follows: ```bash #!/bin/bash @@ -92,14 +93,7 @@ inputs: format: txt ``` -If you are using a `config` file, then the command: - -``` -python3 example.py config.json -``` -is correct. - -### Help command +## Help command ```bash Usage: mic pkg wrapper [OPTIONS] diff --git a/docs/model_configuration/09b-step7.md b/docs/model_configuration/09b-step7.md new file mode 100644 index 0000000..9c33cbe --- /dev/null +++ b/docs/model_configuration/09b-step7.md @@ -0,0 +1,88 @@ +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. + +The MINT Wrapper is a plain text file that contains a series of commands needed for executing a model. +The MINT Wrapper does the following tasks: + +- Copy and extract your inputs in the src directory +- Feed the parameter values to your software. +- Detect errors on execution time. + +The commands in the MINT Wrapper are a mixture of commands we would normally type ourselves on the command line (such as ls or cp). If this sequence of commands is needed to execute your model, we need to preserve it in your model component. Remember that anything you can run normally on the command line can be put into a script with equivalent functionality. + +!!! info + Many models have graphical interfaces for data preparation purposes and set up. However, the scope of this effort is making your software available on any infrastructure. Cloud servers and supercomputers don’t usually provide graphical interfaces, and therefore we cannot assume a graphical interface to be available. It is a good engineering practice to deliver a component that can be used without a graphical interface. + +## How to perform this step? +Just type `mic pkg wrapper` and MIC will attempt to generate the wrapper script automatically. For example, in our Python software: + +```bash +$ (climate) root@2417929e507e:/tmp/mint# mic pkg wrapper + +``` +## Expected results +If MIC has successfully created the wrapper and drafted an executable automatically, you should see the following: + +1. The data has been moved to the data folder +2. The configuration file should have been moved as well +2. `output.sh`, `input.sh` and `run` have been created. + +Next, you should ensure that the executable is correct. From our example, the executable at `/tmp/mint/mic/src/run` looks like follows: + +```bash +#!/bin/bash +set +x +set -e +. .colors.sh + +BASEDIR=$PWD +. $BASEDIR/io.sh 1 2 2 "$@" +CURDIR=`pwd` +## INPUTS VARIABLES +input_nc=${INPUTS1} + + +## PARAMETERS VARIABLES +sim_start_year=${PARAMS1} +sim_end_year=${PARAMS2} + + +set -xe + +####### WRITE YOUR INVOCATION LINE AFTER THIS COMMENT + + +pushd . +python3 WM_climate_indices.py config.json +popd + +set +x +echo -e "$(c G)[success] The model has exited with code SUCCESS" +####### WRITE YOUR INVOCATION LINE BEFORE THIS COMMENT +cd $BASEDIR +. $BASEDIR/output.sh +``` +Make sure that the input files/parameters names correspond to the one you have assigned in the configuration file and that the invocation command is correct. Since you have already matched the names of the input files and parameters in step 4, you do not need to make further modification at this point. + +## Help command +```bash +Usage: mic pkg wrapper [OPTIONS] + + Generates the MIC Wrapper:a directory structure and commands required to + run your model component using the information gathered from previous + steps + + - You must pass the MIC_FILE (mic.yaml) as an argument using the (-f) + option or run the command from the same directory as mic.yaml + + mic pkg wrapper -f + + Example: + + mic pkg wrapper -f mic/mic.yaml + +Options: + -f, --mic_file FILE + --help Show this message and exit. +``` diff --git a/docs/model_configuration/10-step8.md b/docs/model_configuration/10a-step8.md similarity index 80% rename from docs/model_configuration/10-step8.md rename to docs/model_configuration/10a-step8.md index 9b85819..843bfea 100644 --- a/docs/model_configuration/10-step8.md +++ b/docs/model_configuration/10a-step8.md @@ -1,6 +1,10 @@ -When you have finished the modifications of the MIC Wrapper. You can run step 8. This step will attempt to run your model component using the MIC wrapper and will validate that the expected output files are successfully generated. +!!! warning + DO NOT CLOSE the terminal in-between each step. + After inspecting the `mic.yml` file in-between steps, CLOSE the file. -### How to perform this step? +This step will attempt to run your model component using the MIC wrapper and will validate that the expected output files are successfully generated. + +## How to perform this step? Type `mic pkg run`. Below is the result obtained with our simple Java model: @@ -30,13 +34,12 @@ If the model is ok, type "exit" to go back to your computer IMPORTANT: type "exit" and then upload your Model Component ``` -MIC will ask permission to create a folder, which will be placed under `executions` in the `mic` folder. MIC tested whether the execution of the model component finished successfully and whether the expected output (`output.txt`) was generated. After inspecing the result in `tmp/mint/mic/executions/06_25_04_47_52` and confirming that the output is correct, we conclude that the component is ready for publication. We type `exit` to exit the MIC environment. - +MIC will ask permission to create a folder, which will be placed under `executions` in the `mic` folder. MIC tested whether the execution of the model component finished successfully and whether the expected output (`output.txt`) was generated. After inspecting the result in `tmp/mint/mic/executions/06_25_04_47_52` and confirming that the output is correct, we conclude that the component is ready for publication. -### Expected result +## Expected result A successful test of the candidate model component. **After you are done, type `exit` to exit the MIC container and return to your desktop for the final step**. -### Help command +## Help command ```bash Usage: mic pkg run [OPTIONS] @@ -54,4 +57,4 @@ Usage: mic pkg run [OPTIONS] Options: -f, --mic_file FILE --help Show this message and exit. -``` \ No newline at end of file +``` diff --git a/docs/model_configuration/10b-step8.md b/docs/model_configuration/10b-step8.md new file mode 100644 index 0000000..cbc0e2a --- /dev/null +++ b/docs/model_configuration/10b-step8.md @@ -0,0 +1,60 @@ +This step will attempt to run your model component using the MIC wrapper and will validate that the expected output files are successfully generated. + +## How to perform this step? + +Type `mic pkg run`. Below is the result obtained with our simple Java model: + +```bash +$ (climate) root@2417929e507e:/tmp/mint# mic pkg run +Found mic.yaml in /tmp/mint/mic/mic.yaml +MIC needs to create new directory mic/executions/04_15_01_50_06 to run the model component Do you want to continue [Y/n]: Y +Create a execution directory /tmp/mint/mic/executions/04_15_01_50_06 +Copying the inputs +Source: /tmp/mint/mic/data +Destination: /tmp/mint/mic/executions/04_15_01_50_06/src +Running +./run -i1 /tmp/mint/mic/executions/04_15_01_50_06/src/GLDAS2.1_TP_2000_2018.nc -o1 results_mp4.mp4 -o2 results_nc.nc -p1 2015 -p2 2016 ++ pushd . +/tmp/mint/mic/executions/04_15_01_50_06/src /tmp/mint/mic/executions/04_15_01_50_06/src ++ python3 WM_climate_indices.py config.json +2021-04-15 01:51:04 WARNING IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1500, 1000) to (1504, 1008) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). +[swscaler @ 0x6397a40] Warning: data is not aligned! This can lead to a speed loss ++ popd +/tmp/mint/mic/executions/04_15_01_50_06/src ++ set +x +[success] The model has exited with code SUCCESS +[success] The model has generated the output results/results.mp4 +[success] The model has generated the output results/results.nc +Success +You can see the result at /tmp/mint/mic/executions/04_15_01_50_06 +The next step is `mic pkg upload` +The step is going to upload the MIC Wrapper to GitHub, the DockerImage on DockerHub and the Model Configuration on the MINT Model Catalog +You model has passed all the tests. Please, review the outputs files. +If the model is ok, type "exit" to go back to your computer +IMPORTANT: type "exit" and then upload your Model Component +``` + +MIC will ask permission to create a folder, which will be placed under `executions` in the `mic` folder. MIC tested whether the execution of the model component finished successfully and whether the expected outputs (`results.nc` and `results.mp4`) were generated. After inspecting the result in `tmp/mint/mic/executions/04_15_01_50_06` and confirming that the output is correct, we conclude that the component is ready for publication. + +## Expected result +A successful test of the candidate model component. **After you are done, type `exit` to exit the MIC container and return to your desktop for the final step**. + +## Help command +```bash +Usage: mic pkg run [OPTIONS] + + This step will test the model component you created in previous steps. + + - You must pass the MIC_FILE (mic.yaml) using the option (-f) or run the + command from the same directory as mic.yaml + + mic pkg run -f + + Example: + + mic pkg wrapper -f mic/mic.yaml + +Options: + -f, --mic_file FILE + --help Show this message and exit. +``` diff --git a/docs/model_configuration/11-step9.md b/docs/model_configuration/11a-step9.md similarity index 95% rename from docs/model_configuration/11-step9.md rename to docs/model_configuration/11a-step9.md index 734a8f6..c810ac1 100644 --- a/docs/model_configuration/11-step9.md +++ b/docs/model_configuration/11a-step9.md @@ -1,10 +1,10 @@ Now that we have tested our model component, it's time do a snapshot of the mic wrapper, the docker image and save the model metadata in MINT. -MIC assumes that appropriate GitHub credentials, Docker login and MINT metadata credentials have been set up with the `mic credentials` command. +MIC assumes that appropriate GitHub credentials, Docker login and MINT metadata credentials have been set up with the [`mic credentials` command](/commands/configure/). ### How to perform this step? -If you are still on the environment MIC created, type `exit`. It is usually easy to identify whether you are on the MIC environment or not, as you will see `root@:/tmp/mint#` in your console. +If you are still on the environment MIC created, type `exit`. It is usually easy to identify whether you are on the MIC environment or not, as you will see `root@:/tmp/mint#` in your console. Once exited, type `mic pkg upload`: @@ -46,7 +46,7 @@ Select enter the number of version to use [1]: 1 ``` In this case, we selected a model that already exists (`test_192`). If the model component does not have a model associated yet, MIC will prompt you to describe its name and version. -### Expected results +### Expected results MIC will create a GitHub repository for your component (if the repository exist, it will create a new release in it); a Docker image and a MINT model configuration entry. For example, for our Java component, MIC automatically created: @@ -85,4 +85,4 @@ Options: -f, --mic_file FILE -p, --profile --help Show this message and exit. -``` \ No newline at end of file +``` diff --git a/docs/model_configuration/12-end-to-end-example.md b/docs/model_configuration/12a-end-to-end-example.md similarity index 100% rename from docs/model_configuration/12-end-to-end-example.md rename to docs/model_configuration/12a-end-to-end-example.md diff --git a/docs/model_configuration/12b-end-to-end-example.md b/docs/model_configuration/12b-end-to-end-example.md new file mode 100644 index 0000000..62cdc61 --- /dev/null +++ b/docs/model_configuration/12b-end-to-end-example.md @@ -0,0 +1,63 @@ +Here we summarize the nine previous steps with the Python example executable we have used through the documentation. + +Our component takes as input a netcdf file and produces a netcdf file and a mp4 movie as a result. Parameters are passed through a configuration file. This is what the component folder looks like after we execute the first MIC command: + +![Diagram](figures/folder_config.png) + +The sequence of commands required for encapsulating the model are: + +1. Start: `mic pkg start` +2. Trace the execution command: `mic pkg trace python3 WM_climate_indices.py config.json` +3. Expose the start/end years of the simulation with default values of 2015 and 2016 respectively: + + - `mic pkg parameters -n sim_start_year -v 2015` + - `mic pkg parameters -n sim_end_year -v 2016` + +4. Edit the configuration file to match the parameters name to their values in the `mic.yaml` file: + +![config/yaml comparison](figures/config_comp.png) + +5. Select inputs to expose + + - Edit the configuration file to identify which input files should be added: + +```json +{ +"data" : + { + "dataset_name" : "${input_nc}", + "dataset_type" : "GLDAS" +}, +"output" : + { + "dynamic_name" : "False", + "path" : "./", + "fig" : "True" +}, +"index" : + { + "name" : "SPI", + "distribution" : "gamma", + "periodicity" : "monthly", + "scales" : "6", + "data_start_year" : "${sim_start_year}", + "data_end_year" : "${sim_end_year}", + "calibration_start_year" : "2000", + "calibration_end_year" : "2010" +}, +"spatial" : + { + "global" : "False", + "bounding_box": "[23,48,3,15]" +}, +"debug" : "False" +} +``` + - `mic pkg inputs` + +6. Select outputs to expose: `mic pkg outputs` +7. Create wrapper: `mic pkg wrapper`. +8. Run wrapper and verify results: `mic pkg run`. Once you are done, **exit the MIC environment: Type `exit`** +9. Upload: `mic pkg upload` + +Done! diff --git a/docs/model_configuration/figures/DockerFile.png b/docs/model_configuration/figures/DockerFile.png new file mode 100644 index 0000000..1bab3d5 Binary files /dev/null and b/docs/model_configuration/figures/DockerFile.png differ diff --git a/docs/model_configuration/figures/config.png b/docs/model_configuration/figures/config.png new file mode 100644 index 0000000..9ef8484 Binary files /dev/null and b/docs/model_configuration/figures/config.png differ diff --git a/docs/model_configuration/figures/config_comp.png b/docs/model_configuration/figures/config_comp.png new file mode 100644 index 0000000..affb6c2 Binary files /dev/null and b/docs/model_configuration/figures/config_comp.png differ diff --git a/docs/model_configuration/figures/folder_config.png b/docs/model_configuration/figures/folder_config.png new file mode 100644 index 0000000..ea87683 Binary files /dev/null and b/docs/model_configuration/figures/folder_config.png differ diff --git a/docs/model_configuration/figures/folder_config_out.png b/docs/model_configuration/figures/folder_config_out.png new file mode 100644 index 0000000..78b358f Binary files /dev/null and b/docs/model_configuration/figures/folder_config_out.png differ diff --git a/docs/model_configuration/figures/folder_config_trace.png b/docs/model_configuration/figures/folder_config_trace.png new file mode 100644 index 0000000..542ffd7 Binary files /dev/null and b/docs/model_configuration/figures/folder_config_trace.png differ diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..b1444a6 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,96 @@ +# Overview of the MIC process + +MIC assists a user in encapsulating their software in 9 easy steps summarize in the figure below. + +![Diagram](/model_configuration/figures/overview_01.png) + +Below is a summary of the steps. This is not meant as a user guide. See the [detailed instructions](/model_configuration/01-DockerGithub) for a step-by-step guide. + +## The MIC process + +### Step 1: Start a MIC encapsulation component +In this step, MIC will automatically detect the execution environment (e.g., python executables requiring a Python environment) and create a template *MIC file* that will be modified in subsequent steps with information about the model component. + +**Expected outcomes of this step**: A folder (`mic`) with a placeholder MIC file (`mic.yaml`) in that folder. + +### Step 2: Trace your model execution +In this step, you will install any missing dependency needed to execute the software and trace its execution. The execution traces present the order of the statements that are touched during the execution of test cases. + +**Expected outcome of this step**: The tracing process will cause the execution to be slower than normal. At the end of the process,a `.reprozip` trace file will be added to the folder if the execution succeeded. + +### Step 3: Select input parameters to expose +In this step, you will expose specific input parameters to your software component. You can expose as little or as many parameters as needed. It entirely depends on the application and who the expected end user of the component is. Note that it is possible to create different component of the same executable aimed at different audiences. + +**Expected outcome of this step**: The MIC file will be automatically edited to include information about the input parameters that you specified for your component. + +### Step 4: Pre-populate the software configuration files (optional) +Complex software packages (e.g. modeling software) often use configuration files to pass parameters to the executable rather than passing the parameters through a command line invocation. If this is the case, this additional step is needed to flag the input parameters in the config files that will be exposed to a user. Even if you are using a config file, step 3 is required. + +**Expected results after completing this step**: MIC will update the MIC file with pointers to the configuration files that set the parameters that are inputs to the component. + +### Step 5: Specify input files to expose +Although MIC will automatically detect most of the inputs your component used for execution, we need to determine which of these files should be exposed to a user (much like we did for the parameters). + +**Expected outcome of this step**: The MIC file will be automatically edited to include information about the input files. + +### Step 6: Specify outputs to expose +Although MIC will automatically detect most of the outputs your component generated after execution, we need to determine which of these outputs are relevant for a user. + +**Expected outcome of this step**: The MIC file will be automatically edited to include information about the outputs. + +### Step 7: Prepare MIC wrapper +In this step, MIC will create three subfolders (`data`, `src` and `docker`) in the mic folder and *MIC wrapper* that indicates how the component invocation from Step 2 corresponds to the functions implemented in your model software. + +**Expected outcome of this step**: The MIC Wrapper will automatically set up three files (run, io.sh, and output.sh) that capture how the component has been run. They can be found in the /src folder of the MIC directory. + +### Step 8: Run your model through the MIC wrapper +Use the wrapper generated in Step 7 to execute your model. + +**Expected outcome of this step**: A successful test run and encapsulation of your software. + +### Step 9: Upload the model component software and image +MIC will upload the MIC Wrapper and your software in GitHub. MIC will also upload the component image in DockerHub and upload your software in the MINT Model Catalog, which will make it accessible by others through MINT services and interfaces to be run in their own local hosts and servers. This will give them unique identifiers that represent the snapshot of the software that you wanted to encapsulate so that any future updates to your software or your software component can be distinct from each other. + +**Expected results after completing this step**: Your model component wrapper will be uploaded to Github, and your model component image to DockerHub. Both will receive a tag and will be versioned, and will be archived so they are available to anyone anywhere. MINT will have an entry for your component in the MINT Model Catalog, which will be accessible through your browser. Anyone using [DAME](dame-cli.readthedocs.io/) can run your component with their own data. Anyone using MINT can run your component with appropriate datasets available in MINT. + + +## Using MIC to Create Model Components + +The `pkg` command in MIC guides you through nine steps to create a model component that encapsulates your model code and makes it available so others can easily run it in their own local environments. To accomplish this, MIC relies on state-of-the-art virtualization tools (including Docker), software engineering best practices, and semantic reasoning for validation. + +MIC creates: 1) a MIC Directory, 2) a MIC File, 3) a MIC Wrapper, 4) a component virtualization image, 5) archival versions of your model code, MIC directories and files, and image in GitHub and DockerHub, and 6) a model component entry in the MINT Model Catalog. + +Once you have MIC installed, you can type `mic pkg --help` to see the following message that lists all the steps: + +```bash +Usage: mic pkg [OPTIONS] COMMAND [ARGS]... + + Command to pkg your model component + +Options: + --help Show this message and exit. + +Commands: + start Create a Linux environment to run your model. The working + directory selected must contain all the files required for the + execution of your model + + trace Trace any command line and extract the information about your + model execution + + parameters Expose parameters in the mic.yaml file + configs Select configuration file(s) for your model (if applicable) + inputs Expose model inputs into the mic.yaml file + outputs Expose model outputs in the mic.yaml file + wrapper Generate the directory structure and commands required to run + your model component using the information from the previous + steps + + run Run your model component with the MIC Wrapper generated in the + previous step + + upload Upload your code to GitHub, DockerHub and your + model component on the MINT Model Catalog. +``` + +You have to follow the commands in order to successfully upload your component. If you want to know more about a specific step, just do `mic pkg --help`, where `` represents one of the steps listed above (e.g., `trace`). diff --git a/mkdocs.yml b/mkdocs.yml index 385e64d..e8ecc89 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,24 +1,38 @@ site_name: Model Insertion Checker (MIC) nav: - Home: index.md + - Installation: installation.md + - MIC Overview: overview.md + - 'Instructions': + - 'Setting up Docker and GitHub': 'model_configuration/01-DockerGitHub.md' + - 'Preparing your executable': 'model_configuration/02-Preparing your executable.md' + - 'Option 1: Parameters values passed through command line': + - '1. Starting a new component': 'model_configuration/03a-step1.md' + - '2. Tracing execution': 'model_configuration/04a-step2.md' + - '3. Exposing parameters': 'model_configuration/05a-step3.md' + - '4. Editing component configuration files [skipped]': 'model_configuration/06a-step4.md' + - '5. Exposing input files': 'model_configuration/07a-step5.md' + - '6. Exposing outputs': 'model_configuration/08a-step6.md' + - '7. Wrapping execution': 'model_configuration/09a-step7.md' + - '8. Testing run': 'model_configuration/10a-step8.md' + - '9. Uploading to GitHub, DockerHub and MINT ': 'model_configuration/11a-step9.md' + - 'Summarized example ': 'model_configuration/12a-end-to-end-example.md' + - 'Option 2: Parameters values passed through config files': + - '1. Starting a new component': 'model_configuration/03b-step1.md' + - '2. Tracing execution': 'model_configuration/04b-step2.md' + - '3. Exposing parameters': 'model_configuration/05b-step3.md' + - '4. Editing component configuration files': 'model_configuration/06b-step4.md' + - '5. Exposing input files': 'model_configuration/07b-step5.md' + - '6. Exposing outputs': 'model_configuration/08b-step6.md' + - '7. Wrapping execution': 'model_configuration/09b-step7.md' + - '8. Testing run': 'model_configuration/10b-step8.md' + - '9. Uploading to GitHub, DockerHub and MINT ': 'model_configuration/11a-step9.md' + - 'Summarized example ': 'model_configuration/12b-end-to-end-example.md' + - Credentials: 'commands/configure.md' - Available commands: usage.md - - Credentials: 'commands/configure.md' - - 'Creating a model component': - - 'Overview': 'model_configuration/01-overview.md' - - 'Before you start...': 'model_configuration/02-pre-steps.md' - - '1. Start a new component': 'model_configuration/03-step1.md' - - '2. Trace execution': 'model_configuration/04-step2.md' - - '3. Expose model parameters': 'model_configuration/05-step3.md' - - '4. Edit component configuration files': 'model_configuration/06-step4.md' - - '5. Expose model inputs': 'model_configuration/07-step5.md' - - '6. Expose model outputs': 'model_configuration/08-step6.md' - - '7. Wrap execution': 'model_configuration/09-step7.md' - - '8. Test run': 'model_configuration/10-step8.md' - - '9. Upload to GitHub, DockerHub and MINT ': 'model_configuration/11-step9.md' - - 'Summarized example ': 'model_configuration/12-end-to-end-example.md' - Glossary: glossary.md -theme: +theme: name: material logo: 'logo.png' markdown_extensions: