Skip to content
This repository has been archived by the owner on May 31, 2024. It is now read-only.

Commit

Permalink
updated doc
Browse files Browse the repository at this point in the history
  • Loading branch information
@dgarijo
dgarijo committed Aug 20, 2020
1 parent 3eca992 commit 84d5a87
Show file tree
Hide file tree
Showing 4 changed files with 192 additions and 194 deletions.
29 changes: 10 additions & 19 deletions docs/faq.md
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,10 @@

## Frequently Asked Questions

**What do the properties in the table mean?**

If you find any of the fields in the metadata table confusing, just select them for editing. You will be shown a definition of what each property means. For example by editing property 16 (Purpose):

```bash
Current value for Purpose is: ['Crop seasonal production']
Definition: Objective or main functionality that can be achieved by running this model
```

**I don't know all metadata of my model right now. Can I save my progress?**

Yes. You can use the `save` command at any point to save your progress. Your metadata will be saved as a JSON file, which you can load with the `load` command. When you are ready to submit, just type `send`.

**What happens if I submit my model metadata twice?**

Right now MIC does not support editing model metadata. If you submit the same model metadata twice, you will create two entries in the catalog. We are working to support this feature.

## Frequently Asked Questions

**How do I create several variations of a component?**

We recommend generating several components (model configurations). When uploading the components in the last step, you will be able to associate all these model configurations to the same model, relating all of them together.

**What happens if I submit my component twice?**

Right now MIC does not allow editing model metadata, so if a component already exists, it will be uploaded as a different version.
124 changes: 64 additions & 60 deletions docs/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,61 +1,65 @@
Model Insertion CLI (MIC) is a command-line interface for adding models on a Model Catalog Service.

MIC is an application that will guide you through the steps required for encapsulating your model component and exposing a set of inputs and parameters of interest. MIC also allows describing basic model metadata: model version, model configuration, parameters, inputs, outputs, authors and contribuors.

MIC has been tested in OSX and Linux, and we are currently testing it in Windows. It is installed through a simple pip command.

!!! info
MIC is an ALPHA version, which we are still testing and developing continuously. If you find an error or experience any issue, please report them [here](https://github.com/mintproject/mic/issues/new/choose).

## Requirements

MIC has the following requirements:

1. Python >= 3.6
2. Docker


### Python 3

DAME uses Python. Please, follow the steps bellow to install it:

- [Installation on Linux](https://realpython.com/installing-python/#linux)
- [Installation on Windows](https://realpython.com/installing-python/#windows)
- [Installation on Mac](https://realpython.com/installing-python/#macos-mac-os-x)

### Docker

MIC uses Docker test and run model components.

- [Installation on Linux](https://docs.docker.com/engine/install/)
- [Installation on MacOS](https://docs.docker.com/docker-for-mac/install/)


## Installation

To install MIC, open a terminal and run:

```bash
$ pip install mic
```

You did it! If you want to verify the installation just type:

```bash
$ mic version
```

You should see a message similar to:

```bash
mic v1.0.1
```

## Development version

If you want to install the latest development version, open a terminal and type:

```bash
$ pip install git+https://github.com/mintproject/mic.git@develop -U
```
Model Insertion CLI (MIC) is a command-line interface for adding models on a Model Catalog Service.

MIC is an application that will guide you through the steps required for encapsulating your model component and exposing a set of inputs and parameters of interest. MIC also allows describing basic model metadata: model version, model configuration, parameters, inputs, outputs, authors and contribuors.

MIC has been tested in OSX, Linux and Windows. It is installed through a simple pip command.

!!! info
MIC is an ALPHA version, which we are still testing and developing continuously. If you find an error or experience any issue, please report them [here](https://github.com/mintproject/mic/issues/new/choose).

## Requirements

MIC has the following requirements:

1. Python >= 3.6
2. Docker


### Python 3

DAME uses Python. Please, follow the steps bellow to install it:

- [Installation on Linux](https://realpython.com/installing-python/#linux)
- [Installation on Windows](https://realpython.com/installing-python/#windows)
- [Installation on Mac](https://realpython.com/installing-python/#macos-mac-os-x)

### Docker

MIC uses Docker test and run model components.

- [Installation on Linux](https://docs.docker.com/engine/install/)
- [Installation on MacOS](https://docs.docker.com/docker-for-mac/install/)


## Installation

To install MIC, open a terminal and run:

```bash
$ pip install mic
```

You did it! If you want to verify the installation just type:

```bash
$ mic version
```

You should see a message similar to:

```bash
mic v1.0.1
```

## Limitations

Note that MIC has been designed to run Unix-based applications. Windows based applications (e.g., models that execute through an .exe) are not currently supported.

## Development version

If you want to install the latest development version, open a terminal and type:

```bash
$ pip install git+https://github.com/mintproject/mic.git@develop -U
```
Note theat the development version may be unstable.
191 changes: 97 additions & 94 deletions docs/model_configuration/03-step1.md
View file
Original file line number Diff line number Diff line change
@@ -1,94 +1,97 @@
This step is going to create a computational environment based in the executables of your working directory and template MIC file template (`mic.yaml`) which will be placed in a `/mic` directory. This file (which will be completed in subsequent steps) contains the executable information and metadata about your model component.

### How to perform this step?

To run this step, you must `cd` to the folder you usually use for executing your component. Please make sure that all the information needed for executing your model exists in this folder (including data, executable scripts, etc.). You don't need to create any special folder structure, just use the same file structure you would normally use to execute your component.

For example, let us consider a simple java model that reads an input file. In order to test it with MIC, we have prepared a sample input file (input.txt), which we placed in the same folder as my executable:
```bash
$ ls
input.txt test_192-1.0-SNAPSHOT-jar-with-dependencies.jar
```

Then, in the folder, type:

```bash
$ mic pkg start
```
MIC will ask for the component name you want to use (you may use any name you want), and will show you a message similar to this:

```bash
Model component name: test_192
MIC has initialized the component.
[Created] data: C:\Users\dgarijo\Desktop\192\java_model\mic\data
[Created] docker: C:\Users\dgarijo\Desktop\192\java_model\mic\docker
[Created] src: C:\Users\dgarijo\Desktop\192\java_model\mic\src
[Created] mic.yaml: C:\Users\dgarijo\Desktop\192\java_model\mic\mic.yaml
C:\Users\dgarijo\Desktop\192\java_model\mic\mic.yaml created
You can disable the detection of dependencies using the option --no-dependencies
Dockerfile has been created: C:\Users\dgarijo\Desktop\192\java_model\mic\docker\Dockerfile
Downloading the base image and building your image
Step 1/1 : FROM mintproject/java:8

---> 4950fcaa2d0d
Successfully built 4950fcaa2d0d
Successfully tagged test_192:latest

You are in a Linux environment Debian distribution
We detect the following dependencies.

- If you install new dependencies using `apt` or `apt-get`, remember to add them in Dockerfile mic\docker\Dockerfile
- If you install new dependencies using python. Before the step `upload` run

pip freeze > mic/docker/requirements.txt

Please, run your Model Component.
```
As can be seen in the message above, MIC is creating an execution environment from scratch to make sure we capture the minimum set of dependencies needed to execute a model. Since we had a java executable, MIC already selected a Java environment. If we had python files, MIC would have promted us to select which version of Python to start from.

!!! warning
This command must **NOT** be executed on a folder already tracked by GitHub.


#### Expected Results

After executing the previous command, MIC creates a `mic` directory with three sub-directories and a MIC file (mic.yaml):

- data/: It contains your data (now it will be empty).
- src/: It contains your code and MIC Wrapper (i.e., the file that executes your code). In the next step, you are going to specify how to run your model in the command line. MIC will capture all the required information automatically.
- docker/: It contains the required files to create the Docker Image (if everything goes well, you will not have to modify this directory). In later steps, MIC will populate this directory with the files that are needed to capture your computational infrastructure.

The MIC file will have a few lines at the moment, capturing the dependencies of the current environment:

```yaml
step: 1
name: test_192
docker_image: dgarijo/test_192:20.6.1
framework:
- java8
- mintproject/java:8
- .jar
```
#### Help command
```bash
Usage: mic pkg start [OPTIONS] USER_EXECUTION_DIRECTORY

Generates mic.yaml and the directories (data/, src/, docker/) for your
model component. Also initializes a local GitHub repository

The argument: `model_configuration_name` is the name of your model
configuration

Options:
--dependencies / --no-dependencies Enable/Disable tracking dependencies
(enabled by default)
--name TEXT Name of the model component
--help Show this message and exit.
```

!!! info
Next releases of MIC will allow customizing an initial Docker image.

This step is going to create a computational environment based in the executables of your working directory and template MIC file template (`mic.yaml`) which will be placed in a `/mic` directory. This file (which will be completed in subsequent steps) contains the executable information and metadata about your model component.

### How to perform this step?

To run this step, you must `cd` to the folder you usually use for executing your component. Please make sure that all the information needed for executing your model exists in this folder (including data, executable scripts, etc.). You don't need to create any special folder structure, just use the same file structure you would normally use to execute your component.

For example, let us consider a simple java model that reads an input file. In order to test it with MIC, we have prepared a sample input file (input.txt), which we placed in the same folder as my executable:
```bash
$ ls
input.txt test_192-1.0-SNAPSHOT-jar-with-dependencies.jar
```

Then, in the folder, type:

```bash
$ mic pkg start
```
!!! warning
If you want MIC to start from your own Docker image, you should use the flag `mic pkg start --image <yourimage:version>`. Do not forget to install MIC in your image with `pip install mic` so you have all the commands available.

MIC will ask for the component name you want to use (you may use any name you want), and will show you a message similar to this:

```bash
Model component name: test_192
MIC has initialized the component.
[Created] data: C:\Users\dgarijo\Desktop\192\java_model\mic\data
[Created] docker: C:\Users\dgarijo\Desktop\192\java_model\mic\docker
[Created] src: C:\Users\dgarijo\Desktop\192\java_model\mic\src
[Created] mic.yaml: C:\Users\dgarijo\Desktop\192\java_model\mic\mic.yaml
C:\Users\dgarijo\Desktop\192\java_model\mic\mic.yaml created
You can disable the detection of dependencies using the option --no-dependencies
Dockerfile has been created: C:\Users\dgarijo\Desktop\192\java_model\mic\docker\Dockerfile
Downloading the base image and building your image
Step 1/1 : FROM mintproject/java:8

---> 4950fcaa2d0d
Successfully built 4950fcaa2d0d
Successfully tagged test_192:latest

You are in a Linux environment Debian distribution
We detect the following dependencies.

- If you install new dependencies using `apt` or `apt-get`, remember to add them in Dockerfile mic\docker\Dockerfile
- If you install new dependencies using python. Before the step `upload` run

pip freeze > mic/docker/requirements.txt

Please, run your Model Component.
```
As can be seen in the message above, MIC is creating an execution environment from scratch to make sure we capture the minimum set of dependencies needed to execute a model. Since we had a java executable, MIC already selected a Java environment. If we had python files, MIC would have promted us to select which version of Python to start from.

!!! warning
This command must **NOT** be executed on a folder already tracked by GitHub.


#### Expected Results

After executing the previous command, MIC creates a `mic` directory with three sub-directories and a MIC file (mic.yaml):

- data/: It contains your data (now it will be empty).
- src/: It contains your code and MIC Wrapper (i.e., the file that executes your code). In the next step, you are going to specify how to run your model in the command line. MIC will capture all the required information automatically.
- docker/: It contains the required files to create the Docker Image (if everything goes well, you will not have to modify this directory). In later steps, MIC will populate this directory with the files that are needed to capture your computational infrastructure.

The MIC file will have a few lines at the moment, capturing the dependencies of the current environment:

```yaml
step: 1
name: test_192
docker_image: dgarijo/test_192:20.6.1
framework:
- java8
- mintproject/java:8
- .jar
```
#### Help command
```bash
Usage: mic pkg start [OPTIONS] USER_EXECUTION_DIRECTORY

Generates mic.yaml and the directories (data/, src/, docker/) for your
model component. Also initializes a local GitHub repository

The argument: `model_configuration_name` is the name of your model
configuration

Options:
--dependencies / --no-dependencies Enable/Disable tracking dependencies
(enabled by default)
--name TEXT Name of the model component
--help Show this message and exit.
```

!!! info
Next releases of MIC will allow customizing an initial Docker image.

42 changes: 21 additions & 21 deletions docs/model_configuration/12-end-to-end-example.md
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
Here we summarize the nine previous steps with the simple Java example executable we have used through the documentation.

Our component takes as input a text file and produces a text file as a result. It has one parameter 'p' which is used by the JAR executable. This is what the component folder looks like after we execute the first MIC command:

![Diagram](figures/folder.png)

The sequence of commands required for encapsulating the model are:

1. Start: `mic pkg start`
2. Trace the execution command: `mic pkg trace java -jar test_192-1.0-SNAPSHOT-jar-with-dependencies.jar -i input.txt -p 1500 -o output.txt`
3. Expose the 'p' parameter with default value 1350: `mic pkg parameters -f mic/mic.yaml -n p -v 1350`.
4. --Configuration step not necessary because model does not have configuration files--
5. Select inputs to expose: `mic pkg inputs`
6. Select outputs to expose: `mic pkg outputs`
7. Create wrapper: `mic pkg wrapper`. Edit the `run` file to indicate where the parameter `p` needs to be passed on, by replacing it in the invocation command.
8. Run wrapper and verify results: `mic pkg run`. Once you are done, exit the MIC container: Type `exit`
9. Upload: `mic pkg upload`

Done!


Here we summarize the nine previous steps with the simple Java example executable we have used through the documentation.

Our component takes as input a text file and produces a text file as a result. It has one parameter 'p' which is used by the JAR executable. This is what the component folder looks like after we execute the first MIC command:

![Diagram](figures/folder.png)

The sequence of commands required for encapsulating the model are:

1. Start: `mic pkg start`
2. Trace the execution command: `mic pkg trace java -jar test_192-1.0-SNAPSHOT-jar-with-dependencies.jar -i input.txt -p 1500 -o output.txt`
3. Expose the 'p' parameter with default value 1350: `mic pkg parameters -f mic/mic.yaml -n p -v 1350`.
4. --Configuration step not necessary because model does not have configuration files--
5. Select inputs to expose: `mic pkg inputs`
6. Select outputs to expose: `mic pkg outputs`
7. Create wrapper: `mic pkg wrapper`. Edit the `run` file to indicate where the parameter `p` needs to be passed on, by replacing it in the invocation command.
8. Run wrapper and verify results: `mic pkg run`. Once you are done, **exit the MIC environment: Type `exit`**
9. Upload: `mic pkg upload`

Done!


0 comments on commit 84d5a87

Please sign in to comment.