Merge pull request #1 from iiasa/nest_initialization

Nest initialization

README.md

@@ -31,7 +31,7 @@ Once downloaded, the database(s) (a zipped folder for each of the four models) s
 - For WaterCROP:
 - For M-LED: at *line 10* of the `MLED_hourly.R` file, defining the `db_folder` parameter
 - For OnSSET: include the OnSSET replication data folder unzipped in `onsset\onsset_replication` (more details below)
-- For NEST:
+- For NEST: The database only raw data needed in the pre-processing phase. The data needed to run the model is already included in the Github repository
 
 ## Setting up the environment
 
@@ -66,7 +66,11 @@ Each models is developed in a specific programming language and has thus specifi
   - Go to the instructions below to run the different OnSSET parts of the model.
 
 ### For NEST:
-  - XXX
+  - The requirements to run NEST are the same for running MESSAGEix, please check the [MESSAGEix documentation](https://docs.messageix.org/en/stable/install.html)
+  - Python 3.7 or newer
+  - GAMS with an active licence for the solver cplex
+  - R only needed to run pre-processing scripts
+  - once the setup is installed, also install the content of this folder as a `message-ix-models` package from source, following ths instructions in the `message-ix-models` [documentation](https://docs.messageix.org/projects/models/en/latest/install.html)
 
 ## Operating the platform
 
@@ -84,16 +88,18 @@ Each models is developed in a specific programming language and has thus specifi
 - Open the `OnSSET_Scenario_Running.ipynb` Notebook and run all of the cells. This will take a few minutes and will run the different scenarios and calculate the least-cost electrification options for the entire country. It will output it's results into several folders as .CSV files both as full results files for every population cluster in the country as well as summary files (also used later by NEST). 
 
 ### For NEST:
-  - XXX
+  - After having installed the folder by source (see *Setting up the environment*) run the script `nest\message_ix_models\project\leap_re_nest\build.py`
+  - around line 126, at * 4) add water structure* follow the instruction and run the command in the command prompt
 
 ## Soft-linking the models
 
+**RELEASE NOtE:** the current release is still not including all the exchange of data for future scenarios, which will be included in the next release.
+
 ### WaterCrop to M-LED:
   - WaterCrop produces netcdf files of irrigation water requirements and yield growth potential for all African countries. These files are contained (and can be updated) in the `./MLED_database/input_folder/watercrop` folder and corresponding subfolders for each crop. These files are then read in the `scenario_countryname.R` file of M-LED.
 
 ### WaterCROP to NEST:
-  - xxx
-  - 
+  - NEST uses the same input as M-LED
 
 ### M-LED to OnSSET: 
   - The details of transferring the demand data from the MLED modelling into OnSSET compatible files are completed in the `MLED_extraction_to_OnSSET.ipynb` notebook in the onsset root folder
@@ -128,7 +134,7 @@ In particular results can be examined by:
   - Otherwise they can be visusalised in GIS software such as QGIS: https://download.qgis.org/. They can be linked back to the cluster .gpkg shape files using a join on the "id" variable to visualise the shapes in addition to the electrification optimization information. 
   - Go to the RE4AFAGRI visualisation platform (coming soon) to see the existing scenarios. 
 
-### For NEST:
+### For NEST: work in progress
 
 ## Support
 

nest/.gitattributes

@@ -0,0 +1,3 @@
+# Reduce the number of merge conflicts
+doc/whatsnew.rst merge=union
+*.zip filter=lfs diff=lfs merge=lfs -text

nest/.github/codecov.yml

@@ -0,0 +1,3 @@
+comment:
+  layout: "diff, files"
+  behavior: once

nest/.github/pull_request_template.md

@@ -0,0 +1,53 @@
+<!--
+
+Delete each of these instruction comments as you complete it.
+
+Title: use a short, declarative statement similar to a commit message,
+e.g. “Change [thing X] to [fix solve bug|enable feature Y]”
+
+-->
+
+**Required:** write a single sentence that describes the changes made by this PR.
+
+<!-- Optional: write a longer description to help a reviewer understand the PR in ~3 minutes. -->
+
+## How to review
+
+**Required:** describe specific things that reviewer(s) must do, in order to ensure that the PR achieves its goal.
+If no review is required, write “No review:” and describe why.
+
+<!--
+For example, one or more of:
+
+- Read the diff and note that the CI checks all pass.
+- Run a specific code snippet or command and check the output.
+- Build the documentation and look at a certain page.
+- Ensure that changes/additions are self-documenting, i.e. that another
+  developer (someone like the reviewer) will be able to understand what the code
+  does in the future.
+-->
+
+## PR checklist
+
+<!-- This item is always required. -->
+- [ ] Continuous integration checks all ✅
+  <!--
+  The following items are all *required* if the PR results in changes to user-
+  facing behaviour, e.g. new features or fixes to existing behaviour. They are
+  *optional* if the changes are solely to documentation, CI configuration, etc.
+
+  In ambiguous cases, strike them out and add a short explanation, e.g.
+
+  - ~Add or expand tests.~ No change in behaviour, simply refactoring.
+  -->
+- [ ] Add or expand tests; coverage checks both ✅
+- [ ] Add, expand, or update documentation.
+- [ ] Update doc/whatsnew.
+  <!--
+  To do this, add a single line at the TOP of the “Next release” section of
+  doc/whatsnew.rst, where '999' is the GitHub pull request number:
+
+  - Title or single-sentence description from above (:pull:`999`:).
+
+  Commit with a message like “Add #999 to doc/whatsnew”
+  -->

nest/.github/workflows/lint.yaml

@@ -0,0 +1,64 @@
+name: Lint
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main, "migrate-*"]
+
+env:
+  # Maximum McCabe complexity for code
+  max-complexity: 15
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    continue-on-error: true
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - uses: actions/setup-python@v4
+      with:
+        # If the "Latest version testable on GitHub Actions" in pytest.yaml
+        # is not the latest 3.x stable version, adjust here to match:
+        # python-version: "3.x"
+        python-version: "3.10"
+        cache: pip
+        cache-dependency-path: "**/setup.cfg"
+
+    - name: Upgrade pip and install requirements
+      run: |
+        python -m pip install --upgrade pip
+        pip install black flake8 isort mypy setuptools-scm twine wheel
+
+    - name: Check "black" code style
+      run: black --check .
+
+    - name: Lint with flake8 & isort
+      run: |
+        flake8 --count --max-complexity=${{ env.max-complexity }} --show-source --statistics
+        isort --check-only .
+
+    - name: Check out ixmp
+      uses: actions/checkout@v3
+      with:
+        repository: iiasa/ixmp
+        path: ixmp
+
+    - name: Check out message-ix
+      uses: actions/checkout@v3
+      with:
+        repository: iiasa/message_ix
+        path: message-ix
+
+    - name: Check typing with mypy
+      # Give the paths to ixmp and message_ix sources. These need not be
+      # installed for static type checking
+      env:
+        MYPYPATH: "./ixmp/:./message-ix/"
+      # Also install stub packages and others that provide type hints
+      run: |
+        pip install genno iam-units pytest sdmx1 types-PyYAML types-setuptools
+        mypy ./message_ix_models

nest/.github/workflows/publish.yaml

@@ -0,0 +1,47 @@
+name: Publish
+
+on:
+  push:
+    branches: [ main ]
+    tags: [ "v*" ]
+  release:
+    types: [ published ]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - uses: actions/setup-python@v4
+      with:
+        # If the "Latest version testable on GitHub Actions" in pytest.yaml
+        # is not the latest 3.x stable version, adjust here to match:
+        # python-version: "3.x"
+        python-version: "3.10"
+        cache: pip
+        cache-dependency-path: "**/setup.cfg"
+
+    - name: Upgrade pip, setuptools-scm, twine, wheel
+      run: python -m pip install --upgrade pip setuptools-scm twine wheel
+
+    - name: Build package
+      run: |
+        python3 setup.py bdist_wheel sdist
+        twine check dist/*
+
+    - name: Publish to TestPyPI
+      uses: pypa/[email protected]
+      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
+      with:
+        user: __token__
+        password: ${{ secrets.TESTPYPI_TOKEN }}
+        repository_url: https://test.pypi.org/legacy/
+
+    - name: Publish to PyPI
+      uses: pypa/[email protected]
+      if: github.event_name == 'release'
+      with:
+        user: __token__
+        password: ${{ secrets.PYPI_TOKEN }}

nest/.github/workflows/pytest.yaml

@@ -0,0 +1,112 @@
+name: Test
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main , "migrate-*"]
+  schedule:
+  # 05:00 UTC = 06:00 CET = 07:00 CEST
+  - cron: "0 5 * * *"
+
+env:
+  # For setuptools-scm. With fetch --tags below, this ensures that enough
+  # history is fetched to contain the latest tag, so that setuptools-scm can
+  # generate the version number. Update:
+  # - See https://github.com/iiasa/message-ix-models/releases, at "NN commits to
+  #   main since this release". The value should be at least equal to NN + the
+  #   number of commits on any PR branch.
+  # - Reset to a lower value, e.g. 100, after a new release.
+  depth: 100
+
+jobs:
+  pytest:
+    strategy:
+      matrix:
+        # One job per OS; latest python version testable on GitHub actions.
+        # These should match the versions used in the "pytest" workflows of both
+        # ixmp and message_ix.
+        version:
+        - { os: macos-latest, python: "3.10" }
+        - { os: ubuntu-latest, python: "3.10" }
+        - { os: windows-latest, python: "3.10" }
+        # Versions of both ixmp and message_ix to use
+        upstream-version:
+        - v3.4.0  # Minimum version given in setup.cfg
+        - v3.5.0
+        - v3.6.0  # Latest released version
+        - main    # Development version
+
+      fail-fast: false
+
+    runs-on: ${{ matrix.version.os }}
+    name: ${{ matrix.version.os }}-py${{ matrix.version.python }}-upstream-${{ matrix.upstream-version }}
+
+    steps:
+    - name: Cancel previous runs that have not completed
+      uses: styfle/[email protected]
+
+    - name: Check out ixmp
+      uses: actions/checkout@v3
+      with:
+        repository: iiasa/ixmp
+        ref: ${{ matrix.upstream-version }}
+        path: ixmp
+        fetch-depth: ${{ env.depth }}
+
+    - name: Check out message-ix
+      uses: actions/checkout@v3
+      with:
+        repository: iiasa/message_ix
+        ref: ${{ matrix.upstream-version }}
+        path: message-ix
+        fetch-depth: ${{ env.depth }}
+
+    - name: Check out message-ix-models
+      uses: actions/checkout@v3
+      with:
+        lfs: true
+        path: message-ix-models
+        fetch-depth: ${{ env.depth }}
+
+    - name: Fetch tags (for setuptools-scm)
+      run: |
+        (cd ixmp; git fetch --tags --depth=${{ env.depth }})
+        (cd message-ix; git fetch --tags --depth=${{ env.depth }})
+        (cd message-ix-models; git fetch --tags --depth=${{ env.depth }})
+      shell: bash
+
+    - uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.version.python }}
+        cache: pip
+        cache-dependency-path: "**/setup.cfg"
+
+    - name: Upgrade pip
+      run: python -m pip install --upgrade pip
+
+    - uses: iiasa/actions/setup-gams@main
+      with:
+        version: 25.1.1
+        license: ${{ secrets.GAMS_LICENSE }}
+
+    - name: Install packages and dependencies
+      run: |
+        pip install ./ixmp
+        pip install ./message-ix[tests]
+        pip install ./message-ix-models[tests]
+
+    - name: Run test suite using pytest
+      working-directory: message-ix-models
+      run: pytest message_ix_models -rA --verbose --color=yes --cov-report=xml --cov-report=term-missing
+
+    - name: Test documentation build using Sphinx
+      if: startsWith(matrix.os, 'ubuntu')
+      env:
+        RTD_TOKEN_MESSAGE_DATA: ${{ secrets.RTD_TOKEN_MESSAGE_DATA }}
+      run: make --directory=message-ix-models/doc html
+
+    - name: Upload test coverage to Codecov.io
+      uses: codecov/codecov-action@v3
+      with:
+        root_dir: message-ix-models