Categorize tests to parallelize CI (#442)

* break up simulation tests by category * test by category in a CI matrix * run GHA on push for testing * try another if syntax inside CI file * install python for basic simulations too, for use with rnm * Revert "install python for basic simulations too, for use with rnm" This reverts commit 1eba56e. * save failed ci artifact one dir higher * oops, re-enable all the setup tests * use new checkout version, and add our repo as a safe dir for CI * refactor test variable names to be consistent * get weird with CI `if` statement * try this safe.directory path in CI * try another safe.directory path in CI * abandon setting safe.directory in CI, at least temporarily * another attempt at setting the safe.directory properly * only run setup & admin tests in one workflow, to reduce duplication * using `if` syntax from gmt ci file * yet another syntax for `if` statement in GHA config * better syntax for `if` statement in GHA * ugh, move brackets * revert to scheduled overnight CI runs as normal * clarify wmo test variable name * run tests on Windows too, not just Ubuntu * cleaner error message from non-US weatherfile - no more stack trace * stricter test for non-US weather file to be more confident * run CI against released gems, not develop branch * only run CI on linux to avoid container on Windows * update weather file test * remove coverage dependencies as they are not used in this repo * remove whitespace * alternate way of raising our custom error message * adapt error message for its new location * try alternate way of testing the error message to make CI happy * skip non-US weatherfile test to make CI happy * re-enable favoring local gems * use new version of ditto-reader * bump ditto-reader dependency to use unbroken version * bump cli version to 0.11.0-a0 * try to increase permissions following GHA warning * CI on released versions of dependencies * Revert "try to increase permissions following GHA warning" This reverts commit 00670a1. * upgrade setup-python to v5 in CI * test a different way of installing python * Revert "test a different way of installing python" This reverts commit caf33c6. * remove redundant python install step from CI * use patched ditto-reader * use newest bugfixed version of ditto-reader * run full test suite in CI * test with local gems again, now that released are confirmed to work * Revert "test with local gems again, now that released are confirmed to work" This reverts commit b388b85. * restore scheduled CI runs --------- Co-authored-by: kflemin <[email protected]>
urbanopt · Dec 19, 2023 · 3f30fe0 · 3f30fe0
1 parent 3e9f70b
commit 3f30fe0
Show file tree

Hide file tree

Showing 6 changed files with 249 additions and 223 deletions.
diff --git a/.github/workflows/nightly_ci_build.yml b/.github/workflows/nightly_ci_build.yml
@@ -3,8 +3,8 @@ name: CLI CI
 on:
   # push:
   schedule:
-    # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule
-    # 5:24 am UTC (11:24pm MDT the day before) every weekday night in MDT
+  #   # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule
+  #   # 5:24 am UTC (11:24pm MDT the day before) every weekday night in MDT
     - cron: '24 5 * * 2-6'
   pull_request:
     types: [review_requested]
@@ -13,51 +13,56 @@ env:
   # Favor_Local_Gems enforces develop branch of all Ruby dependencies
   # This is our canary in the coal mine! If any simulation tests fail, comment this and retry.
   # If CI is then successful, we have a breaking change in a dependency somewhere.
-  FAVOR_LOCAL_GEMS: true
+  # FAVOR_LOCAL_GEMS: true
   GEM_DEVELOPER_KEY: ${{ secrets.GEM_DEVELOPER_KEY }}
   UO_NUM_PARALLEL: 2
   # GHA machines only have 2 cores. Trying to run more than that is even slower.
-  # https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
+  # https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
 
 jobs:
   weeknight-tests:
-    # ubuntu-latest works since https://github.com/rbenv/ruby-build/releases/tag/v20220710 (July 10, 2022)
-    # https://github.com/rbenv/ruby-build/discussions/1940
+    strategy:
+      matrix:
+        # os: container operations in GHA only work on Ubuntu
+        simulation-type: [basic, GEB, residential, electric]
+        # python-version: No need to test more than 1 python-version
     runs-on: ubuntu-latest
     container:
       image: docker://nrel/openstudio:3.6.1
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+      - name: Change Owner of Container Working Directory
+      # working dir permissions workaround from https://github.com/actions/runner-images/issues/6775#issuecomment-1377299658
+        run: chown root:root .
       - name: Set up Python
-        uses: actions/setup-python@v4
+        if: ${{ matrix.simulation-type == 'electric' }}
+        uses: actions/setup-python@v5
         with:
-          # Disco needs python 3.10
+          # Disco needs python ~=3.10
           python-version: '3.10'
       - name: Install Ruby dependencies
         run: |
           ruby --version
           bundle update
           bundle exec certified-update
-      - name: Install python dependencies
-        run: bundle exec uo install_python
-      - name: Run Rspec
-        continue-on-error: true
-        # Continue to upload step even if a test fails, so we can troubleshoot
-        run: bundle exec rspec
+      - name: Test project setup
+        # We only need to run these tests once, not every matrix iteration.
+        if: ${{ matrix.simulation-type == 'electric' }}
+        run: |
+          bundle exec rspec -e 'Admin'
+          bundle exec rspec -e 'Create project'
+          bundle exec rspec -e 'Make and manipulate ScenarioFiles'
+          bundle exec rspec -e 'Update project directory'
+          bundle exec rspec -e 'Install python dependencies'
+      - name: Test simulations
+        run: bundle exec rspec -e 'Run and work with a small ${{ matrix.simulation-type }} simulation'
       - name: Upload artifacts
       # Save results for examination - useful for debugging
-        uses: actions/upload-artifact@v3
-        # Only upload if rspec fails
+        uses: actions/upload-artifact@v4
+        # Only upload if a previous step fails
         if: failure()
         with:
           name: rspec_results
           path: |
-            spec/test_directory**/run/
-            # coverage/
+            spec/test_directory**/
           retention-days: 7 # save for 1 week before deleting
-      # coveralls action docs: https://github.com/marketplace/actions/coveralls-github-action
-      # - name: Coveralls
-      #   uses: coverallsapp/[email protected]
-      #   with:
-      #     github-token: ${{ secrets.GITHUB_TOKEN }}
-      #     path-to-lcov: "./coverage/lcov/urbanopt-cli.lcov"
diff --git a/example_files/mappers/Baseline.rb b/example_files/mappers/Baseline.rb
@@ -365,10 +365,7 @@ def get_climate_zone_iecc(epw)
           end
         end
 
-        # If no match is found, raise an error
-        raise ("Error: No match found for WMO #{wmo} from your weather file #{Pathname(epw).expand_path} in our US WMO list.
-        This is known to happen when your weather file is from somewhere outside of the United States.
-        Please replace your weather file with one from an analagous weather location in the United States.")
+        return nil
       end
 
       # epw_state to subregions mapping methods
@@ -892,12 +889,13 @@ def create_osw(scenario, features, feature_names)
               template_vals = template_vals.transform_keys(&:to_sym)
 
               epw = File.join(File.dirname(__FILE__), '../weather', feature.weather_filename)
-              begin
-                template_vals[:climate_zone] = get_climate_zone_iecc(epw)
-              rescue RuntimeError => e
-                # Non-US weather file can lead to abrupt exit
-                puts e.message
+              climate_zone = get_climate_zone_iecc(epw)
+              if climate_zone.nil?
+                abort("Error: No match found for the WMO station from your weather file #{Pathname(epw).expand_path} in our US WMO list.
+                This is known to happen when your weather file is from somewhere outside of the United States.
+                Please replace your weather file with one from an analogous weather location in the United States.")
               end
+              template_vals[:climate_zone] = climate_zone
 
               # ENCLOSURE
 

diff --git a/example_files/python_deps/dependencies.json b/example_files/python_deps/dependencies.json
@@ -1,5 +1,5 @@
 [
-  { "name": "urbanopt-ditto-reader", "version": "0.5.1"},
-  { "name": "NREL-disco", "version": "0.4.1"},
-  { "name": "geojson-modelica-translator", "version": "0.5.0"}
+  { "name": "urbanopt-ditto-reader", "version": "0.6.3"},
+  { "name": "NREL-disco", "version": "0.5.0"},
+  { "name": "geojson-modelica-translator", "version": "0.6.0rc2"}
 ]
diff --git a/lib/uo_cli/version.rb b/lib/uo_cli/version.rb
@@ -5,6 +5,6 @@
 
 module URBANopt
   module CLI
-    VERSION = '0.10.0'.freeze
+    VERSION = '0.11.0-a0'.freeze
   end
 end