From 542816b4510b0b2538b3550d807ee3559c7ff234 Mon Sep 17 00:00:00 2001 From: Zwart Date: Mon, 17 Jun 2024 15:40:59 -0700 Subject: [PATCH 1/3] adding python example to targets qmd --- .github/workflows/combined.yaml | 5 +++++ dashboard/targets.qmd | 13 +++++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/.github/workflows/combined.yaml b/.github/workflows/combined.yaml index 83c5d33b25..b7b1393248 100644 --- a/.github/workflows/combined.yaml +++ b/.github/workflows/combined.yaml @@ -66,6 +66,11 @@ jobs: fetch-depth: 0 set-safe-directory: '*' + - name: Install Python packages + run: | + apt-get update && apt-get install -y python3-pip + pip3 install pandas + - uses: quarto-dev/quarto-actions/setup@v2 with: version: 1.4.146 diff --git a/dashboard/targets.qmd b/dashboard/targets.qmd index 5851e9b6ea..55248f3cbb 100644 --- a/dashboard/targets.qmd +++ b/dashboard/targets.qmd @@ -74,10 +74,19 @@ The daily mean target file is located at the following URL. url <- "https://sdsc.osn.xsede.org/bio230014-bucket01/challenges/targets/project_id=usgsrc4cast/duration=P1D/river-chl-targets.csv.gz" ``` -You can directly load it into R using the following +You can directly load it into memory for R ```{r} -aquatics_targets <- read_csv(url, show_col_types = FALSE) +aquatics_targets <- readr::read_csv(url, show_col_types = FALSE) +``` + +or python + +```{python} +import pandas as pd + +url = "https://sdsc.osn.xsede.org/bio230014-bucket01/challenges/targets/project_id=usgsrc4cast/duration=P1D/river-chl-targets.csv.gz" +aquatics_targets = pd.read_csv(url) ``` The file contains the following columns From d6d8ac0989741ef75f2bcf3418b0a89c491089b3 Mon Sep 17 00:00:00 2001 From: Zwart Date: Mon, 17 Jun 2024 15:55:06 -0700 Subject: [PATCH 2/3] adding python to instructions --- dashboard/instructions.qmd | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/dashboard/instructions.qmd b/dashboard/instructions.qmd index a9932bd1f1..dbc1cb9d19 100644 --- a/dashboard/instructions.qmd +++ b/dashboard/instructions.qmd @@ -14,7 +14,7 @@ We provide an overview of the steps for submitting with the details below: 2) Register and describe your model at . You are not required to register if your forecast submission uses the word "example" in your model_id". Any forecasts with "example" in the model_id will not be used in forecast evaluation analyses. Use `usgsrc4cast` as the challenge you are registering for. 3) Generate a forecast! 4) Write the forecast output to a file that follows our standardized format (described below). -5) Submit your forecast using an R function (provided below). +5) Submit your forecast using an R or python function (provided below). 6) Watch your forecast be evaluated as new data are collected. ## Generating a forecast @@ -47,8 +47,8 @@ See the [variable tables](targets.qmd#sec-targets) for the horizon that is assoc ### Forecasts can be submitted everyday Since forecasts can be submitted everyday, automation is important. -We provide an [example GitHub](https://github.com/eco4cast/neon4cast-example) repository that can be used to automate your forecast with GitHub Actions. -It also includes the use of a custom Docker Container [eco4cast/rocker-neon4cast:latest](https://github.com/eco4cast/neon4cast-ci/blob/main/Dockerfile) that has many of the packages and functions needed to generate and submit forecasts. +We provide an example GitHub repository for both [R](https://github.com/eco4cast/usgsrc4cast-R-example) and [Python](https://github.com/eco4cast/usgsrc4cast-python-example) that can be used to automate your forecast with GitHub Actions. +It also includes the use of a custom Docker Container for [R (eco4cast/rocker-neon4cast:latest)](https://github.com/eco4cast/neon4cast-ci/blob/main/Dockerfile) or [Python (eco4cast/usgsrc4cast-python:latest)](https://github.com/eco4cast/usgsrc4cast-ci/blob/main/python/Dockerfile) that has many of the packages and functions needed to generate and submit forecasts. ## You can forecast at any of the USGS sites @@ -172,17 +172,29 @@ Individual forecast files can be uploaded any time. Teams will submit their forecast csv files through an R function. The csv file can only contain one unique `model_id` and one unique `project_id`. -The function is available using the following code +The submit function is available using the following code in R ```{r eval = FALSE} source("https://raw.githubusercontent.com/eco4cast/usgsrc4cast-ci/prod/R/eco4cast-helpers/submit.R") source("https://raw.githubusercontent.com/eco4cast/usgsrc4cast-ci/prod/R/eco4cast-helpers/forecast_output_validator.R") + +submit(forecast_file = "project_id-year-month-day-model_id.csv", project_id = "usgsrc4cast") ``` -The submit function is +or Python -```{r eval = FALSE} -submit(forecast_file = "project_id-year-month-day-model_id.csv") +```{python eval = FALSE} +import requests + +def download_and_exec_script(url): + response = requests.get(url) + response.raise_for_status() + exec(response.text, globals()) + +download_and_exec_script("https://raw.githubusercontent.com/eco4cast/usgsrc4cast-ci/prod/python/submit.py") +download_and_exec_script("https://raw.githubusercontent.com/eco4cast/usgsrc4cast-ci/prod/python/forecast_output_validator.py") + +submit(forecast_file = "project_id-year-month-day-model_id.csv", project_id = "usgsrc4cast") ``` ## Post-submission From d37beb1e527aef6ee84542284695a4a5e64e47af Mon Sep 17 00:00:00 2001 From: Zwart Date: Mon, 17 Jun 2024 15:55:23 -0700 Subject: [PATCH 3/3] adding matplotlib --- python/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/Dockerfile b/python/Dockerfile index c140285fe4..e5bf46fd2c 100644 --- a/python/Dockerfile +++ b/python/Dockerfile @@ -13,7 +13,7 @@ RUN sudo ./aws/install RUN apt-get update && apt-get -y install python3 python3-pip # Install Python packages -RUN pip install --no-cache-dir pandas pyarrow statsmodels boto3 requests +RUN pip install --no-cache-dir pandas pyarrow statsmodels boto3 requests matplotlib COPY cron.sh /etc/services.d/cron/run