add garbage collection

eco4cast · Dec 8, 2023 · 9ae7360 · 9ae7360
1 parent 6ed447f
commit 9ae7360
Show file tree

Hide file tree

Showing 2 changed files with 8 additions and 2 deletions.
diff --git a/catalog/scores/collection.json b/catalog/scores/collection.json
@@ -203,13 +203,13 @@
   ],
   "assets": {
     "data": {
-      "href": "\"s3://anonymous@bio230014-bucket01/challenges/scores/parquet/?endpoint_override=sdsc.osn.xsede.org\"",
+      "href": "s3://anonymous@bio230014-bucket01/challenges/scores/parquet/?endpoint_override=sdsc.osn.xsede.org",
       "type": "application/x-parquet",
       "title": "Database Access",
       "roles": [
         "data"
       ],
-      "description": "Use `arrow` for remote access to the database. This R code will return results for the VERA Forecasting Challenge.\n\n### R\n\n```{r}\n# Use code below\n\nall_results <- arrow::open_dataset(\"s3://anonymous@bio230014-bucket01/challenges/scores/parquet/?endpoint_override=sdsc.osn.xsede.org\")\ndf <- all_results |> dplyr::collect()\n\n```\n       \n\nYou can use dplyr operations before calling `dplyr::collect()` to `summarise`, `select` columns, and/or `filter` rows prior to pulling the data into a local `data.frame`. Reducing the data that is pulled locally will speed up the data download speed and reduce your memory usage.\n\n\n"
+      "description": "Use `arrow` for remote access to the database. This R code will return results for the VERA Forecasting Challenge.\n\n### R\n\n```{r}\n# Use code below\n\nall_results <- arrow::open_dataset(s3://anonymous@bio230014-bucket01/challenges/scores/parquet/?endpoint_override=sdsc.osn.xsede.org)\ndf <- all_results |> dplyr::collect()\n\n```\n       \n\nYou can use dplyr operations before calling `dplyr::collect()` to `summarise`, `select` columns, and/or `filter` rows prior to pulling the data into a local `data.frame`. Reducing the data that is pulled locally will speed up the data download speed and reduce your memory usage.\n\n\n"
     },
     "thumbnail": {
       "href": "https://raw.githubusercontent.com/eco4cast/neon4cast-ci/main/catalog/thumbnail_plots/52760199990_d1a0f154fe_c.jpg",

diff --git a/catalog/update_stac.R b/catalog/update_stac.R
@@ -18,36 +18,42 @@ print('BUILDING FORECASTS')
 source('catalog/forecasts/forecast_models.R')
 
 rm(list = ls()) # remove all environmental vars between forecast and scores
+gc()
 
 # scores
 print('BUILDING SCORES')
 source('catalog/scores/scores_models.R')
 
 rm(list = ls())
+gc()
 
 # inventory
 print('BUILDING INVENTORY')
 source('catalog/inventory/create_inventory_page.R')
 
 rm(list = ls())
+gc()
 
 # summaries
 print('BUILDING FORECAST SUMMARIES')
 source('catalog/summaries/summaries_models.R')
 
 rm(list = ls())
+gc()
 
 # targets
 print('BUILDING TARGETS')
 source('catalog/targets/create_targets_page.R')
 
 rm(list = ls())
+gc()
 
 # noaa
 print('BUILDING NOAA')
 source('catalog/noaa_forecasts/noaa_forecasts.R')
 
 rm(list = ls())
+gc()
 
 # sites
 print('BUILDING SITES')