WIP: Futures #153
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,6 @@ | ||
| graft ./assets/* | ||
| include ./assets/futures/* | ||
| include ./assets/drs_config.toml | ||
| include ./assets/evaluation_system.conf | ||
| include ./deploy.py | ||
| include src/freva/*.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| { | ||
|
There was a problem hiding this comment. This took me a while: Those are the templates to create content from the futures, aren't they? Please put a separate README in there. I tried multiple times to execute those notebook like an idiot.. |
||
| "cells": [ | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "da58d5ea", | ||
| "metadata": { | ||
| "tags": [ | ||
| "solr-parameters" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Define the solr parameter\n", | ||
| "project=\"reanalysis\"\n", | ||
| "product=\"reanalysis\"\n", | ||
| "institute=\"ecmwf\"\n", | ||
| "model=\"ifs\"\n", | ||
| "experiment=\"era5\" \n", | ||
| "ensemble=\"reanalysis\"\n", | ||
| "time=\"200001010000-201012310000\"\n", | ||
| "time_frequency=\"mon\"\n", | ||
| "realm=\"atmos\"\n", | ||
| "variable=[\"tx10p\", \"tx90p\", \"tn10p\", \"tn90p\"]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "ad6886ba", | ||
| "metadata": { | ||
| "tags": [ | ||
| "parameters" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "ntasks=8\n", | ||
| "future_code = \"\"" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "0e174599", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from tempfile import TemporaryDirectory\n", | ||
| "import freva\n", | ||
| "_=freva.config(\"/home/wilfred/workspace/freva/compose/local-eval-system.conf\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "dbbd8c5a", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "outdir = TemporaryDirectory()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "2da586e1", | ||
| "metadata": { | ||
| "tags": [ | ||
| "code" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from dateutil import parser\n", | ||
| "time_period=\",\".join((str(parser.parse(t).year) for t in time.split(\"-\")))\n", | ||
| "res = freva.run_plugin(\n", | ||
| " \"climdexcalc\",\n", | ||
| " project=project,\n", | ||
| " product=product,\n", | ||
| " institute=institute,\n", | ||
| " model=model,\n", | ||
| " experiment=experiment,\n", | ||
| " ensemble=ensemble,\n", | ||
| " timeperiod=time_period,\n", | ||
| " indices=\",\".join(variable),\n", | ||
| " ntasks=ntasks,\n", | ||
| " outputdir=outdir.name,\n", | ||
| " batchmode=True,\n", | ||
| ")\n", | ||
| "res.wait()\n", | ||
| "output = res.get_result_paths()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "4f256fd7", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "freva.add_user_data(\n", | ||
| " product,\n", | ||
| " *output,\n", | ||
| " how=\"move\",\n", | ||
| " _project=project,\n", | ||
| " institute=institute,\n", | ||
| " model=model,\n", | ||
| " experiment=experiment,\n", | ||
| " ensemble=ensemble,\n", | ||
| " realm=realm,\n", | ||
| " future=future_code\n", | ||
| ")\n", | ||
| "outdir.cleanup()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "7daf2cac", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "celltoolbar": "Tags", | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.10.9" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "0e005d16", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import freva\n", | ||
| "from freva._futures import Futures" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "da58d5ea", | ||
| "metadata": { | ||
| "tags": [ | ||
| "solr-parameters" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Define the solr parameter\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "ad6886ba", | ||
| "metadata": { | ||
| "tags": [ | ||
| "parameters" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Define additional parameters to run the plugin\n", | ||
| "future_id = \"\"" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "2da586e1", | ||
| "metadata": { | ||
| "tags": [ | ||
| "code" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Run the plugin\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "4f256fd7", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# res.wait()\n", | ||
| "output = res.get_result_paths()\n", | ||
| "Futures.from_existing_code_hash(future_id).add_data(\n", | ||
| " project,\n", | ||
| " product,\n", | ||
| " *output,\n", | ||
| " history_id=res.id,\n", | ||
| " institute=institute,\n", | ||
| " model=model,\n", | ||
| " experiment=experiment,\n", | ||
| " ensemble=ensemble,\n", | ||
| " realm=realm,\n", | ||
| " time_frequency=time_frequency,\n", | ||
| " cmor_table=cmor_table,\n", | ||
| ")" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "celltoolbar": "Tags", | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.10.9" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to add some user-data via
freva user-data add --institute "Andrej" --model "AndrejMod" --experiment AndrejExp --project foo paths /home/afast/Downloads/ncIt broke with:
But I was not able to set the hash as it is missing in the parameters. Is it even desired that the user has to set the hash manually?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I manually put
hashinto the code in order to bypass this error message and to add my local data into the solr. This didn't work ashashis not part of the solr. Should it?