diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e99f90b60..49cc1a664 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -22,7 +22,7 @@ repos: src/actinia_core.egg-info/ ) - repo: https://github.com/mundialis/github-workflows - rev: 1.0.0 + rev: 1.0.3 hooks: - id: linting diff --git a/CHANGELOG.md b/CHANGELOG.md index e768cc968..5b005dcc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,5 @@ # Changelog + All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), @@ -6,658 +7,823 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Types of changes -* __Added__ for new features. -* __Changed__ for changes in existing functionality. -* __Deprecated__ for soon-to-be removed features. -* __Removed__ for now removed features. -* __Fixed__ for any bug fixes. -* __Security__ in case of vulnerabilities. +- __Added__ for new features. +- __Changed__ for changes in existing functionality. +- __Deprecated__ for soon-to-be removed features. +- __Removed__ for now removed features. +- __Fixed__ for any bug fixes. +- __Security__ in case of vulnerabilities. +## [Unreleased](<>) -## [Unreleased] https://github.com/actinia-org/actinia-core/compare/4.3.1...main -## [4.X.X] - YYYY-MM-DD -released from \ +## [4.X.X](<>) - YYYY-MM-DD + +released from main ... -## [4.10.0] - 2023-08-31 +## [4.10.0](https://github.com/actinia-org/actinia-core/releases/tag/4.10.0) - 2023-08-31 + released from main + ## Changed -* Reset test markers by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/462 -* Use reusable pre-commit hooks for linting by @mmacata in https://github.com/actinia-org/actinia-core/pull/466 + +- Reset test markers by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/462 +- Use reusable pre-commit hooks for linting by @mmacata in https://github.com/actinia-org/actinia-core/pull/466 ## Added -* Possibility for no authentication by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/464 -"generated with gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.10.0" -f target_commitish=main -q .body" +- Possibility for no authentication by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/464 +"generated with gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.10.0" -f target_commitish=main -q .body" **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.9.3...4.10.0 -## [4.9.3] - 2023-08-01 +## [4.9.3](https://github.com/actinia-org/actinia-core/releases/tag/4.9.3) - 2023-08-01 + released from main + ### Fixed -* Update GRASS GIS to fix missing GDAL PostgreSQL support (see https://github.com/actinia-org/actinia-docker/pull/47 and https://github.com/OSGeo/grass/pull/3058) +- Update GRASS GIS to fix missing GDAL PostgreSQL support (see https://github.com/actinia-org/actinia-docker/pull/47 and https://github.com/OSGeo/grass/pull/3058) **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.9.2...4.9.3 "generated with gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.9.3" -f target_commitish=main -q .body" -## [4.9.2] - 2023-06-29 +## [4.9.2](https://github.com/actinia-org/actinia-core/releases/tag/4.9.2) - 2023-06-29 + released from main + ### Added -* pre-commit: activate tests by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/453 + +- pre-commit: activate tests by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/453 ### Fixed -* Fixes currently failing test by @mmacata in https://github.com/actinia-org/actinia-core/pull/455 -* Make resource storage delete asynchron by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/456 -* Black by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/458 -* fix HOME env by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/459 + +- Fixes currently failing test by @mmacata in https://github.com/actinia-org/actinia-core/pull/455 +- Make resource storage delete asynchron by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/456 +- Black by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/458 +- fix HOME env by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/459 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.9.1...4.9.2 - "generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.9.2" -f target_commitish=main -q .body`" +"generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.9.2" -f target_commitish=main -q .body`" +## [4.9.1](https://github.com/actinia-org/actinia-core/releases/tag/4.9.1) - 2023-06-14 -## [4.9.1] - 2023-06-14 released from main + ### Fixed -* Update version number to new_version by @mmacata in https://github.com/actinia-org/actinia-core/pull/449 -* Update RELEASE.md by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/451 -* Bump werkzeug from 2.0.3 to 2.3.6 by @dependabot in https://github.com/actinia-org/actinia-core/pull/414 + +- Update version number to new_version by @mmacata in https://github.com/actinia-org/actinia-core/pull/449 +- Update RELEASE.md by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/451 +- Bump werkzeug from 2.0.3 to 2.3.6 by @dependabot in https://github.com/actinia-org/actinia-core/pull/414 ## New Contributors -* @dependabot made their first contribution in https://github.com/actinia-org/actinia-core/pull/414 + +- @dependabot made their first contribution in https://github.com/actinia-org/actinia-core/pull/414 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.9.0...4.9.1 generated with gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.9.1" -f target_commitish=main -q .body" +## [4.9.0](https://github.com/actinia-org/actinia-core/releases/tag/4.9.0) - 2023-06-07 -## [4.9.0] - 2023-06-07 released from main + ### Added -* PyPI publish by @mmacata in https://github.com/actinia-org/actinia-core/pull/373 + +- PyPI publish by @mmacata in https://github.com/actinia-org/actinia-core/pull/373 ### Changed -* Configure Renovate by @renovate in https://github.com/actinia-org/actinia-core/pull/425 -* Update docker/build-push-action action to v4 by @renovate in https://github.com/actinia-org/actinia-core/pull/437 -* Update docker/login-action action to v2 by @renovate in https://github.com/actinia-org/actinia-core/pull/438 -* Update docker/setup-buildx-action action to v2 by @renovate in https://github.com/actinia-org/actinia-core/pull/439 -* Update docker/setup-qemu-action action to v2 by @renovate in https://github.com/actinia-org/actinia-core/pull/440 -* Update mundialis/actinia-core Docker tag to v4.8.0 by @renovate in https://github.com/actinia-org/actinia-core/pull/434 -* Update docker image by @mmacata in https://github.com/actinia-org/actinia-core/pull/442 -* Increase allowed length of additional version info key by @mmacata in https://github.com/actinia-org/actinia-core/pull/443 -* Align comment with reality by @mmacata in https://github.com/actinia-org/actinia-core/pull/444 -* Update redis_queue.md by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/445 -* Update grass docker image reference by @mmacata in https://github.com/actinia-org/actinia-core/pull/446 + +- Configure Renovate by @renovate in https://github.com/actinia-org/actinia-core/pull/425 +- Update docker/build-push-action action to v4 by @renovate in https://github.com/actinia-org/actinia-core/pull/437 +- Update docker/login-action action to v2 by @renovate in https://github.com/actinia-org/actinia-core/pull/438 +- Update docker/setup-buildx-action action to v2 by @renovate in https://github.com/actinia-org/actinia-core/pull/439 +- Update docker/setup-qemu-action action to v2 by @renovate in https://github.com/actinia-org/actinia-core/pull/440 +- Update mundialis/actinia-core Docker tag to v4.8.0 by @renovate in https://github.com/actinia-org/actinia-core/pull/434 +- Update docker image by @mmacata in https://github.com/actinia-org/actinia-core/pull/442 +- Increase allowed length of additional version info key by @mmacata in https://github.com/actinia-org/actinia-core/pull/443 +- Align comment with reality by @mmacata in https://github.com/actinia-org/actinia-core/pull/444 +- Update redis_queue.md by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/445 +- Update grass docker image reference by @mmacata in https://github.com/actinia-org/actinia-core/pull/446 ### Fixed -* Change runtime if runtime is higher 2147483647 by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/447 + +- Change runtime if runtime is higher 2147483647 by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/447 ## New Contributors -* @renovate made their first contribution in https://github.com/actinia-org/actinia-core/pull/425 + +- @renovate made their first contribution in https://github.com/actinia-org/actinia-core/pull/425 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.8.0...4.9.0 "generated with gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.9.0" -f target_commitish=main -q .body" +## [4.8.0](https://github.com/actinia-org/actinia-core/releases/tag/4.8.0) - 2023-05-10 -## [4.8.0] - 2023-05-10 released from main + ### Fixed -* change deprecated tag creation by @mmacata in https://github.com/actinia-org/actinia-core/pull/419 -* update GH action versions by @neteler in https://github.com/actinia-org/actinia-core/pull/420 -* docker CI: only run CI in actinia-org remote by @neteler in https://github.com/actinia-org/actinia-core/pull/424 + +- change deprecated tag creation by @mmacata in https://github.com/actinia-org/actinia-core/pull/419 +- update GH action versions by @neteler in https://github.com/actinia-org/actinia-core/pull/420 +- docker CI: only run CI in actinia-org remote by @neteler in https://github.com/actinia-org/actinia-core/pull/424 + ### Added -* Allow one job queue per user by @mmacata in https://github.com/actinia-org/actinia-core/pull/421 -* README.md: Thanks to all contributors by @neteler in https://github.com/actinia-org/actinia-core/pull/423 -* Allow redis queue config to be overwritten by env vars by @mmacata in https://github.com/actinia-org/actinia-core/pull/426 + +- Allow one job queue per user by @mmacata in https://github.com/actinia-org/actinia-core/pull/421 +- README.md: Thanks to all contributors by @neteler in https://github.com/actinia-org/actinia-core/pull/423 +- Allow redis queue config to be overwritten by env vars by @mmacata in https://github.com/actinia-org/actinia-core/pull/426 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.7.1...4.8.0 - generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.8.0" -f target_commitish=main -q .body` +generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.8.0" -f target_commitish=main -q .body` +## [4.7.1](https://github.com/actinia-org/actinia-core/releases/tag/4.7.1) - 2023-02-09 -## [4.7.1] - 2023-02-09 ### Fixed -* fix tgis merge by interim results by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/413 + +- fix tgis merge by interim results by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/413 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.7.0...4.7.1 generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.7.1" -f target_commitish=main -q .body` +## [4.7.0](https://github.com/actinia-org/actinia-core/releases/tag/4.7.0) - 2023-01-25 -## [4.7.0] - 2023-01-25 released from main + ### Added -* Add raster VRT support and support for mapset names by interim results by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/410 +- Add raster VRT support and support for mapset names by interim results by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/410 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.6.2...4.7.0 "generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.7.0" -f target_commitish=main -q .body`" - ## [4.6.2] - 2023-01-17 +## [4.6.2](https://github.com/actinia-org/actinia-core/releases/tag/4.6.2) - 2023-01-17 + released from main + ### Fixed -* New GRASS GIS 8.2 with fix https://github.com/OSGeo/grass/pull/2735 (https://github.com/OSGeo/grass/pull/2742) + +- New GRASS GIS 8.2 with fix https://github.com/OSGeo/grass/pull/2735 (https://github.com/OSGeo/grass/pull/2742) **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.6.1...4.6.2 +## [4.6.1](https://github.com/actinia-org/actinia-core/releases/tag/4.6.1) - 2023-01-04 -## [4.6.1] - 2023-01-04 released from main + ### Fixed -* try to fix wheel by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/405 + +- try to fix wheel by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/405 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.6.0...4.6.1 generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name="4.6.1" -f target_commitish=main -q .body` -## [4.6.0] - 2022-12-17 +## [4.6.0](https://github.com/actinia-org/actinia-core/releases/tag/4.6.0) - 2022-12-17 + released from main + ### Changed -* Job resumption working without new process chain by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/400 + +- Job resumption working without new process chain by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/400 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.5.0...4.6.0 "generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.6.0" -f target_commitish=main -q .body`" +## [4.5.0](https://github.com/actinia-org/actinia-core/releases/tag/4.5.0) - 2022-12-14 -## [4.5.0] - 2022-12-14 released from main + ### Changed -* Add possibility to use keycloak authentication token by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/381 -* More secure path mangling by @marcjansen in https://github.com/actinia-org/actinia-core/pull/281 -* Improve interim results by include additional mapsets by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/399 + +- Add possibility to use keycloak authentication token by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/381 +- More secure path mangling by @marcjansen in https://github.com/actinia-org/actinia-core/pull/281 +- Improve interim results by include additional mapsets by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/399 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.4.0...4.5.0 - generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.5.0" -f target_commitish=main -q .body` +generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.5.0" -f target_commitish=main -q .body` + +## [4.4.0](https://github.com/actinia-org/actinia-core/releases/tag/4.4.0) - 2022-11-22 - ## [4.4.0] - 2022-11-22 released from main + ### Added -* Resource storage older than X days deletion by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/392 -* Start with the adjustment of the interim results by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/390 + +- Resource storage older than X days deletion by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/392 +- Start with the adjustment of the interim results by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/390 ### Fixed -* update RELEASE documentation by @juleshaas in https://github.com/actinia-org/actinia-core/pull/389 -* replace deprecated (Timed)JSONWebSignatureSerializer by @metzm in https://github.com/actinia-org/actinia-core/pull/386 - * !! **Old API keys and tokens might not be valid anymore** !! -* update requirements.txt by @mmacata in https://github.com/actinia-org/actinia-core/pull/393 -* Update alpine dependency image version by @mmacata in https://github.com/actinia-org/actinia-core/pull/394 + +- update RELEASE documentation by @juleshaas in https://github.com/actinia-org/actinia-core/pull/389 +- replace deprecated (Timed)JSONWebSignatureSerializer by @metzm in https://github.com/actinia-org/actinia-core/pull/386 + - !! **Old API keys and tokens might not be valid anymore** !! +- update requirements.txt by @mmacata in https://github.com/actinia-org/actinia-core/pull/393 +- Update alpine dependency image version by @mmacata in https://github.com/actinia-org/actinia-core/pull/394 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.3.1...4.4.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.4.0" -f target_commitish=main -q .body` -## [4.3.1] - 2022-10-21 +## [4.3.1](https://github.com/actinia-org/actinia-core/releases/tag/4.3.1) - 2022-10-21 + released from main + ### Fixed -* Add initialisation of self.queue by @juleshaas in https://github.com/actinia-org/actinia-core/pull/388 -* Change pc style for module description for importer and exporter in module plugin by @juleshaas in https://github.com/actinia-org/actinia-core/pull/387 + +- Add initialisation of self.queue by @juleshaas in https://github.com/actinia-org/actinia-core/pull/388 +- Change pc style for module description for importer and exporter in module plugin by @juleshaas in https://github.com/actinia-org/actinia-core/pull/387 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.3.0...4.3.1 "generated with gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.3.1" -f target_commitish=main -q .body" -## [4.3.0] - 2022-09-22 +## [4.3.0](https://github.com/actinia-org/actinia-core/releases/tag/4.3.0) - 2022-09-22 + released from main + ### Added -* Black by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/378 -* Add job queue name to response by @mmacata in https://github.com/actinia-org/actinia-core/pull/380 + +- Black by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/378 +- Add job queue name to response by @mmacata in https://github.com/actinia-org/actinia-core/pull/380 ### Fixed -* Fix install description by @mmacata in https://github.com/actinia-org/actinia-core/pull/375 -* small typo by @linakrisztian in https://github.com/actinia-org/actinia-core/pull/367 -* Fix redoc rendering by @ninsbl in https://github.com/actinia-org/actinia-core/pull/379 + +- Fix install description by @mmacata in https://github.com/actinia-org/actinia-core/pull/375 +- small typo by @linakrisztian in https://github.com/actinia-org/actinia-core/pull/367 +- Fix redoc rendering by @ninsbl in https://github.com/actinia-org/actinia-core/pull/379 ## New Contributors -* @linakrisztian made their first contribution in https://github.com/actinia-org/actinia-core/pull/367 -* @ninsbl made their first contribution in https://github.com/actinia-org/actinia-core/pull/379 + +- @linakrisztian made their first contribution in https://github.com/actinia-org/actinia-core/pull/367 +- @ninsbl made their first contribution in https://github.com/actinia-org/actinia-core/pull/379 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.2.1...4.3.0 "generated with gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name=4.3.0 -f target_commitish=main -q .body" -## [4.2.1] - 2022-08-05 +## [4.2.1](https://github.com/actinia-org/actinia-core/releases/tag/4.2.1) - 2022-08-05 + released from main + ### Fixed -* Allow users to create and delete mapsets by @juleshaas and @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/374 + +- Allow users to create and delete mapsets by @juleshaas and @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/374 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.2.0...4.2.1 "generated with gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.2.1" -f target_commitish=main -q .body" -## [4.2.0] - 2022-08-03 +## [4.2.0](https://github.com/actinia-org/actinia-core/releases/tag/4.2.0) - 2022-08-03 + released from main + ### Added -* Add option to shutdown worker when queue is empty by @mmacata in https://github.com/actinia-org/actinia-core/pull/358 -* Add dependencies for actinia-parallel-plugin by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/356 -* Allow users with role user to create and delete own mapsets by @juleshaas in https://github.com/actinia-org/actinia-core/pull/365 + +- Add option to shutdown worker when queue is empty by @mmacata in https://github.com/actinia-org/actinia-core/pull/358 +- Add dependencies for actinia-parallel-plugin by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/356 +- Allow users with role user to create and delete own mapsets by @juleshaas in https://github.com/actinia-org/actinia-core/pull/365 ### Changed -* Always use local queue for sync requests by @mmacata in https://github.com/actinia-org/actinia-core/pull/361 -* Actinia docs: Update tutorial for running S-2 NDVI example by @griembauer in https://github.com/actinia-org/actinia-core/pull/363 + +- Always use local queue for sync requests by @mmacata in https://github.com/actinia-org/actinia-core/pull/361 +- Actinia docs: Update tutorial for running S-2 NDVI example by @griembauer in https://github.com/actinia-org/actinia-core/pull/363 ### Fixed -* Sync config by @mmacata in https://github.com/actinia-org/actinia-core/pull/360 -* docs: replace broken api docs links by @metzm in https://github.com/actinia-org/actinia-core/pull/362 -* actinia tests: fix test descriptions by @neteler in https://github.com/actinia-org/actinia-core/pull/371 + +- Sync config by @mmacata in https://github.com/actinia-org/actinia-core/pull/360 +- docs: replace broken api docs links by @metzm in https://github.com/actinia-org/actinia-core/pull/362 +- actinia tests: fix test descriptions by @neteler in https://github.com/actinia-org/actinia-core/pull/371 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.1.0...4.2.0 generated with gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.2.0" -f target_commitish=main -q .body +## [4.1.0](https://github.com/actinia-org/actinia-core/releases/tag/4.1.0) - 2022-07-20 -## [4.1.0] - 2022-07-20 released from main + ### Changed + - Actinia Tutorials Update by @Momen-Mawad in https://github.com/actinia-org/actinia-core/pull/350 ### Added + - Tests for importer: parameters resample and resolution by @juleshaas in https://github.com/actinia-org/actinia-core/pull/349 - Enable separate redis queue per job by @mmacata in https://github.com/actinia-org/actinia-core/pull/355 - Endpoint configuration by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/357 ### Fixed + - Fix dev setup by @mmacata in https://github.com/actinia-org/actinia-core/pull/352 - docs: actinia concepts got lost by @metzm in https://github.com/actinia-org/actinia-core/pull/353 ## New Contributors -* @Momen-Mawad made their first contribution in https://github.com/actinia-org/actinia-core/pull/350 + +- @Momen-Mawad made their first contribution in https://github.com/actinia-org/actinia-core/pull/350 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.0.1...4.1.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.1.0" -f target_commitish=main -q .body` -## [4.0.1] - 2022-05-20 +## [4.0.1](https://github.com/actinia-org/actinia-core/releases/tag/4.0.1) - 2022-05-20 + released from main + ### Fixed -* Fix wheel build in gha by @mmacata in https://github.com/actinia-org/actinia-core/pull/338 -* STAC Importer Issues by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/318 -* allow character "&" for t.rast.bandcalc by @metzm in https://github.com/actinia-org/actinia-core/pull/351 + +- Fix wheel build in gha by @mmacata in https://github.com/actinia-org/actinia-core/pull/338 +- STAC Importer Issues by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/318 +- allow character "&" for t.rast.bandcalc by @metzm in https://github.com/actinia-org/actinia-core/pull/351 + ### Added -* Add imported modules from actinia-tiling-plugin by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/342 -* Add more modules and configuration to add additinal modules in config by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/341 -* Prepare docker alpine to 3.15 by @mmacata in https://github.com/actinia-org/actinia-core/pull/343 -* Docker alpine3.15 update part 2 by @mmacata in https://github.com/actinia-org/actinia-core/pull/346 -* Use versionless GRASS GIS (8) instead of hardcoded grass7x by @neteler in https://github.com/actinia-org/actinia-core/pull/347 -* Allow a user and guest to request his own user (breaking change) by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/348 + +- Add imported modules from actinia-tiling-plugin by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/342 +- Add more modules and configuration to add additinal modules in config by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/341 +- Prepare docker alpine to 3.15 by @mmacata in https://github.com/actinia-org/actinia-core/pull/343 +- Docker alpine3.15 update part 2 by @mmacata in https://github.com/actinia-org/actinia-core/pull/346 +- Use versionless GRASS GIS (8) instead of hardcoded grass7x by @neteler in https://github.com/actinia-org/actinia-core/pull/347 +- Allow a user and guest to request his own user (breaking change) by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/348 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.0.0...4.0.1 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="4.0.1" -f target_commitish=main -q .body` ------------------------------ +______________________________________________________________________ + +## [4.0.0](https://github.com/actinia-org/actinia-core/releases/tag/4.0.0) - 2022-04-01 -## [4.0.0] - 2022-04-01 released from main + ### Changed -* Refactor rest by @mmacata in https://github.com/actinia-org/actinia-core/pull/320 -* Refactor rest - part 2 by @mmacata in https://github.com/actinia-org/actinia-core/pull/322 -* Move apidocs by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/337 + +- Refactor rest by @mmacata in https://github.com/actinia-org/actinia-core/pull/320 +- Refactor rest - part 2 by @mmacata in https://github.com/actinia-org/actinia-core/pull/322 +- Move apidocs by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/337 This is a major release because it breaks the actinia python API. Plugins were already adjusted accordingly. The HTTP REST API is not affected and moved (with this release completely) to [actinia-api](https://github.com/actinia-org/actinia-api). **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/3.3.0...4.0.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name=4.0.0 -f target_commitish=main -q .body` ------------------------------ +______________________________________________________________________ + +## [3.3.0](https://github.com/actinia-org/actinia-core/releases/tag/3.3.0) - 2022-03-31 -## [3.3.0] - 2022-03-31 released from main + ### Added -* Stac result export by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/290 -* Stac result export by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/334 -* Pystac issue by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/335 -* Pystac Issue changing name of the function by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/336 + +- Stac result export by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/290 +- Stac result export by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/334 +- Pystac issue by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/335 +- Pystac Issue changing name of the function by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/336 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/3.2.0...3.3.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name=3.3.0 -f target_commitish=main -q .body` -## [3.2.0] - 2022-03-23 +## [3.2.0](https://github.com/actinia-org/actinia-core/releases/tag/3.2.0) - 2022-03-23 + released from main + ### Added -* Reactivate redis queue by @mmacata in https://github.com/actinia-org/actinia-core/pull/304 -* Importer resampling and resolution by @juleshaas in https://github.com/actinia-org/actinia-core/pull/298 + +- Reactivate redis queue by @mmacata in https://github.com/actinia-org/actinia-core/pull/304 +- Importer resampling and resolution by @juleshaas in https://github.com/actinia-org/actinia-core/pull/298 ### Fixed -* Simplify lint workflow by @mmacata in https://github.com/actinia-org/actinia-core/pull/321 -* adapt s2 importer to work without Google BigQuery by @griembauer in https://github.com/actinia-org/actinia-core/pull/295 + +- Simplify lint workflow by @mmacata in https://github.com/actinia-org/actinia-core/pull/321 +- adapt s2 importer to work without Google BigQuery by @griembauer in https://github.com/actinia-org/actinia-core/pull/295 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/3.1.3...3.2.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name=3.2.0 -f target_commitish=main -q .body` +## [3.1.3](https://github.com/actinia-org/actinia-core/releases/tag/3.1.3) - 2022-03-11 -## [3.1.3] - 2022-03-11 released from main + ### Fixed -* export_strds: specify path to the directory where output is written by @metzm in https://github.com/actinia-org/actinia-core/pull/316 + +- export_strds: specify path to the directory where output is written by @metzm in https://github.com/actinia-org/actinia-core/pull/316 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/3.1.2...3.1.3 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name=3.1.3 -f target_commitish=main -q .body` +## [3.1.2](https://github.com/actinia-org/actinia-core/releases/tag/3.1.2) - 2022-02-23 -## [3.1.2] - 2022-02-23 released from main + ### Changed -* Move most docker config by @mmacata in https://github.com/actinia-org/actinia-core/pull/311 + +- Move most docker config by @mmacata in https://github.com/actinia-org/actinia-core/pull/311 + ### Fixed -* Mark breaking changes by @mmacata in https://github.com/actinia-org/actinia-core/pull/312 + +- Mark breaking changes by @mmacata in https://github.com/actinia-org/actinia-core/pull/312 + ### Added -* Add pipeline to add python wheel to release assets by @mmacata in https://github.com/actinia-org/actinia-core/pull/313 + +- Add pipeline to add python wheel to release assets by @mmacata in https://github.com/actinia-org/actinia-core/pull/313 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/3.1.1...3.1.2 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name=3.1.2 -f target_commitish=main -q .body` +## [3.1.1](https://github.com/actinia-org/actinia-core/releases/tag/3.1.1) - 2022-02-03 -## [3.1.1] - 2022-02-03 released from main + ## Added -* GHA and Dockerfile using github token by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/310 + +- GHA and Dockerfile using github token by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/310 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="$ESTIMATED_VERSION" -f target_commitish=main -q .body` +## [3.1.0](https://github.com/actinia-org/actinia-core/releases/tag/3.1.0) - 2022-02-02 -## [3.1.0] - 2022-02-02 released from main + ### Added -* Add RELEASE.md by @mmacata in https://github.com/actinia-org/actinia-core/pull/302 -* Retry webhook by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/299 + +- Add RELEASE.md by @mmacata in https://github.com/actinia-org/actinia-core/pull/302 +- Retry webhook by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/299 ### Changed -* allow '&' in parameters for more modules by @metzm in https://github.com/actinia-org/actinia-core/pull/308 + +- allow '&' in parameters for more modules by @metzm in https://github.com/actinia-org/actinia-core/pull/308 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="3.1.0" -f target_commitish=main -q .body` +## [3.0.1](https://github.com/actinia-org/actinia-core/releases/tag/3.0.1) - 2022-01-19 -## [3.0.1] - 2022-01-19 released from main + ### Added -* CITATION.cff: citable actinia core source code with DOI by @neteler in https://github.com/actinia-org/actinia-core/pull/300 -* Add software DOI badge by @neteler -* Change actinia software DOI badge to generic DOI by @neteler + +- CITATION.cff: citable actinia core source code with DOI by @neteler in https://github.com/actinia-org/actinia-core/pull/300 +- Add software DOI badge by @neteler +- Change actinia software DOI badge to generic DOI by @neteler **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/3.0.0...3.0.1 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="3.0.1" -f target_commitish=main -q .body` +## [3.0.0](https://github.com/actinia-org/actinia-core/releases/tag/3.0.0) - 2022-01-13 -## [3.0.0] - 2022-01-13 released from main + ### Breaking (API) -* Actinia core v3 by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/296, see also https://github.com/actinia-org/actinia-api/pull/2 + +- Actinia core v3 by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/296, see also https://github.com/actinia-org/actinia-api/pull/2 + ### Fixed -* Fix tests by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/297 -* Update actinia-api version by @mmacata + +- Fix tests by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/297 +- Update actinia-api version by @mmacata **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/2.3.1...3.0.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="3.0.0" -f target_commitish=main -q .body` ------------------------------ +______________________________________________________________________ +## [2.3.1](https://github.com/actinia-org/actinia-core/releases/tag/2.3.1) - 2021-12-22 -## [2.3.1] - 2021-12-22 released from main + ### Fixed -* installation instructions: update by @neteler in https://github.com/actinia-org/actinia-core/pull/287 -* update actinia-module-plugin + +- installation instructions: update by @neteler in https://github.com/actinia-org/actinia-core/pull/287 +- update actinia-module-plugin ### Added -* Test for STAC implementation by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/276 + +- Test for STAC implementation by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/276 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/2.3.0...2.3.1 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="2.3.1" -f target_commitish=main -q .body` -## [2.3.0] - 2021-12-16 +## [2.3.0](https://github.com/actinia-org/actinia-core/releases/tag/2.3.0) - 2021-12-16 + released from main + ### Added -* STAC in actinia by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/275 + +- STAC in actinia by @joaherrerama in https://github.com/actinia-org/actinia-core/pull/275 ### Changed -* Enhance dev setup by @mmacata in https://github.com/actinia-org/actinia-core/pull/288 -* update actinia-api version by @mmacata in https://github.com/actinia-org/actinia-core/pull/289 + +- Enhance dev setup by @mmacata in https://github.com/actinia-org/actinia-core/pull/288 +- update actinia-api version by @mmacata in https://github.com/actinia-org/actinia-core/pull/289 ## New Contributors -* @joaherrerama made their first contribution in https://github.com/actinia-org/actinia-core/pull/275 + +- @joaherrerama made their first contribution in https://github.com/actinia-org/actinia-core/pull/275 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/2.2.0...2.3.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="2.3.0" -f target_commitish=main -q .body` -## [2.2.0] - 2021-12-08 +## [2.2.0](https://github.com/actinia-org/actinia-core/releases/tag/2.2.0) - 2021-12-08 + released from main + ### Fixed -* try to fix uncontrolled data used in path by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/280 -* Add stac plugin to alpine config by @mmacata in https://github.com/actinia-org/actinia-core/pull/282 + +- try to fix uncontrolled data used in path by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/280 +- Add stac plugin to alpine config by @mmacata in https://github.com/actinia-org/actinia-core/pull/282 ### Added -* Add api_version to version endpoint by @mmacata in https://github.com/actinia-org/actinia-core/pull/283 -* Enable debugging via vscode by @mmacata in https://github.com/actinia-org/actinia-core/pull/284 -* New splitup in unit and integration tests by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/285 + +- Add api_version to version endpoint by @mmacata in https://github.com/actinia-org/actinia-core/pull/283 +- Enable debugging via vscode by @mmacata in https://github.com/actinia-org/actinia-core/pull/284 +- New splitup in unit and integration tests by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/285 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/2.0.1...2.2.0 generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="2.2.0" -f target_commitish=main -q .body` -## [2.0.1] - 2021-11-25 +## [2.0.1](https://github.com/actinia-org/actinia-core/releases/tag/2.0.1) - 2021-11-25 + released from main + ### Fixed -* Fix: create user endpoint by @griembauer in https://github.com/actinia-org/actinia-core/pull/268 -* revision of examples in actinia documentation by @juleshaas in https://github.com/actinia-org/actinia-core/pull/269 -* Using v2 by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/277 -* Use separate repo for api docs by @mmacata in https://github.com/actinia-org/actinia-core/pull/278 + +- Fix: create user endpoint by @griembauer in https://github.com/actinia-org/actinia-core/pull/268 +- revision of examples in actinia documentation by @juleshaas in https://github.com/actinia-org/actinia-core/pull/269 +- Using v2 by @anikaweinmann in https://github.com/actinia-org/actinia-core/pull/277 +- Use separate repo for api docs by @mmacata in https://github.com/actinia-org/actinia-core/pull/278 **Full Changelog**: https://github.com/actinia-org/actinia-core/compare/2.0.0...2.0.1 (generated with `gh api repos/mundialis/actinia_core/releases/generate-notes -f tag_name="2.0.1" -f target_commitish=main -q .body`) +## [2.0.0](https://github.com/actinia-org/actinia-core/releases/tag/2.0.0) - 2021-10-13 -## [2.0.0] - 2021-10-13 released from main + ### Breaking -* Add vector upload (isse 180, #260) + +- Add vector upload (isse 180, #260) + ### Fixed -* Fix cleanup error (#248) -* Fix several misspelled words (#254) + +- Fix cleanup error (#248) +- Fix several misspelled words (#254) + ### Changed -* Allow & in r.mapcalc processing (#253) -* Allow endpoints with and without trailing slash (#257) -* Test and improve actinia tutorials (#258) -* Resolve the insecure temporary files (#262, #256) + +- Allow & in r.mapcalc processing (#253) +- Allow endpoints with and without trailing slash (#257) +- Test and improve actinia tutorials (#258) +- Resolve the insecure temporary files (#262, #256) + ### Added -* Make the version output more flexible (#252) -* List all available mapsets (issue 162, #249) -* Flake8 Linting for test folder (#255) ------------------------------ +- Make the version output more flexible (#252) +- List all available mapsets (issue 162, #249) +- Flake8 Linting for test folder (#255) + +______________________________________________________________________ +## [1.2.1](https://github.com/actinia-org/actinia-core/releases/tag/1.2.1) - 2021-09-21 -## [1.2.1] - 2021-09-21 released from main + ### Fixed -* Add band_names property to STRDSInfoModel (#243) for g79 support -## [1.2.0] - 2021-09-07 +- Add band_names property to STRDSInfoModel (#243) for g79 support + +## [1.2.0](https://github.com/actinia-org/actinia-core/releases/tag/1.2.0) - 2021-09-07 + released from main + ### Added -* Add json to stdoutParser options (#239) -* Enhance json output (#240) + +- Add json to stdoutParser options (#239) +- Enhance json output (#240) ### Fixed -* Prevent duplicate workflow run for release (#238) +- Prevent duplicate workflow run for release (#238) + +## [1.1.0](https://github.com/actinia-org/actinia-core/releases/tag/1.1.0) - 2021-08-20 -## [1.1.0] - 2021-08-20 released from main + ### Added -* GHA docker (#165) -> no separate release for g79 is needed anymore! -* Integrate tgis in mapset merge (#233) -* Add support for STRDS export (#236) + +- GHA docker (#165) -> no separate release for g79 is needed anymore! +- Integrate tgis in mapset merge (#233) +- Add support for STRDS export (#236) ### Changed -* cleaning documentation in actinia_core/docs/docs (#213) -* add openeo addons (#235) + +- cleaning documentation in actinia_core/docs/docs (#213) +- add openeo addons (#235) ### Fixed -* fix webhook url in message (#231) -* Fix "check service" URL in install instructions (#232) ------------------------------ +- fix webhook url in message (#231) +- Fix "check service" URL in install instructions (#232) + +______________________________________________________________________ +## [1.0.3.79](https://github.com/actinia-org/actinia-core/releases/tag/1.0.3.79) - 2021-07-15 -## [1.0.3.79] - 2021-07-15 released from grass79 See https://github.com/actinia-org/actinia-core/releases/tag/1.0.3 -## [1.0.3] - 2021-07-15 +## [1.0.3](https://github.com/actinia-org/actinia-core/releases/tag/1.0.3) - 2021-07-15 + released from main + ### FIXED -* update CHANGELOG + README (#229) -* add bandref to RasterInfoModel (#230) -## [1.0.2.79] - 2021-07-07 +- update CHANGELOG + README (#229) +- add bandref to RasterInfoModel (#230) + +## [1.0.2.79](https://github.com/actinia-org/actinia-core/releases/tag/1.0.2.79) - 2021-07-07 + released from grass79 See https://github.com/actinia-org/actinia-core/releases/tag/1.0.2 -## [1.0.2] - 2021-07-07 +## [1.0.2](https://github.com/actinia-org/actinia-core/releases/tag/1.0.2) - 2021-07-07 + released from main + ### Changed -* update actinia-module-plugin version (#228) -## [1.0.1.79] - 2021-06-29 +- update actinia-module-plugin version (#228) + +## [1.0.1.79](https://github.com/actinia-org/actinia-core/releases/tag/1.0.1.79) - 2021-06-29 + released from grass79 See https://github.com/actinia-org/actinia-core/releases/tag/1.0.1 -## [1.0.1] - 2021-06-24 +## [1.0.1](https://github.com/actinia-org/actinia-core/releases/tag/1.0.1) - 2021-06-24 + released from main + ### Changed + - update module importing in `scripts` to `core/common` folder format (https://github.com/actinia-org/actinia-core/pull/224) -## [1.0.0] - 2021-06-02 +## [1.0.0](https://github.com/actinia-org/actinia-core/releases/tag/1.0.0) - 2021-06-02 + released from main Happy Codesprint release :) https://github.com/actinia-org/actinia-core/projects/1 ### Breaking -* Local GeoTIFF import (#216) + +- Local GeoTIFF import (#216) ### Added -* Get all mapsets locks (#200) -* gource script for code development visualization (#219) + +- Get all mapsets locks (#200) +- gource script for code development visualization (#219) ### Changed -* rename master to main (#194) -* Part of issue #190 - * Move common folder one level up + use absolute imports (#199) - * Rename resources folder folder to rest + use absolute imports (#207) - * Create models folder (#208) - * move common to subfolder core (#215) - * update folders in scripts (#220) - * Move modules used by plugins (#221) - * Update plugin versions (#222) - * ATTENTION: the renaming of modules leads to an error with old redis resource entries, see `Unpickling of resources created with actinia < 1.0.0 fails with actinia > 1.0.0` #226 -* Part of issue #189 - * Splitup ephemeral processing (#196) - * splitup methods in aws_sentinel_interface (#205) - * Splitup common folder (part 1) (#209) -* change GHA on push and pr (#217) + +- rename master to main (#194) +- Part of issue #190 + - Move common folder one level up + use absolute imports (#199) + - Rename resources folder folder to rest + use absolute imports (#207) + - Create models folder (#208) + - move common to subfolder core (#215) + - update folders in scripts (#220) + - Move modules used by plugins (#221) + - Update plugin versions (#222) + - ATTENTION: the renaming of modules leads to an error with old redis resource entries, see `Unpickling of resources created with actinia < 1.0.0 fails with actinia > 1.0.0` #226 +- Part of issue #189 + - Splitup ephemeral processing (#196) + - splitup methods in aws_sentinel_interface (#205) + - Splitup common folder (part 1) (#209) +- change GHA on push and pr (#217) ### Fixed -* fix docker build (#187) -* make better use of cache for test dockerimage (#198) -* Fix docker test permissions (#201) -* lint (#203) + +- fix docker build (#187) +- make better use of cache for test dockerimage (#198) +- Fix docker test permissions (#201) +- lint (#203) ### Documentation -* Conversion of Sphinx based tutorial to markdown/mkdocs (#204) -* Documentation update (#210) -* update docker readme (#218) +- Conversion of Sphinx based tutorial to markdown/mkdocs (#204) +- Documentation update (#210) +- update docker readme (#218) +______________________________________________________________________ ------------------------------ +## [0.99.29](https://github.com/actinia-org/actinia-core/releases/tag/0.99.29) - 2021-05-05 + +released from master -## [0.99.29] - 2021-05-05 -released from master\ ### Changed + - Update actinia-module-plugin version (#185) - Will include GRASS GIS version with fix for XY location (https://github.com/OSGeo/grass/pull/1564) ## Warning -Due to a conflict between 0.99.28 and actinia-module-plugin < 2.1.1, the plugin will not work as expected. This is fixed in version `0.99.29`. Also the fix for XY location in GRASS GIS is included in alpine build. -## [0.99.29.79] - 2021-05-05 +Due to a conflict between 0.99.28 and actinia-module-plugin \< 2.1.1, the plugin will not work as expected. This is fixed in version `0.99.29`. Also the fix for XY location in GRASS GIS is included in alpine build. + +## [0.99.29.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.29.79) - 2021-05-05 + released from grass79 + ### see 0.99.29 +## [0.99.28](https://github.com/actinia-org/actinia-core/releases/tag/0.99.28) - 2021-05-04 -## [0.99.28] - 2021-05-04 released from master ### Added + - introducing changelog (#183) - Job resumption (#143) ### Changed + - Alpine Docker: update to snap 8.0.3 (#179) - Update actinia module plugin to 2.1.0 (#182) - raise AsyncProcessError if mapset-lock to be checked does not exist or mapset to be unlocked does not exist (#178) ### Fixed + - Create the config file in case it doesn't exist yet (#173) +## [0.99.28.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.28.79) - 2021-05-04 -## [0.99.28.79] - 2021-05-04 released from grass79 + ### see 0.99.28 ----------- autogenerated below - to change content, change release notes and recreate with command at end of this file ------------ -## [0.99.27.79] - 2021-04-13 +## [0.99.27.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.27.79) - 2021-04-13 + released from grass79 -* GHA Tests (#166) -* update actinia-module-plugin version (https://github.com/actinia-org/actinia-module-plugin/releases/tag/2.0.0) -## [0.99.27] - 2021-04-13 +- GHA Tests (#166) +- update actinia-module-plugin version (https://github.com/actinia-org/actinia-module-plugin/releases/tag/2.0.0) + +## [0.99.27](https://github.com/actinia-org/actinia-core/releases/tag/0.99.27) - 2021-04-13 + released from master -* GHA Tests (#166) -* update actinia-module-plugin version (https://github.com/actinia-org/actinia-module-plugin/releases/tag/2.0.0) -## [0.99.26.79] - 2021-04-01 +- GHA Tests (#166) +- update actinia-module-plugin version (https://github.com/actinia-org/actinia-module-plugin/releases/tag/2.0.0) + +## [0.99.26.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.26.79) - 2021-04-01 + released from grass79 -* ace, importer, exporter: move out to separate repos (#160) -* replace webhook pw with XXX (#163) -* allow redis config via environment variable (#164) -## [0.99.26] - 2021-04-01 +- ace, importer, exporter: move out to separate repos (#160) +- replace webhook pw with XXX (#163) +- allow redis config via environment variable (#164) + +## [0.99.26](https://github.com/actinia-org/actinia-core/releases/tag/0.99.26) - 2021-04-01 + released from master -* use non-root user for prod docker example (#155) -* ace, importer, exporter: move out to separate repos (#160) -* replace webhook pw with XXX (#163) -* allow redis config via environment variable (#164) -## [0.99.25.79] - 2021-03-19 +- use non-root user for prod docker example (#155) +- ace, importer, exporter: move out to separate repos (#160) +- replace webhook pw with XXX (#163) +- allow redis config via environment variable (#164) + +## [0.99.25.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.25.79) - 2021-03-19 + released from grass79 + - fix landsat test (#159) - actinia export formats: minor cleanup (#157) -## [0.99.25] - 2021-03-19 +## [0.99.25](https://github.com/actinia-org/actinia-core/releases/tag/0.99.25) - 2021-03-19 + released from master + - fix landsat test (#159) - actinia export formats: minor cleanup (#157) -## [0.99.24] - 2021-03-11 +## [0.99.24](https://github.com/actinia-org/actinia-core/releases/tag/0.99.24) - 2021-03-11 + released from master + - Fix several misspelled words (#152) - Don't use mutable default arguments when avoidable (#153) - Enhance redis connection error logging + fixes clear text passwords (#154) @@ -667,54 +833,73 @@ released from master - Flake8 linting (#146) - Fix missing import of logging.handlers (#156) -## [0.99.23] - 2021-03-08 +## [0.99.23](https://github.com/actinia-org/actinia-core/releases/tag/0.99.23) - 2021-03-08 + released from master Add stdout as `process_result` for endpoint `locations/{location}}/mapset/{mapset}/processing_async` (#149) + ## Warning + Due to a backporting issue, the `alpine` and `ubuntu` versions of `actinia_core:0.99.23` have a problem in the output of `g.proj -ulgp` where corner coordinates in degrees (`nw_long` etc.) will have very large values or `inf`. This is fixed in version `0.99.24` -## [0.99.22.79] - 2021-02-25 +## [0.99.22.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.22.79) - 2021-02-25 + released from grass79 -* Split actinia-gdi into multiple plugins -* Save interim results -## [0.99.22] - 2021-02-25 +- Split actinia-gdi into multiple plugins +- Save interim results + +## [0.99.22](https://github.com/actinia-org/actinia-core/releases/tag/0.99.22) - 2021-02-25 + released from master + - Split actinia-gdi into multiple plugins - Update deployment + dev setup for plugins (#147) - docker: update module-plugin (#148) -## [0.99.21.79] - 2021-02-16 +## [0.99.21.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.21.79) - 2021-02-16 + released from grass79 ~~Save interim results~~ -## [0.99.21] - 2021-02-16 +## [0.99.21](https://github.com/actinia-org/actinia-core/releases/tag/0.99.21) - 2021-02-16 + released from master + - Save interim results (#137) -## [0.99.20] - 2021-02-16 +## [0.99.20](https://github.com/actinia-org/actinia-core/releases/tag/0.99.20) - 2021-02-16 + released from master + - Update snappy version to 8.0.1 (#145) -## [0.99.20.79] - 2021-02-16 +## [0.99.20.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.20.79) - 2021-02-16 + released from grass79 Update snappy version -## [0.99.19] - 2021-01-22 +## [0.99.19](https://github.com/actinia-org/actinia-core/releases/tag/0.99.19) - 2021-01-22 + released from master -* repair actinia-user (#136) -* update docker readme and setup (#138) -* GHA: add flake8 tests (#140) -* PEP8 fixes (#139), (#142) -* GRASS GIS updates, including json parser: fix mapsets in tokenizing (#1252) -## [0.99.18.79] - 2020-12-22 +- repair actinia-user (#136) +- update docker readme and setup (#138) +- GHA: add flake8 tests (#140) +- PEP8 fixes (#139), (#142) +- GRASS GIS updates, including json parser: fix mapsets in tokenizing (#1252) + +## [0.99.18.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.18.79) - 2020-12-22 + released from grass79 -* more version information -* Ubuntu image update -## [0.99.18] - 2020-12-22 +- more version information +- Ubuntu image update + +## [0.99.18](https://github.com/actinia-org/actinia-core/releases/tag/0.99.18) - 2020-12-22 + released from master + - docker: update of base image to Ubuntu 20.04 (#128) - Create codeql-analysis.yml (#129) - Enhance version output (#132) @@ -722,50 +907,66 @@ released from master - clean version - update log level + paths -## [0.99.17.79] - 2020-12-16 +## [0.99.17.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.17.79) - 2020-12-16 + released from grass79 Release with GRASS GIS 7.9 -## [0.99.17] - 2020-12-15 +## [0.99.17](https://github.com/actinia-org/actinia-core/releases/tag/0.99.17) - 2020-12-15 + released from master + ## Fixed + - vector_layers add zone to properties (#126) -## [0.99.16.79] - 2020-12-11 +## [0.99.16.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.16.79) - 2020-12-11 + released from grass79 Same as 0.99.15 but based on GRASS GIS 7.9 + - Nothing changed in actinia but GRASS GIS (libgis: fix reading of WKT and SRID https://github.com/OSGeo/grass/commit/9c9d19ecccc54db369edad9fd72e9fb5121af6ae) -## [0.99.15.79] - 2020-12-03 +## [0.99.15.79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.15.79) - 2020-12-03 + released from grass79 Same as 0.99.15 but based on GRASS GIS 7.9 -## [0.99.15] - 2020-12-03 +## [0.99.15](https://github.com/actinia-org/actinia-core/releases/tag/0.99.15) - 2020-12-03 + released from master + - Update GRASS GIS because of g.extension branch parameter - Move requirements-alpine.txt to requirements.txt (#125) -## [0.99.14-g79] - 2020-11-18 +## [0.99.14-g79](https://github.com/actinia-org/actinia-core/releases/tag/0.99.14-g79) - 2020-11-18 + released from grass79 Same as 0.99.14 but based on GRASS GIS 7.9 -## [0.99.14] - 2020-11-18 +## [0.99.14](https://github.com/actinia-org/actinia-core/releases/tag/0.99.14) - 2020-11-18 + released from master + - Tests running inside docker build (#120) - start.sh: silence error when GRASS GIS DB already created (#123) - enhance documentation (#122) - Pin versions for proj-using python libs (#124) -## [0.99.13] - 2020-10-23 +## [0.99.13](https://github.com/actinia-org/actinia-core/releases/tag/0.99.13) - 2020-10-23 + released from master + - PEP8 fixes (#113) - Merge Ubuntu docker folders to g78 (#116) - Fix docker folder names (#117) - update gitignore and re-add ubuntu config (#118) - update snap version to 8.0 (#121) -## [0.99.12] - 2020-09-29 +## [0.99.12](https://github.com/actinia-org/actinia-core/releases/tag/0.99.12) - 2020-09-29 + released from master + - add colorlog and psutil to ubuntu19 requirements (#103) - Fix groups (#102), (#104) - return codes: explanations added (#107) @@ -775,21 +976,29 @@ released from master - PEP8: fix errors (#112) - raster_exporter: COG support added (#108) -## [0.99.11] - 2020-08-24 +## [0.99.11](https://github.com/actinia-org/actinia-core/releases/tag/0.99.11) - 2020-08-24 + released from master + - webhook status can be 200 and 204 (#100) + ## Fixed + - group (in persistent processing) - add group to copied directories (#101) +## [0.99.10](https://github.com/actinia-org/actinia-core/releases/tag/0.99.10) - 2020-08-04 -## [0.99.10] - 2020-08-04 released from master + ## Fixed + - allow webhook pw with colon (#98) - webhook pw with colon (#99) -## [0.99.9] - 2020-07-30 +## [0.99.9](https://github.com/actinia-org/actinia-core/releases/tag/0.99.9) - 2020-07-30 + released from master + - use actinia-gdi version 0.1.8 - fix gdal for ubuntu docker (#90) - update version in actinia-gdi alpine Dockerfile (0.1.9, 0.1.10, 0.1.11) (#91), (#93), (#94) @@ -798,39 +1007,51 @@ released from master - add py3-joblib after requirements (#96) - Update requirements.txt and add few tests (#97) -## [latest] - 2020-05-13 -released from master +## [0.99.8](https://github.com/actinia-org/actinia-core/releases/tag/0.99.8) - 2020-05-13 -## [0.99.8] - 2020-05-13 released from master + ## Fixed + - Fix missing openeo-addons repo cloning (#88) - Messages: fix Sentinel 2A to Sentinel-2 (#87) + ### Added + - Stdout Logger with optional colored or json formatter - Enable stdout logging (#47) -## [0.99.7] - 2020-04-16 +## [0.99.7](https://github.com/actinia-org/actinia-core/releases/tag/0.99.7) - 2020-04-16 + released from master + - Switch alpine docker to alpine 3.11 and compiled pdal (#84) - exporter: generate overviews with LZW compression (#85) - use s1 toolbox from snappy only (#86) - update actinia-gdi version in Dockerfile (0.1.6, 0.1.7) (add filtering of variables with default values) -## [0.99.6] - 2020-03-05 +## [0.99.6](https://github.com/actinia-org/actinia-core/releases/tag/0.99.6) - 2020-03-05 + released from master + ## Fixed + - cancel/termination operation for redis version >= 3.0.0 #83 -## [0.99.5] - 2020-02-18 +## [0.99.5](https://github.com/actinia-org/actinia-core/releases/tag/0.99.5) - 2020-02-18 + released from master + - Includes all addons from CSV in alpine dockerimage - Sync grass_addons_list.csv in Dockerfiles (#81) -## [0.99.4] - 2020-02-06 +## [0.99.4](https://github.com/actinia-org/actinia-core/releases/tag/0.99.4) - 2020-02-06 + released from master CI release -## [0.99.3] - 2020-02-06 +## [0.99.3](https://github.com/actinia-org/actinia-core/releases/tag/0.99.3) - 2020-02-06 + released from master + - added band_reference and band_reference for image collections (GRASS GIS 7.9+) (#70) - set ACTINIA_CUSTOM_TEST_CFG to standard actinia cfg dir - torch dependency: avoid hardcoded version (#72) @@ -847,8 +1068,10 @@ released from master - Dockerfile rewrite for Alpine (#55) - add openeo addons in dockerfile (#80) -## [v0.99.2] - 2019-11-28 +## [v0.99.2](https://github.com/actinia-org/actinia-core/releases/tag/v0.99.2) - 2019-11-28 + released from master + - Update to new @ URL delimiter character (#61) - fix ace for e.g. column='z_antenna double precision' parsing (#62) - Enforce pyproj==1.9.6 for PROJ 4.9.3 @@ -863,12 +1086,16 @@ released from master - increase java_max_mem to avoid NullPointer exception for snappy (#64) - include current folder '.' in LD_LIBRARY_PATH for SNAP -## [v0.99.1] - 2019-10-31 +## [v0.99.1](https://github.com/actinia-org/actinia-core/releases/tag/v0.99.1) - 2019-10-31 + released from master + - fix version to not be dirty after release (#59) -## [v0.99.0] - 2019-10-31 +## [v0.99.0](https://github.com/actinia-org/actinia-core/releases/tag/v0.99.0) - 2019-10-31 + released from master + - Enhance READMEs, examples and docker setup - execute tests after actinia installation - optimize gunicorn startup options (#41) @@ -888,55 +1115,76 @@ released from master # 0.2.X -## [v0.2.2] - 2019-09-26 - geopackage-release +## [v0.2.2](https://github.com/actinia-org/actinia-core/releases/tag/v0.2.2) - 2019-09-26 + +- geopackage-release + released from master mainly cleanup of documentation and tests, especially for location names. And support of GeoPackage for importer and exporter!! TODO: enhance description https://github.com/actinia-org/actinia-core/compare/v0.2.1...v0.2.2 -## [v0.2.1] - 2019-05-14 +## [v0.2.1](https://github.com/actinia-org/actinia-core/releases/tag/v0.2.1) - 2019-05-14 + released from master + - let redis use a password in v0.2.1 -## [v0.2.0] - 2019-03-12 +## [v0.2.0](https://github.com/actinia-org/actinia-core/releases/tag/v0.2.0) - 2019-03-12 + released from master + - use python3 for GRASS in v0.2 # 0.1.X -## [v0.1.1] - 2019-03-01 +## [v0.1.1](https://github.com/actinia-org/actinia-core/releases/tag/v0.1.1) - 2019-03-01 + released from master + - last release with python2 for GRASS support -## [python3] - 2019-03-01 +## [python3](https://github.com/actinia-org/actinia-core/releases/tag/python3) - 2019-03-01 + released from master + - merge actinia and grass python requirements for python3 -## [v0.1.0] - 2019-01-24 +## [v0.1.0](https://github.com/actinia-org/actinia-core/releases/tag/v0.1.0) - 2019-01-24 + released from master + - ace and fire # 0.0.X -## [v0.0.7] - 2018-07-16 +## [v0.0.7](https://github.com/actinia-org/actinia-core/releases/tag/v0.0.7) - 2018-07-16 + released from master + - Fixed webhook and Addonpath in v 0.0.7 -## [v0.0.6] - 2018-07-13 +## [v0.0.6](https://github.com/actinia-org/actinia-core/releases/tag/v0.0.6) - 2018-07-13 + released from master + - Add GRASS Addon support in v 0.0.6 -## [v0.0.5] - 2018-07-10 +## [v0.0.5](https://github.com/actinia-org/actinia-core/releases/tag/v0.0.5) - 2018-07-10 + released from master + - More webhooks + postgis export in v 0.0.5 +______________________________________________________________________ ------ Autogenerated with: + ``` -curl https://api.github.com/repos/mundialis/actinia_core/releases?per_page=50 | jq -r '.[] | "## [\(.tag_name)] - \(.published_at | strptime("%Y-%m-%dT%H:%M:%SZ") | strftime("%Y-%m-%d"))\nreleased from \(.target_commitish)\n\(.body) \n"' +curl https://api.github.com/repos/actinia-org/actinia-core/releases?per_page=50 | jq -r '.[] | "## [\(.tag_name)](https://github.com/actinia-org/actinia-core/releases/tag/\(.tag_name)) - \(.published_at | strptime("%Y-%m-%dT%H:%M:%SZ") | strftime("%Y-%m-%d"))\nreleased from \(.target_commitish)\n\(.body) \n"' ``` + After that, few releases need to be sorted correctly. Sorting is not 100% correct, default is per publish date and `?order_by=created_at` doesn't behave as expected. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5b278354c..761026f1b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,12 +23,12 @@ Actinia has been mainly developed by open source GIS developers of the GRASS GIS Since the project is open source, all users, regardless of the profession or skill level, have the ability to contribute to actinia. Here's a few suggestion on how: -* Help actinia users that is less experienced than yourself -* Write a bug report -* Request a new feature -* Write documentation for your favorite actinia usage -* Fix a bug -* Implement a new feature +- Help actinia users that is less experienced than yourself +- Write a bug report +- Request a new feature +- Write documentation for your favorite actinia usage +- Fix a bug +- Implement a new feature In the following sections you can find some guidelines on how to contribute. As actinia is managed on GitHub most contributions require that you have a GitHub @@ -49,11 +49,11 @@ thorough bug report. A good bug report includes at least: -* A title that quickly explains the problem -* A description of the problem and how it can be reproduced -* Version of actinia being used -* Version numbers of any other relevant software being used, e.g. operating system -* A description of what already has been done to solve the problem +- A title that quickly explains the problem +- A description of the problem and how it can be reproduced +- Version of actinia being used +- Version numbers of any other relevant software being used, e.g. operating system +- A description of what already has been done to solve the problem The more information that is given up front, the more likely it is that a developer will find interest in solving the problem. You will probably get follow-up questions @@ -89,6 +89,7 @@ on GitHub. ## Code contributions ### Legalese + Committers are the front line gatekeepers to keep the code base clear of improperly contributed code. It is important to the actinia users, developers and the OSGeo community to avoid contributing any code to the project without it being clearly licensed under the project license. @@ -114,18 +115,18 @@ All unusual situations need to be discussed and/or documented. Committer should adhere to the following guidelines, and may be personally legally liable for improperly contributing code to the source repository: -* Make sure the contributor (and possibly employer) is aware of the contribution terms. -* Code coming from a source other than the contributor (such as adapted from another project) +- Make sure the contributor (and possibly employer) is aware of the contribution terms. +- Code coming from a source other than the contributor (such as adapted from another project) should be clearly marked as to the original source, copyright holders, license terms and so forth. This information can be in the file headers, but should also be added to the project licensing file if not exactly matching normal project licensing (LICENSE.txt). -* Existing copyright headers and license text should never be stripped from a file. If a copyright +- Existing copyright headers and license text should never be stripped from a file. If a copyright holder wishes to give up copyright they must do so in writing to the project steering committee before copyright messages are removed. If license terms are changed it has to be by agreement (written in email is ok) of the copyright holders. -* When substantial contributions are added to a file (such as substantial patches) the +- When substantial contributions are added to a file (such as substantial patches) the author/contributor should be added to the list of copyright holders for the file. -* If there is uncertainty about whether a change is proper to contribute to the code base, please +- If there is uncertainty about whether a change is proper to contribute to the code base, please seek more information from the project steering committee. ### Git workflow with actinia @@ -145,8 +146,8 @@ Details here... ### Additional Resources -* [General GitHub documentation](https://help.github.com/) -* [GitHub pull request documentation](https://help.github.com/articles/about-pull-requests/) +- [General GitHub documentation](https://help.github.com/) +- [GitHub pull request documentation](https://help.github.com/articles/about-pull-requests/) ## Acknowledgements diff --git a/README.md b/README.md index 8b6779562..93de0d591 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,6 @@ Article [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2631917.svg)](https: [![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) [![Docker pulls](https://img.shields.io/docker/pulls/mundialis/actinia-core.svg)](https://hub.docker.com/r/mundialis/actinia-core) - **[Actinia](https://actinia.mundialis.de/)** is an open source REST API for scalable, distributed, high performance processing of geographical data that mainly uses [GRASS GIS](https://grass.osgeo.org/) for computational tasks. @@ -20,6 +19,7 @@ Sentinel-2 scenes in an ephemeral databases. The computational results of epheme are available via object storage as GeoTIFF files. ## API documentation + -* User defined processing - * The actinia process chain - * Sentinel-2A NDVI process chain -* License -* Authors + +- User defined processing + - The actinia process chain + - Sentinel-2A NDVI process chain +- License +- Authors diff --git a/docs/docs/installation.md b/docs/docs/installation.md index 2fe4b5ed4..83d7b4161 100644 --- a/docs/docs/installation.md +++ b/docs/docs/installation.md @@ -1,8 +1,6 @@ -Installation -============ +# Installation -Installation with docker ------------------------- +## Installation with docker Requirements: docker and docker-compose @@ -13,17 +11,16 @@ git clone https://github.com/mundialis/actinia-core.git cd actinia-core docker-compose -f docker/docker-compose.yml up ``` + Now you have a running actinia instance locally! Check with + ``` curl http://127.0.0.1:8088/api/v3/version ``` -* For how to add data to this setup, how to run actinia locally with most recent GRASS GIS or how to fix common startup errors, see more details in [docker README](https://github.com/actinia-org/actinia-core/blob/main/docker/README.md). - +- For how to add data to this setup, how to run actinia locally with most recent GRASS GIS or how to fix common startup errors, see more details in [docker README](https://github.com/actinia-org/actinia-core/blob/main/docker/README.md). - -Local installation ------------------- +## Local installation A local installation requires a full GRASS GIS installation. Make sure all requirements are fulfilled to compile GRASS GIS from the git @@ -32,19 +29,17 @@ repository. These installation instructions are based on a Ubuntu Linux system (other operating systems and distributions are fine, too). - First install PROJ and GRASS GIS by downloading the latest packages and GRASS GIS git version and compile it. -1. Install the latest projection library +1. Install the latest projection library ```bash sudo apt-get update sudo apt-get install proj-bin proj-data ``` - -2. Set Python 3 environment +2. Set Python 3 environment Note that this step changes `python` to point to `python3`, system-wide: @@ -53,8 +48,7 @@ Note that this step changes `python` to point to `python3`, system-wide: sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1 ``` - -3. Compile and install GRASS GIS and additional modules: +3. Compile and install GRASS GIS and additional modules: ```bash # or some other location of your choice, /tmp will not keep data between reboots @@ -116,8 +110,7 @@ cd ../.. sudo make install ``` - -4. Download the test datasets ("locations") and place them into a +4. Download the test datasets ("locations") and place them into a specific directory that will be used by actinia as a persistent database: ```bash @@ -138,8 +131,7 @@ rm -f LL.tar.gz mv LL latlong_wgs84 ``` - -5. Actinia Core must be installed and run within a (virtual) Python3 environment: +5. Actinia Core must be installed and run within a (virtual) Python3 environment: ```bash mkdir -p $HOME/actinia/workspace/tmp @@ -152,8 +144,7 @@ pip install -r requirements.txt python setup.py install ``` - -6. Actinia API must be installed and run within a (virtual) Python3 environment: +6. Actinia API must be installed and run within a (virtual) Python3 environment: ```bash mkdir -p $HOME/actinia/workspace/tmp @@ -165,8 +156,7 @@ cd actinia-api python setup.py install ``` - -7. Make sure that a redis service is running and create a user with +7. Make sure that a redis service is running and create a user with actinia-user tool ```bash @@ -183,8 +173,7 @@ actinia-user create -u superadmin -w abcdefgh -r superadmin -g admin -c 10000000 actinia-server --host 0.0.0.0 --port 5000 ``` - -8. Check the service with curl +8. Check the service with curl ```bash ACTINIA_VERSION="v3" diff --git a/docs/docs/introduction.md b/docs/docs/introduction.md index 58e9feec3..0dc3a4df8 100644 --- a/docs/docs/introduction.md +++ b/docs/docs/introduction.md @@ -1,5 +1,4 @@ -Introduction -============ +# Introduction Actinia is a REST service to process geographical data that can be managed by the GRASS GIS software system. The software is designed to @@ -23,8 +22,7 @@ modules. The URLs that provide access to the GRASS database reflect these concepts. Hence, the location, the mapset and the required raster map are part of the URL to access the service. -What is REST? -------------- +## What is REST? The Representational state transfer (REST)[^4] is an architectural style based on HTTP[^5] that uses GET[^6], DELETE[^7], POST[^8] and PUT[^9] @@ -34,8 +32,8 @@ While GET requests can be send easily from a browser, POST, PUT or DELETE request can not. To access the full potential of actinia you will need a HTTP client, that talks all HTTP communication methods. -Examples --------- +## Examples + We will use the Unix shell and curl to access the REST API. First open a shell of choice (we use bash here) and setup the login information, the IP address and the port on which the actinia service is running, so you @@ -56,12 +54,13 @@ List all locations that are available in the actinia persistent database: curl ${AUTH} -X GET "${ACTINIA_URL}/locations" ``` -List all mapsets in the location latlong\_wgs84: +List all mapsets in the location latlong_wgs84: ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/latlong_wgs84/mapsets" ``` -List all raster layers in location latlong\_wgs84 and mapset Sentinel2A: + +List all raster layers in location latlong_wgs84 and mapset Sentinel2A: ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/latlong_wgs84/mapsets/Sentinel2A/raster_layers" @@ -90,7 +89,7 @@ curl ${AUTH} -X POST "${ACTINIA_URL}/landsat_process/LC80440342016259LGN00/TOAR/ ``` NDVI computation of Sentinel-2A scene -S2A\_MSIL1C\_20170212T104141\_N0204\_R008\_T31TGJ\_20170212T104138: +S2A_MSIL1C_20170212T104141_N0204_R008_T31TGJ_20170212T104138: ```bash curl ${AUTH} -X POST "${ACTINIA_URL}/sentinel2_process/ndvi/S2A_MSIL1C_20170212T104141_N0204_R008_T31TGJ_20170212T104138" @@ -101,20 +100,12 @@ file in a cloud storage for download. **Footnotes** -[^1]: - -[^2]: - -[^3]: - -[^4]: - -[^5]: - -[^6]: - -[^7]: - -[^8]: - -[^9]: +[^1]: https://grass.osgeo.org/ +[^2]: https://en.wikipedia.org/wiki/Representational_State_Transfer +[^3]: https://grass.osgeo.org/grass-stable/manuals/helptext.html#2.-background:-grass-gis-location-structure +[^4]: https://en.wikipedia.org/wiki/Representational_State_Transfer +[^5]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol +[^6]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods +[^7]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods +[^8]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods +[^9]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods diff --git a/docs/docs/license.md b/docs/docs/license.md index fde0b1f58..63a35f04f 100644 --- a/docs/docs/license.md +++ b/docs/docs/license.md @@ -1,9 +1,7 @@ -License -======= +# License GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 - See: https://raw.githubusercontent.com/mundialis/actinia_core/main/LICENSE.txt diff --git a/docs/docs/tutorial_data_access.md b/docs/docs/tutorial_data_access.md index 3b7692bc8..b86d91f56 100644 --- a/docs/docs/tutorial_data_access.md +++ b/docs/docs/tutorial_data_access.md @@ -1,5 +1,4 @@ -Tutorial -======== +# Tutorial In the following tutorial we will access the persistent database analysing raster and raster-time-series data. We will use the import and @@ -12,9 +11,7 @@ line tool **curl**[^1]. **Curl** should be available on many Linux systems. However, tools like *postman*[^2] allow a more comfortable way to access actinia. - -Using curl for HTTP requests ----------------------------- +## Using curl for HTTP requests We will use the Unix shell and curl to access the REST API. First open a shell of choice (we use bash here) and setup the login information, the @@ -27,8 +24,8 @@ export ACTINIA_URL=https://actinia.mundialis.de/latest export AUTH='-u demouser:gu3st!pa55w0rd' # other user credentials can be provided in the same way ``` -Access to locations and mapsets in the persistent database ----------------------------------------------------------- + +## Access to locations and mapsets in the persistent database The following API call lists all available locations in the actinia persistent database (the `-i` includes the HTTP response headers): @@ -51,7 +48,7 @@ The output should look similar to this: ``` To show the region settings and the projection of the GRASS GIS standard -location *nc\_spm\_08* the following REST call must be used: +location *nc_spm_08* the following REST call must be used: ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/nc_spm_08/info" @@ -60,9 +57,9 @@ location *nc\_spm\_08* the following REST call must be used: The JSON response is the standard response of the actinia REST API. Most API calls respond using this JSON structure. The difference between API calls is the result part that is located in the JSON section with the -name *process\_results*. The response includes all steps that were +name *process_results*. The response includes all steps that were executed to receive the projection information and the region -information. It is located in the *process\_log* section of the JSON +information. It is located in the *process_log* section of the JSON response. In addition API specific information as well as the processing time are available in the response. To make the JSON response more readable you can use parsing-tools like jq. @@ -160,7 +157,7 @@ The output should look similar to this then: } ``` -To list all mapsets located in the location *nc\_spm\_08* the following +To list all mapsets located in the location *nc_spm_08* the following API call is used: ```bash @@ -168,7 +165,7 @@ API call is used: ``` The response of this synchronous call lists all mapsets of the location -in the *process\_results* section: +in the *process_results* section: ```json { @@ -239,7 +236,7 @@ Using the following API call will show all information about the mapset ``` The response shows the region of the mapset and the projection of the -location in the *process\_results* section: +location in the *process_results* section: ```json { @@ -333,8 +330,7 @@ location in the *process\_results* section: } ``` -Access to raster layers in the persistent database --------------------------------------------------- +## Access to raster layers in the persistent database The location ECAD contains yearly climate data (precipitation and temperature) of Europe for 60 years. We list all raster layers of the @@ -345,7 +341,7 @@ location ECAD in mapset *PERMANENT*: ``` The response lists all raster layers of the mapset in the -*process\_results* section: +*process_results* section: ```json { @@ -415,14 +411,14 @@ The response lists all raster layers of the mapset in the ``` Show info about the raster layer -*temperature\_mean\_yearly\_celsius\_60*: +*temperature_mean_yearly_celsius_60*: ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/ECAD/mapsets/PERMANENT/raster_layers/temperature_mean_yearly_celsius_60" ``` The response lists information about the raster layer -*temperature\_mean\_yearly\_celsius\_60* in the *process\_results* +*temperature_mean_yearly_celsius_60* in the *process_results* section: ```json @@ -509,8 +505,7 @@ section: } ``` -Access to raster time-series in the persistent database -------------------------------------------------------- +## Access to raster time-series in the persistent database Actinia supports the analysis of time-series data based on the temporal framework of GRASS GIS[^3], [^4]. A time-series datatype is located in @@ -518,14 +513,13 @@ location *ECAD* with mapsets *PERMANENT*. The time-series datatype is called space-time raster dataset (strds) and represents a time-stamped series of yearly temperature and precipitation data for Europe. - We list all strds with the following API call: ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/ECAD/mapsets/PERMANENT/strds" ``` -We receive two strds in the *process\_results* section of the JSON +We receive two strds in the *process_results* section of the JSON response: ```json @@ -592,15 +586,15 @@ response: ``` Use the following API call to receive information about the strds -*temperature\_mean\_1950\_2013\_yearly\_celsius*. +*temperature_mean_1950_2013_yearly_celsius*. ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/ECAD/mapsets/PERMANENT/strds/temperature_mean_1950_2013_yearly_celsius" ``` All relevant information about strds -*temperature\_mean\_1950\_2013\_yearly\_celsius* is located in the -*process\_results* section of the JSON response: +*temperature_mean_1950_2013_yearly_celsius* is located in the +*process_results* section of the JSON response: ```json { @@ -691,14 +685,14 @@ All relevant information about strds ``` List all raster layers that are registered in the strds -*temperature\_mean\_1950\_2013\_yearly\_celsius* with time-stamps: +*temperature_mean_1950_2013_yearly_celsius* with time-stamps: ```bash curl ${AUTH} -X GET "${ACTINIA_URL}/locations/ECAD/mapsets/PERMANENT/strds/temperature_mean_1950_2013_yearly_celsius/raster_layers" ``` A list of about 60 raster layers with minimum, maximum values, -time-stamps and spatial extent will be located in the *process\_results* +time-stamps and spatial extent will be located in the *process_results* section of the JSON response: ```json @@ -799,10 +793,7 @@ section of the JSON response: **Footnotes** -[^1]: - -[^2]: - -[^3]: - -[^4]: +[^1]: https://en.wikipedia.org/wiki/CURL +[^2]: https://www.getpostman.com/apps +[^3]: http://www.sciencedirect.com/science/article/pii/S136481521300282X +[^4]: http://www.tandfonline.com/doi/abs/10.1080/13658816.2017.1306862?journalCode=tgis20 diff --git a/docs/docs/tutorial_landsat_ndvi.md b/docs/docs/tutorial_landsat_ndvi.md index c4c37b889..d12824393 100644 --- a/docs/docs/tutorial_landsat_ndvi.md +++ b/docs/docs/tutorial_landsat_ndvi.md @@ -1,19 +1,18 @@ -Landsat NDVI computation -======================== +# Landsat NDVI computation Actinia provides several API calls to compute satellite specific parameters: - + - We will use the Unix shell and curl to access the REST API. First open a shell of choice (we use bash here) and setup the login information, the IP address and the port on which the actinia service is running, so you can simply change the IP and port if your server uses a different - address: +We will use the Unix shell and curl to access the REST API. First open a shell of choice (we use bash here) and setup the login information, the IP address and the port on which the actinia service is running, so you can simply change the IP and port if your server uses a different +address: - ```bash - export ACTINIA_URL=https://actinia.mundialis.de/latest - export AUTH='-u demouser:gu3st!pa55w0rd' - # other user credentials can be provided in the same way - ``` +```bash +export ACTINIA_URL=https://actinia.mundialis.de/latest +export AUTH='-u demouser:gu3st!pa55w0rd' +# other user credentials can be provided in the same way +``` The NDVI is an important parameter that is derived from multi-spectral satellite images. The following asynchronous API call computes the NDVI @@ -23,7 +22,6 @@ scene downloading, reprojection, atmospheric correction, statistical analysis and preview rendering in a single call using a self describing url. - ```bash curl ${AUTH} -X POST "${ACTINIA_URL}/landsat_process/LC80440342016259LGN00/TOAR/NDVI" ``` @@ -68,7 +66,7 @@ calls. ``` The final result will contain a complete processing list as well as -URL\'s to the resulting PNG preview image, the created NDVI GeoTiff file +URL's to the resulting PNG preview image, the created NDVI GeoTiff file as well as univariate statistics of the computed NDVI scene. ```json @@ -669,7 +667,7 @@ as well as univariate statistics of the computed NDVI scene. ``` The statistical analysis of the resulting NDVI raster map layer is -available in the *process\_results* section: +available in the *process_results* section: ```json { @@ -692,10 +690,10 @@ available in the *process\_results* section: The following links can be found in the **urls** section of the JSON response that point to the created resources: - + - + The preview image should look like this: - ![image](landsat_ndvi.png) +![image](landsat_ndvi.png) diff --git a/docs/docs/tutorial_process_chain.md b/docs/docs/tutorial_process_chain.md index 8b684839b..ab3a1f188 100644 --- a/docs/docs/tutorial_process_chain.md +++ b/docs/docs/tutorial_process_chain.md @@ -1,8 +1,6 @@ -User defined processing -======================= +# User defined processing -The actinia process chain -------------------------- +## The actinia process chain Actinia provides the **process chain** approach to specify import, processing and export of geodata using the actinia GRASS GIS processing @@ -23,14 +21,14 @@ that is used as computational environment. A process chain is a list of GRASS GIS modules[^1] that will be executed in serial, based on the order of the list. GRASS GIS modules are specified as process definitions[^2] that include the name of the command, the -inputs and outputs], including import and export definitions as +inputs and outputs\], including import and export definitions as well as the module flags. The following example defines a single process that runs the GRASS GIS module *r.slope.aspect*[^3] to compute the -*slope* for the raster map layer *elev\_ned\_30m* that is located in the +*slope* for the raster map layer *elev_ned_30m* that is located in the mapset[^4] *PERMANENT*. The output of the module is named -*elev\_ned\_30m\_slope* and should be exported as a GeoTiff file. +*elev_ned_30m_slope* and should be exported as a GeoTiff file. ```json { @@ -58,10 +56,10 @@ mapset[^4] *PERMANENT*. The output of the module is named **Import and export** -The actinia process chain supports the specification of URL\'s to raster +The actinia process chain supports the specification of URL's to raster layers in the input definition. The following process chain imports a raster map layer that is located in an object storage with the name -*elev\_ned\_30m\_new* and sets the computational region for the +*elev_ned_30m_new* and sets the computational region for the following processing step with the GRASS GIS module *g.region*[^5]. Then slope and aspect are computed with *r.slope.aspect* and specified for export as GeoTiff files. @@ -110,21 +108,20 @@ export as GeoTiff files. } ``` - **Output parsing** Many GRASS GIS modules produce textual output in form of lists, tables and key/value pairs. Actinia supports the analysis of this data and the parsing and transformation in JSON accessible data. For each GRASS GIS -module the property \"stdout\" can be specified to transform the output +module the property "stdout" can be specified to transform the output of the module into a list of values, a key/value list or a table. The following options are available as format: - - \"kv\" parses the module output and creates key/value pairs - - \"list\" parses the module output and creates a list of values - - \"table\" parses the module output and creates a list of lists - with values aka 2D array aka table +- "kv" parses the module output and creates key/value pairs +- "list" parses the module output and creates a list of values +- "table" parses the module output and creates a list of lists + with values aka 2D array aka table Additionally the unique id that will be used in the response process result dictionary and the delimiter must be defined. @@ -597,10 +594,9 @@ response: ``` The result of the stdout output parsing for each module is located in -the \"process\_results\" section of the json response. +the "process_results" section of the json response. -Sentinel-2A NDVI process chain --- +## Sentinel-2A NDVI process chain We will use the Unix shell and curl to access the REST API. First open a shell of choice (we use bash here) and setup the login information, the IP address and the port on which the actinia service is running, so you can simply change the IP and port if your server uses a different address: @@ -610,9 +606,10 @@ export ACTINIA_URL=https://actinia.mundialis.de/latest export AUTH='-u demouser:gu3st!pa55w0rd' # other user credentials can be provided in the same way ``` + We create a process chain that computes the NDVI from a Sentinel-2A scene based on the bands 8 and 4 with the GRASS GIS module *r.mapcalc*. We -use the North Carolina sample location **nc\_spm\_08** as processing +use the North Carolina sample location **nc_spm_08** as processing environment and the computational region of sentinel band B04 for the NDVI processing. Then we calculate univariate statistics for the Sentinel-2A scene. The computed NDVI raster layer will be exported as @@ -620,13 +617,13 @@ geotiff file that can be accessed via an URL. The following JSON code has 6 process definitions: - 1. Import of two bands (B04 and B08) of the Sentinel-2A scene - *S2A\_MSIL2A\_20220420T154941\_N0400\_R054\_T18SUE\_20220421T000632* - 2. Set the computational region to imported raster layer B04 - 3. Use *r.mapcalc* to compute the NDVI - 4. Use *r.univar* to compute univariate statistics of the computed NDVI - raster layer - 5. Export the computed NDVI as GeoTiff +1. Import of two bands (B04 and B08) of the Sentinel-2A scene + *S2A_MSIL2A_20220420T154941_N0400_R054_T18SUE_20220421T000632* +1. Set the computational region to imported raster layer B04 +1. Use *r.mapcalc* to compute the NDVI +1. Use *r.univar* to compute univariate statistics of the computed NDVI + raster layer +1. Export the computed NDVI as GeoTiff ```json { @@ -1022,16 +1019,13 @@ The finished response should look like this: "user_id": "demouser" } ``` + Use the Link at the end of the response under `"resources"` to download the final NDVI map to your local computer. You can load it into a GIS of your choice, it should look like this: - ![image](nc_s2_ndvi_actinia_process_chain.png) +![image](nc_s2_ndvi_actinia_process_chain.png) **Footnotes** -[^1]: - -[^2]: - -[^3]: - -[^4]: -[^5]: +[^1]: https://grass.osgeo.org/grass-stable/manuals/index.html +[^2]: https://redocly.github.io/redoc/?url=https://actinia.mundialis.de/latest/swagger.json#tag/Module-Viewer/paths/~1grass_modules/get +[^3]: https://grass.osgeo.org/grass-stable/manuals/r.slope.aspect.html +[^4]: https://grass.osgeo.org/grass-stable/manuals/grass_database.html +[^5]: https://grass.osgeo.org/grass-stable/manuals/g.region.html diff --git a/docs/docs/tutorial_sentinel2_ndvi.md b/docs/docs/tutorial_sentinel2_ndvi.md index 7a9ad697c..04fe3ea3b 100644 --- a/docs/docs/tutorial_sentinel2_ndvi.md +++ b/docs/docs/tutorial_sentinel2_ndvi.md @@ -1,5 +1,4 @@ -Sentinel-2 NDVI computation -=========================== +# Sentinel-2 NDVI computation We will use the Unix shell and curl to access the REST API. First open a shell of choice (we use bash here) and setup the login information, the IP address and the port on which the actinia service is running, so you can simply change the IP and port if your server uses a different address: @@ -12,7 +11,7 @@ export AUTH='-u demouser:gu3st!pa55w0rd' The following asynchronous API call computes the NDVI of the Sentinel-2 scene -**S2A\_MSIL1C\_20161206T030112\_N0204\_R032\_T50RKR\_20161206T030749**: +**S2A_MSIL1C_20161206T030112_N0204_R032_T50RKR_20161206T030749**: ```bash curl ${AUTH} -X POST "${ACTINIA_URL}/sentinel2_process/ndvi/S2A_MSIL1C_20161206T030112_N0204_R032_T50RKR_20161206T030749" @@ -56,7 +55,7 @@ Be aware that you have to change the status url as the resource id will change f ``` The final result will contain a complete processing list as well as -URL\'s to the resulting PNG preview image, the created NDVI GeoTiff file +URL's to the resulting PNG preview image, the created NDVI GeoTiff file as well as univariate statistics of the computed NDVI scene. ```json @@ -654,9 +653,9 @@ as well as univariate statistics of the computed NDVI scene. The following URLs point to the resulting PNG preview image and the NDVI GeoTiff file: - - + + The PNG preview should look like this: - ![image](sentinel_ndvi.png) +![image](sentinel_ndvi.png) diff --git a/docs/docs/tutorial_strds_sampling.md b/docs/docs/tutorial_strds_sampling.md index 99620f7dc..5f0e6b709 100644 --- a/docs/docs/tutorial_strds_sampling.md +++ b/docs/docs/tutorial_strds_sampling.md @@ -1,8 +1,6 @@ -Time-series sampling -==================== +# Time-series sampling -Sampling of a STRDS with vector points --------------------------------------- +## Sampling of a STRDS with vector points The actinia REST API provides an API call to sample a space-time raster dataset (STRDS) at coordinate points. The coordinates must be specified @@ -19,7 +17,7 @@ export AUTH='-u demouser:gu3st!pa55w0rd' ``` The following example shows the synchronous API call that samples the -STRDS *temperature\_mean\_1950\_2013\_yearly\_celsius* at three +STRDS *temperature_mean_1950_2013_yearly_celsius* at three different coordinates. ```bash @@ -31,7 +29,7 @@ different coordinates. ``` The result of the sampling is a list of time stamped values, that are -located in the *process\_results* section of the JSON response: +located in the *process_results* section of the JSON response: ```json { diff --git a/docs/gource/README.md b/docs/gource/README.md index 0a513d49b..2c008fa49 100755 --- a/docs/gource/README.md +++ b/docs/gource/README.md @@ -1,6 +1,5 @@ ## Gource: visualization of source code development - ### Installation Gource page: https://github.com/acaudwell/Gource @@ -24,7 +23,6 @@ gource --title "actinia development" -$RES --bloom-intensity 0.5 --camera-mode t --caption-file gource_captions.txt -caption-size 24 --caption-colour FF0066 --caption-duration 4 ../.. ``` - ### Create the animation as a MP4 film ``` diff --git a/job_resumption.md b/job_resumption.md index 090a19574..c46070a4f 100644 --- a/job_resumption.md +++ b/job_resumption.md @@ -6,62 +6,72 @@ If you want to be able to resume a job in actinia, you must enable There are two possible options for this: 1. `save_iterim_results=True`: after each processing step -the temporary mapset in which the calculation is done is saved. -This requires a lot of memory depending on the calculated data and the amount -of calculations. -2. `save_iterim_results=onError` saves only intermediate results if an error -has occurred in the processing. This means that the temporary mapset is only -saved if an error occurs. Here the job resumption can fail, if the failing step -modifies the temporary mapset (e.g. deletion of data which are needed) so that -the fixed module call can not be executed after that modification. In the event -of such an error, the process chain can then be adjusted so that the missing -data is generated again, or the entire process can be started as a new job. + the temporary mapset in which the calculation is done is saved. + This requires a lot of memory depending on the calculated data and the amount + of calculations. +1. `save_iterim_results=onError` saves only intermediate results if an error + has occurred in the processing. This means that the temporary mapset is only + saved if an error occurs. Here the job resumption can fail, if the failing step + modifies the temporary mapset (e.g. deletion of data which are needed) so that + the fixed module call can not be executed after that modification. In the event + of such an error, the process chain can then be adjusted so that the missing + data is generated again, or the entire process can be started as a new job. The interim results will be saved in the resource storage. Which can be configured in the config: + ``` [GRASS] grass_resource_dir = /actinia/resources ``` + The interim results will be deleted automatically if a job resource is successfully resumed. The resource storage can also be cleaned manually via the endpoint (see the example below). ## Dev setup + - adjust config, e.g. for option 1 - ``` - [MISC] - save_interim_results = True - save_interim_results_endpoints_cfg = /etc/default/actinia_interim_endpoints.csv - ``` - or for option 2 - ``` - [MISC] - save_interim_results = onError - save_interim_results_endpoints_cfg = /etc/default/actinia_interim_endpoints.csv - ``` - - The `/etc/default/actinia_interim_endpoints.csv` file can be set if not only - the interim results of the processing endpoints of actinia_core should be saved. - If for example also for the processing endpoints of the actinia-module-plugin - the interim results should be saved and resource resumption should be prossible - the file should contain the following: - ``` - GdiAsyncEphemeralExportResource;AsyncEphemeralExportResource - GdiAsyncPersistentResource;AsyncPersistentResource - ``` - The first entry is the class of the actinia-module-plugin endpoint and the - seconde one is the actinia_core class which is used to do the job resumption. - Since these differ, the complete process chain without template must be - specified for job resumption (see below in the examples). + + ``` + [MISC] + save_interim_results = True + save_interim_results_endpoints_cfg = /etc/default/actinia_interim_endpoints.csv + ``` + + or for option 2 + + ``` + [MISC] + save_interim_results = onError + save_interim_results_endpoints_cfg = /etc/default/actinia_interim_endpoints.csv + ``` + + The `/etc/default/actinia_interim_endpoints.csv` file can be set if not only + the interim results of the processing endpoints of actinia_core should be saved. + If for example also for the processing endpoints of the actinia-module-plugin + the interim results should be saved and resource resumption should be prossible + the file should contain the following: + + ``` + GdiAsyncEphemeralExportResource;AsyncEphemeralExportResource + GdiAsyncPersistentResource;AsyncPersistentResource + ``` + + The first entry is the class of the actinia-module-plugin endpoint and the + seconde one is the actinia_core class which is used to do the job resumption. + Since these differ, the complete process chain without template must be + specified for job resumption (see below in the examples). - Startup actinia with above config in preferred way, e.g. -`cd ~/repos/actinia` + press F5 + `cd ~/repos/actinia` + press F5 ## Additional mapsets + For parallelization on different regions some GRASS GIS processes might create additional mapsets and use the data from these mapsets in further calculations without copying them to the temporary mapsets. To add the possibility to also resumpt jobs where such addional mapsets are created in a previous step you can configure additional mapsets which should be included in the interim results by setting a pattern for the mapset name, e.g.: + ``` [MISC] save_interim_results = onError @@ -69,8 +79,8 @@ save_interim_results_endpoints_cfg = /etc/default/actinia_interim_endpoints.csv include_additional_mapset_pattern = test_tmp_* ``` - ## Job resumption examples + ``` actiniapost() { curl -X POST -H 'Content-Type: application/json' -H 'accept: application/json' -u $1 -d @$2 $3 > resp.json && cat resp.json | jq; export STATUS_URL=$(cat resp.json | jq .urls.status | sed 's/"//g'); curl -L -u $1 $STATUS_URL_POST | jq; echo 'curl -L -u '\'$1\'' $STATUS_URL_POST | jq' @@ -91,6 +101,7 @@ actiniadelete(){ ``` For local setup set e.g.: + ``` export ACTINIA_USER='actinia-gdi' export ACTINIA_PASSWORD='actinia-gdi' @@ -102,11 +113,13 @@ URL=http://127.0.0.1:8088/api/v3 ### Job resumption The job resumption works for the following endpoints: -* ephemeral processing: `ENDPOINT=locations/nc_spm_08/processing_async` -* ephemeral processing with export: `ENDPOINT=locations/nc_spm_08/processing_async_export` -* persistent processing: `ENDPOINT=locations/nc_spm_08/mapsets/test44/processing_async` + +- ephemeral processing: `ENDPOINT=locations/nc_spm_08/processing_async` +- ephemeral processing with export: `ENDPOINT=locations/nc_spm_08/processing_async_export` +- persistent processing: `ENDPOINT=locations/nc_spm_08/mapsets/test44/processing_async` `pc_error.json`: process chain with error + ``` { "list": [ @@ -235,6 +248,7 @@ The job resumption works for the following endpoints: ``` `pc_noerror.json`: fixed process chain + ``` { "list": [ @@ -380,6 +394,7 @@ Here an example process chain where a folder `/test` is requested which does not The process can be resumpt after the folder is created inside actinia. `pc_error_not_in_pc.json`: Process chain with not existing folder `/test` + ``` { "list": [ @@ -428,9 +443,11 @@ curl -L -u $AUTH $STATUS_URL_POST | jq ``` ### Cleaning up resources which are not restarted + If a job is not restarted and the processing is not finished successfully the interim results will not be automatically cleaned. For this you can delete the resource storage: + ``` # get resource storage information actiniaget $AUTH $URL/resource_storage @@ -444,7 +461,9 @@ actiniadelete $AUTH $URL/resource_storage ``` ### Using templates via the actinia-module-plugin + `pc_template_error.json`: process chain with using template index_NDVI + ``` { "list": [ @@ -487,6 +506,7 @@ actiniadelete $AUTH $URL/resource_storage ``` `pc_template_forput.json`: complete process chain without template (can be found in actinia response) : + ``` { "list": [ diff --git a/logging.md b/logging.md index bef892808..f951cd706 100644 --- a/logging.md +++ b/logging.md @@ -2,20 +2,20 @@ Actinia supports logging to fluentd as well as to stdout. ## tl;dr: -* Rename logfile in actinia.cfg in project setups from +- Rename logfile in actinia.cfg in project setups from `worker_logfile = /actinia_core/workspace/tmp/actinia_worker` to `worker_logfile = /actinia_core/workspace/tmp/actinia_worker.log`. If you leave it, it won't break something but would be nice for consistency. -* Adjust logging config as wished (fluentd/stdout with json/stdout colored). +- Adjust logging config as wished (fluentd/stdout with json/stdout colored). If your setup includes Kibana, `log_stdout_format = json` is recommended while `log_stdout_format = colored` provides better human readability. More infos below. - ## Standard Output Logging Infos ### Configuration Example: + ``` [LOGGING] log_interface = stdout @@ -26,32 +26,31 @@ log_level = 4 ``` The `log_interface`s `fluentd` and `stdout` are mutually exclusive, while actinia - will always log to a file. If `log_interface = stdout`, it can be chosen between - `json` output (e.g. for kibana) and `colored` for human readability. - The `log_file_format` has the same options and is independent. The option - `log_stderr_format` is useful for debugging and won't appear in fluentd. In - case this is a stacktrace, it is logged line by line, each wrapped by the log - formatter (either json or stdout, depending on which one is set). As it is - hardly readable and context can get lost, only use with `json` when needed - by e.g. kibana, else use option `plain` for a readable stacktrace. It will also - log to file with configured formatter and log level `CRITICAL`. - - For more information, see - https://github.com/actinia-org/actinia-core/blob/main/src/actinia_core/resources/common/config.py#L132. +will always log to a file. If `log_interface = stdout`, it can be chosen between +`json` output (e.g. for kibana) and `colored` for human readability. +The `log_file_format` has the same options and is independent. The option +`log_stderr_format` is useful for debugging and won't appear in fluentd. In +case this is a stacktrace, it is logged line by line, each wrapped by the log +formatter (either json or stdout, depending on which one is set). As it is +hardly readable and context can get lost, only use with `json` when needed +by e.g. kibana, else use option `plain` for a readable stacktrace. It will also +log to file with configured formatter and log level `CRITICAL`. +For more information, see +https://github.com/actinia-org/actinia-core/blob/main/src/actinia_core/resources/common/config.py#L132. ### Dev / Debug Hints -* one disadvantage of general actinia logging implementation is, that log messages - are passed to `redis_fluentd_logger_base` where they are initially logged. - This way, it always logs as source - `/usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev18+g4d82691.dirty-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:104` - instead of where the message was generated. To change this, refactoring would be necessary. +- one disadvantage of general actinia logging implementation is, that log messages + are passed to `redis_fluentd_logger_base` where they are initially logged. + This way, it always logs as source + `/usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev18+g4d82691.dirty-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:104` + instead of where the message was generated. To change this, refactoring would be necessary. ### Examples - __Simple log__ + ``` { "time": "2020-05-06T09:07:33.431255Z", @@ -66,6 +65,7 @@ __Simple log__ "node": "dc05d9be5c7d" } ``` + ``` [2020-05-08 09:19:33,909] INFO : actinia-core.endpoints -Loading plugin actinia_statistic_plugin [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/endpoints.py:172] ``` @@ -83,11 +83,13 @@ __Simple log__ "threadName": "MainThread" } ``` + ``` [2020-05-08 09:19:48,430] INFO : actinia-core.process_queue -Run process: {'endpoint': 'asyncephemeralexportresource', 'method': 'POST', 'path': '/api/v3/locations/latlong_wgs84/processing_async_export', 'request_url': 'http://127.0.0.1:8088/api/v3/locations/latlong_wgs84/processing_async_export'} [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/resources/common/process_queue.py:355] ``` __Gunicorn log__ + ``` { "time": "2020-05-06T09:07:34.837921Z", @@ -101,11 +103,13 @@ __Gunicorn log__ "threadName": "ThreadPoolExecutor-0_0" } ``` + ``` [2020-05-08 09:19:48,426] INFO : gunicorn.access.glogging -172.20.0.1 - actinia-gdi [08/May/2020:09:19:48 +0000] "POST /api/v3/locations/latlong_wgs84/processing_async_export HTTP/1.1" 200 755 "-" "curl/7.58.0" [in /usr/lib/python3.8/site-packages/gunicorn/glogging.py:344] ``` __messages_logger INFO__ + ``` { "time": "2020-05-06T09:07:34.891945Z", @@ -125,11 +129,13 @@ __messages_logger INFO__ "logger": "messages_logger" } ``` + ``` [2020-05-08 09:19:48,512] INFO : actinia-core.redis_fluentd_logger_base -{'node': '3f3e57b7ab83', 'ctime': 'Fri May 8 09:19:48 2020', 'user_id': 'actinia-gdi', 'log_level': 'INFO', 'status': 'message', 'message': "{'id': 'v_what_vect_1', 'module': 'v.what.vect', 'inputs': [{'param': 'map', 'value': 'point'}, {'param': 'column', 'value': 'isIn'}, {'param': 'query_map', 'value': 'polygon'}, {'param': 'query_column', 'value': 'polygon'}]}", 'logger': 'messages_logger'} [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:99] ``` __api_logger__ + ``` { "time": "2020-05-06T09:07:34.829927Z", @@ -155,11 +161,13 @@ __api_logger__ "logger": "api_logger" } ``` + ``` [2020-05-08 09:19:48,419] INFO : actinia-core.redis_fluentd_logger_base -{'time_stamp': '2020-05-08 09:19:48.419294', 'node': '3f3e57b7ab83', 'api_info': {'endpoint': 'asyncephemeralexportresource', 'method': 'POST', 'path': '/api/v3/locations/latlong_wgs84/processing_async_export', 'request_url': 'http://127.0.0.1:8088/api/v3/locations/latlong_wgs84/processing_async_export'}, 'request_str': "", 'user_id': 'actinia-gdi', 'status': 'api_call', 'logger': 'api_logger'} [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:94] ``` __resources_logger "accepted"__ + ``` { "time": "2020-05-06T09:07:34.835133Z", @@ -195,11 +203,13 @@ __resources_logger "accepted"__ "logger": "resources_logger" } ``` + ``` [2020-05-08 09:19:48,424] INFO : actinia-core.redis_fluentd_logger_base -{'status': 'accepted', 'user_id': 'actinia-gdi', 'resource_id': 'resource_id-53e2a5ae-d4c0-4b9d-8d24-03030a4b314a', 'accept_timestamp': 1588929588.421158, 'accept_datetime': '2020-05-08 09:19:48.421166', 'timestamp': 1588929588.4238856, 'datetime': '2020-05-08 09:19:48.423887', 'message': 'Resource accepted', 'time_delta': 0.002741098403930664, 'process_results': {}, 'process_chain_list': [], 'http_code': 200, 'urls': {'resources': [], 'status': 'http://127.0.0.1:8088/api/v3/resources/actinia-gdi/resource_id-53e2a5ae-d4c0-4b9d-8d24-03030a4b314a'}, 'api_info': {'endpoint': 'asyncephemeralexportresource', 'method': 'POST', 'path': '/api/v3/locations/latlong_wgs84/processing_async_export', 'request_url': 'http://127.0.0.1:8088/api/v3/locations/latlong_wgs84/processing_async_export'}, 'logger': 'resources_logger'} [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:91] ``` __resources_logger "running"__ + ``` { "time": "2020-05-06T09:07:35.188501Z", @@ -238,11 +248,13 @@ __resources_logger "running"__ "logger": "resources_logger" } ``` + ``` [2020-05-08 09:19:48,514] INFO : actinia-core.redis_fluentd_logger_base -{'status': 'running', 'user_id': 'actinia-gdi', 'resource_id': 'resource_id-53e2a5ae-d4c0-4b9d-8d24-03030a4b314a', 'accept_timestamp': 1588929588.421158, 'accept_datetime': '2020-05-08 09:19:48.421166', 'timestamp': 1588929588.5140734, 'datetime': '2020-05-08 09:19:48.514093', 'message': 'Checking access to URL: https://raw.githubusercontent.com/mmacata/pagestest/gh-pages/bonn.geojson', 'time_delta': 0.09295296669006348, 'progress': {'step': 0, 'num_of_steps': 0}, 'process_chain_list': [], 'http_code': 200, 'urls': {'resources': [], 'status': 'http://127.0.0.1:8088/api/v3/resources/actinia-gdi/resource_id-53e2a5ae-d4c0-4b9d-8d24-03030a4b314a'}, 'api_info': {'endpoint': 'asyncephemeralexportresource', 'method': 'POST', 'path': '/api/v3/locations/latlong_wgs84/processing_async_export', 'request_url': 'http://127.0.0.1:8088/api/v3/locations/latlong_wgs84/processing_async_export'}, 'logger': 'resources_logger'} [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:91] ``` __resources_logger "error" shortened__ + ``` { "time": "2020-05-06T09:07:38.357543Z", @@ -307,11 +319,13 @@ __resources_logger "error" shortened__ "logger": "resources_logger" } ``` + ``` [2020-05-08 09:19:54,787] ERROR : actinia-core.redis_fluentd_logger_base -{'status': 'error', 'user_id': 'actinia-gdi', 'resource_id': 'resource_id-53e2a5ae-d4c0-4b9d-8d24-03030a4b314a', 'accept_timestamp': 1588929588.421158, 'accept_datetime': '2020-05-08 09:19:48.421166', 'timestamp': 1588929594.7862213, 'datetime': '2020-05-08 09:19:54.786227', 'message': "[Errno 2] No such file or directory: '/usr/bin/zip'", ..., 'logger': 'resources_logger'} [in /usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev20+gacc075c-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:87] ``` __stderr_logger one random line of stacktrace__ + ``` { "time": "2020-05-06T13:25:01.369275Z", @@ -326,6 +340,7 @@ __stderr_logger one random line of stacktrace__ "node": "dc05d9be5c7d" } ``` + ``` Exception in thread Thread-1 @@ -350,10 +365,8 @@ TypeError an integer is required (got type NoneType) ``` - - - ## Fluentd Logging Infos -* Configure in actinia.cfg the connection to fluentd -* Have an elasticsearch and kibana instance ready -* Start all components and set kibana to look at fluentd-* pattern + +- Configure in actinia.cfg the connection to fluentd +- Have an elasticsearch and kibana instance ready +- Start all components and set kibana to look at fluentd-\* pattern diff --git a/redis_queue.md b/redis_queue.md index 13d31ade8..de3ce577d 100644 --- a/redis_queue.md +++ b/redis_queue.md @@ -1,7 +1,9 @@ # Redis Queue for Jobs ## Dev setup + - adjust config, e.g. + ``` [QUEUE] redis_queue_server_url = redis-queue @@ -9,7 +11,9 @@ redis_queue_server_password = pass worker_prefix = job_queue queue_type = per_job ``` + or + ``` [QUEUE] number_of_workers = 1 @@ -17,8 +21,9 @@ queue_type = redis ``` - Startup actinia with above config in preferred way, e.g. -`cd ~/repos/actinia` + press F5 + `cd ~/repos/actinia` + press F5 - Start Container for worker + ``` MY_ACTINIA_DATA=$HOME/actinia docker run --rm -it --entrypoint sh \ @@ -29,13 +34,14 @@ docker run --rm -it --entrypoint sh \ -v $MY_ACTINIA_DATA/grassdb_user:/actinia_core/userdata \ --network actinia-docker_actinia-dev mundialis/actinia:2.5.6 ``` + - inside container, start worker listening to specified queue + ``` QUEUE_NAME=job_queue_0 actinia-worker $QUEUE_NAME -c /etc/default/actinia --quit ``` - ## Redis Details ``` @@ -53,13 +59,17 @@ redis-cli -a 'pass' 5) "rq:failed:job_queue_0" 1) "rq:finished:job_queue_0" ``` + ### actinia_worker_count + - only in redis_interface for current queue - created at first HTTP POST request - currently outcommented ### workers + - exists if at least one worker is active, else deleted. + ```r 127.0.0.1:6379> TYPE rq:workers set @@ -102,9 +112,11 @@ hash ``` ### job + - created on job start. Then job is "accepted" - also for synchronous requests, e.g. GET mapsets, tpl processing, ... - deleted after a while - TODO check when? + ```r 127.0.0.1:6379> TYPE rq:job:b6de9170-0fa6-4118-8eb7-9d5f43a37c23 hash @@ -134,9 +146,11 @@ hash ``` ### queues + - rq:queue:job_queue_0 only exists if job in queue - as soon as worker takes job, queue is removed - => if no job is left in queue, it is removed + ```r 127.0.0.1:6379> TYPE rq:queues set @@ -165,6 +179,7 @@ zset ``` ### misc + ```r 127.0.0.1:6379> TYPE rq:clean_registries:job_queue_0 string @@ -173,6 +188,7 @@ string ``` ## Example how to set timeout + ``` # requesting jobs in queue (queue name: job_queue_resource_id-665c5ecb-b7b1-4613-9189-2274f0e01cd7) LRANGE rq:queue:job_queue_resource_id-665c5ecb-b7b1-4613-9189-2274f0e01cd7 0 -1 diff --git a/scripts/README.md b/scripts/README.md index c21e889ed..f914efcbc 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -22,9 +22,9 @@ The option `mapset=MAPSET_NAME` allows the execution of commands in the In case not yet present on the system, the following Python libraries are needed: -* Linux: `pip3 install click requests simplejson` -* Windows users (Installer: [OSGeo4W](https://trac.osgeo.org/osgeo4w/) > Advanced installation > Search window): - * three Python packages: python3-click, python3-requests, python3-simplejson +- Linux: `pip3 install click requests simplejson` +- Windows users (Installer: [OSGeo4W](https://trac.osgeo.org/osgeo4w/) > Advanced installation > Search window): + - three Python packages: python3-click, python3-requests, python3-simplejson ### Installation of ace tools @@ -69,21 +69,21 @@ that on the actinia REST server. Currently available datasets are (organized by projections): -* North Carolina sample dataset (NC State-Plane metric CRS, EPSG: 3358): - * base cartography (`nc_spm_08/PERMANENT`; source: https://grassbook.org/datasets/datasets-3rd-edition/) - * Landsat subscenes (`nc_spm_08/landsat`; source: https://grass.osgeo.org/download/sample-data/) - * MODIS LST time series (`nc_spm_08/modis_lst`; source: NASA) -* Latitude-Longitude location (LatLong WGS84, EPSG:4326): - * empty (`latlong_wgs84/PERMANENT/`) - * 16-days NDVI, MOD13C1, V006, CMG 0.05 deg res. (`latlong_wgs84/modis_ndvi_global/`; source: https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mod13c1_v006) - * LST growing degree days asia 2017 (`latlong_wgs84/asia_gdd_2017/`; source: https://www.mundialis.de/en/temperature-data/) - * LST tropical days asia 2017 (`latlong_wgs84/asia_tropical_2017/`) - * LST temperature daily asia 2017, including min, max and avg (`latlong_wgs84/asia_lst_daily_2017/`) -* Europe (EU LAEA CRS, EPSG:3035): - * EU DEM 25m V1.1 (`eu_laea/PERMANENT/`; source: https://land.copernicus.eu/imagery-in-situ/eu-dem) - * CORINE Landcover 2012, g100_clc12_V18_5 (`eu_laea/corine_2012/`; source: https://land.copernicus.eu/pan-european/corine-land-cover/clc-2012) -* World Mollweide (EPSG 54009): - * GHS_POP_GPW42015_GLOBE_R2015A_54009_250_v1_0 (`world_mollweide/pop_jrc`; source: https://ghsl.jrc.ec.europa.eu/ghs_pop.php) +- North Carolina sample dataset (NC State-Plane metric CRS, EPSG: 3358): + - base cartography (`nc_spm_08/PERMANENT`; source: https://grassbook.org/datasets/datasets-3rd-edition/) + - Landsat subscenes (`nc_spm_08/landsat`; source: https://grass.osgeo.org/download/sample-data/) + - MODIS LST time series (`nc_spm_08/modis_lst`; source: NASA) +- Latitude-Longitude location (LatLong WGS84, EPSG:4326): + - empty (`latlong_wgs84/PERMANENT/`) + - 16-days NDVI, MOD13C1, V006, CMG 0.05 deg res. (`latlong_wgs84/modis_ndvi_global/`; source: https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mod13c1_v006) + - LST growing degree days asia 2017 (`latlong_wgs84/asia_gdd_2017/`; source: https://www.mundialis.de/en/temperature-data/) + - LST tropical days asia 2017 (`latlong_wgs84/asia_tropical_2017/`) + - LST temperature daily asia 2017, including min, max and avg (`latlong_wgs84/asia_lst_daily_2017/`) +- Europe (EU LAEA CRS, EPSG:3035): + - EU DEM 25m V1.1 (`eu_laea/PERMANENT/`; source: https://land.copernicus.eu/imagery-in-situ/eu-dem) + - CORINE Landcover 2012, g100_clc12_V18_5 (`eu_laea/corine_2012/`; source: https://land.copernicus.eu/pan-european/corine-land-cover/clc-2012) +- World Mollweide (EPSG 54009): + - GHS_POP_GPW42015_GLOBE_R2015A_54009_250_v1_0 (`world_mollweide/pop_jrc`; source: https://ghsl.jrc.ec.europa.eu/ghs_pop.php) Note that only selected datasets are available to the demo user (access is managed on a per-user base). @@ -237,7 +237,9 @@ To generate the actinia process chain JSON request simply add the -d (dry-run) f # example: compute slope from an elevation map ace location="nc_spm_08" grass_command="r.slope.aspect elevation=elevation slope=myslope" -d ``` + which will deliver the output: + ```json { "version": "1", @@ -282,11 +284,10 @@ which will deliver the output: At time (more to come) the following export formats are currently supported: -* raster: `COG` (requires GDAL >= 3.1 on server), `GTiff` -* vector: `GPKG`, `GML`, `GeoJSON`, `ESRI_Shapefile`, `SQLite`, `CSV` -* database: `PostgreSQL` -* table: `CSV`, `TXT` - +- raster: `COG` (requires GDAL >= 3.1 on server), `GTiff` +- vector: `GPKG`, `GML`, `GeoJSON`, `ESRI_Shapefile`, `SQLite`, `CSV` +- database: `PostgreSQL` +- table: `CSV`, `TXT` ## Displaying a map - map rendering @@ -298,7 +299,7 @@ ace location="latlong_wgs84" grass_command="r.info globcover@globcover" ace location="latlong_wgs84" render_raster="globcover@globcover" ``` -![ESA Globcover map shown by actinia](esa_globcover_rendered_by_ace.png "ESA Globcover map shown by actinia") +![ESA Globcover map shown by actinia](esa_globcover_rendered_by_ace.png "ESA Globcover map shown by actinia") ## Examples for ephemeral processing @@ -419,6 +420,7 @@ r.info elev r.slope.aspect elevation=elev slope=slope_elev+GTiff r.info slope_elev ``` + Save the script in a text file, e.g. `/tmp/ace_dtm_statistics.sh`. Just for inspection, to generate the actinia process chain JSON request @@ -427,7 +429,9 @@ add the -d (dry-run) flag: ```bash ace -d location="nc_spm_08" script="/tmp/ace_dtm_statistics.sh" ``` + The output should look like this: + ```json { "version": "1", @@ -496,8 +500,6 @@ ace location="nc_spm_08" script="/tmp/ace_dtm_statistics.sh" The resulting data are provided as REST resources for download. - - #### Example 2: Orthophoto image segmentation with export Store the following script as text file `/tmp/ace_segmentation.sh`: @@ -529,7 +531,6 @@ ace location="nc_spm_08" script="/tmp/ace_segmentation.sh" The results are provided as REST resources. - ## Examples for persistent processing GRASS GIS commands can be executed in a user specific persistent database. @@ -589,7 +590,6 @@ acp location="nc_spm_08" grass_command="r.info elev@test_mapset" acp location="nc_spm_08" grass_command="r.info slope_elev@test_mapset" ``` - ## Super admin tasks (note: the "demouser" is not enabled for this) diff --git a/src/actinia_core/README.md b/src/actinia_core/README.md index 1b547895a..47bd4f7eb 100644 --- a/src/actinia_core/README.md +++ b/src/actinia_core/README.md @@ -2,18 +2,20 @@ The module structure is currently under rework. See [according github issue](htt This README should support restructuring and identify dependencies outside of this repository. This mainly concerns the plugins: -* [actinia-module-plugin](https://github.com/actinia-org/actinia-module-plugin) (amp) -* [actinia_statistic_plugin](https://github.com/actinia-org/actinia_statistic_plugin) (statistic) -* [actinia_satallite_plugin](https://github.com/actinia-org/actinia_satellite_plugin) (satellite) -* [actinia-metadata-plugin](https://github.com/actinia-org/actinia-metadata-plugin) uses no sources of actinia_core -* [actinia-stac-plugin](https://github.com/actinia-org/actinia-stac-plugin) (stac) -* [actinia-tiling-plugin](https://github.com/actinia-org/actinia-tiling-plugin) (tiling) -* [actinia-api](https://github.com/actinia-org/actinia-api) (api) + +- [actinia-module-plugin](https://github.com/actinia-org/actinia-module-plugin) (amp) +- [actinia_statistic_plugin](https://github.com/actinia-org/actinia_statistic_plugin) (statistic) +- [actinia_satallite_plugin](https://github.com/actinia-org/actinia_satellite_plugin) (satellite) +- [actinia-metadata-plugin](https://github.com/actinia-org/actinia-metadata-plugin) uses no sources of actinia_core +- [actinia-stac-plugin](https://github.com/actinia-org/actinia-stac-plugin) (stac) +- [actinia-tiling-plugin](https://github.com/actinia-org/actinia-tiling-plugin) (tiling) +- [actinia-api](https://github.com/actinia-org/actinia-api) (api) The modules inside common folder are not only commonly used by actinia but also by actinia plugins. Some other modules outside of core folder are concerned as well. ----------------- +______________________________________________________________________ + ### Overview of modules used outside of actinia | module | import| @@ -55,13 +57,12 @@ Some other modules outside of core folder are concerned as well. | actinia_core.rest.resource_management | ResourceManagerBase | | actinia_core.testsuite | ActiniaTestCaseBase, URL_PREFIX | - -URL_PREFIX moved to actinia_api ------------- +## URL_PREFIX moved to actinia_api ### Complete list of where modules are used and what exactly is imported __amp, satellite, statistic__ + ``` actinia_core.endpoints import create_endpoints actinia_core.rest.base.resource_base import ResourceBase @@ -72,11 +73,13 @@ actinia_core.core.common.response_models import ProcessingResponseModel ``` __amp, statistic__ + ``` actinia_core.processing.actinia_processing.ephemeral_processing import EphemeralProcessing ``` __actinia-module-plugin__ + ``` actinia_core.processing.common.ephemeral_processing_with_export import start_job actinia_core.rest.ephemeral_processing_with_export import SCHEMA_DOC @@ -95,6 +98,7 @@ actinia_core.core.redis_user import RedisUserInterface ``` __satellite, statistic__ + ``` actinia_core.testsuite import ActiniaTestCaseBase, URL_PREFIX actinia_core.core.common.api_logger import log_api_call @@ -105,6 +109,7 @@ actinia_core.core.common.response_models import ProcessingErrorResponseModel ``` __satellite__ + ``` actinia_core.processing.actinia_processing.ephemeral.ephemeral_processing_with_export import EphemeralProcessingWithExport actinia_core.rest.persistent_processing import PersistentProcessing @@ -121,6 +126,7 @@ actinia_core.core.common.sentinel_processing_library import Sentinel2Processing ``` __stac__ + ``` actinia_core.rest.base.resource_base import ResourceBase actinia_core.core.common.redis_base import RedisBaseInterface @@ -128,11 +134,13 @@ actinia_core.core.common.config import Configuration ``` __api__ + ``` actinia_core.models.response_models import ProcessingResponseModel ``` __tiling__ + ``` actinia_core.core.common.config import global_config actinia_core.core.common.process_chain import ProcessChainConverter @@ -149,6 +157,7 @@ actinia_core.testsuite import ActiniaTestCaseBase, URL_PREFIX ``` __parallel__ + ``` actinia_core.core.common.app import auth actinia_core.core.common.config import global_config diff --git a/tests/README.md b/tests/README.md index 9396141d0..042aecf4d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,7 +1,5 @@ # Tests - - ## Running tests locally __If not stated otherwise, you need to be in folder `actinia_core`__ @@ -10,15 +8,17 @@ When writing new tests it is useful to run selected tests locally and isolated. 1. In the `actinia_core/docker/actinia-core-tests/Dockerfile` comment out `RUN make test` (last line) -2. Then run `docker build`: +1. Then run `docker build`: + ``` docker build -f docker/actinia-core-tests/Dockerfile -t actinia-test . ``` + 3. To run only a few tests you can mark the tests for development with -`@pytest.mark.dev` and add `import pytest` to the `.py` file/s with the tests you want to run. -(For best practice examples on the use of pytest-decorators, see `unittests/test_version.py`) + `@pytest.mark.dev` and add `import pytest` to the `.py` file/s with the tests you want to run. + (For best practice examples on the use of pytest-decorators, see `unittests/test_version.py`) -4. Start the docker container and mount your `tests` folder: +1. Start the docker container and mount your `tests` folder: ``` docker run -it -v /path/to/tests:/src/actinia_core/tests actinia-test:latest -i @@ -47,30 +47,32 @@ make devtest # After making changes to the test, you need to close and restart the docker container (docker run ...) before testing again. ``` - ## Problems + Redis authorization is set by `from actinia_core.core.common import redis_interface`: - * so this does set the password: - ``` - import os - from actinia_core.core.common import redis_interface # PROBLEM!!!! - from actinia_core.core.common.config import global_config - - custom_actinia_cfg = str(os.environ["ACTINIA_CUSTOM_TEST_CFG"]) - global_config.read(custom_actinia_cfg) - global_config.REDIS_SERVER_PW - ``` - * and this works fine: - ``` - import os - from actinia_core.core.common.config import global_config - - custom_actinia_cfg = str(os.environ["ACTINIA_CUSTOM_TEST_CFG"]) - global_config.read(custom_actinia_cfg) - global_config.REDIS_SERVER_PW - ``` + +- so this does set the password: + ``` + import os + from actinia_core.core.common import redis_interface # PROBLEM!!!! + from actinia_core.core.common.config import global_config + + custom_actinia_cfg = str(os.environ["ACTINIA_CUSTOM_TEST_CFG"]) + global_config.read(custom_actinia_cfg) + global_config.REDIS_SERVER_PW + ``` +- and this works fine: + ``` + import os + from actinia_core.core.common.config import global_config + + custom_actinia_cfg = str(os.environ["ACTINIA_CUSTOM_TEST_CFG"]) + global_config.read(custom_actinia_cfg) + global_config.REDIS_SERVER_PW + ``` ## Not working processes -* test_login.py: test_create_get_delete_user -* test_async_process_validation.py: test_async_processing_new (Webhook) -* test_asynx_process_postgis_import_export.py: all tests cause of missing PostgreSQL db + +- test_login.py: test_create_get_delete_user +- test_async_process_validation.py: test_async_processing_new (Webhook) +- test_asynx_process_postgis_import_export.py: all tests cause of missing PostgreSQL db