Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renamed GRASS GIS locations to projects #565

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
19c6f4a
add decorator for deprecated locations
anikaweinmann Oct 16, 2024
0b1e445
linting
anikaweinmann Oct 17, 2024
a327f27
add project endpoints
anikaweinmann Oct 17, 2024
3ea49a9
endpoints for projects
anikaweinmann Oct 17, 2024
32e9034
rename location
anikaweinmann Oct 17, 2024
fd11517
rename location
anikaweinmann Oct 17, 2024
628ac83
unify maintainer
anikaweinmann Oct 17, 2024
e3c8a54
fix GRASS Initialisation
anikaweinmann Oct 17, 2024
49b4418
black
anikaweinmann Oct 17, 2024
2ef9463
fixes for tests and G83
anikaweinmann Oct 18, 2024
eff93b7
fixes
anikaweinmann Nov 14, 2024
dc672e1
add rest of changes
anikaweinmann Nov 15, 2024
bd3a7a0
CT review
anikaweinmann Nov 15, 2024
7498542
CT review
anikaweinmann Nov 15, 2024
8c1d5cb
Update tests/test_job_resumption.py
anikaweinmann Nov 15, 2024
4cc99f5
fix endpoints
anikaweinmann Nov 15, 2024
211c500
Merge branch 'renamed_locations' of github.com:actinia-org/actinia-co…
anikaweinmann Nov 15, 2024
ed2565f
fix job resumption tests
anikaweinmann Nov 15, 2024
a06ddf1
linting
anikaweinmann Nov 15, 2024
3ec62c3
Update src/actinia_core/core/common/api_logger.py
anikaweinmann Nov 20, 2024
f3ce464
Tests for G84
anikaweinmann Nov 21, 2024
1515f51
Test pipelines for G8.3 and G8.4
anikaweinmann Nov 21, 2024
ea142b5
Merge branch 'renamed_locations' of github.com:actinia-org/actinia-co…
anikaweinmann Nov 21, 2024
587a56e
Update src/actinia_core/rest/base/resource_base.py
anikaweinmann Nov 21, 2024
5d5e8a8
Merge branch 'main' into renamed_locations
anikaweinmann Nov 21, 2024
a6c24d8
fix test workflow
anikaweinmann Nov 21, 2024
71419f6
Merge branch 'renamed_locations' of github.com:actinia-org/actinia-co…
anikaweinmann Nov 21, 2024
52736cc
fix tests for G83
anikaweinmann Nov 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 48 additions & 4 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,35 @@ on:

jobs:

# Unittests for GRASS 8.3
alpine-unittests-G83:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Replace run only unittest command
run: |
sed -i "1s+.*+FROM mundialis/actinia-core:4.14.1 AS actinia_test+" docker/actinia-core-tests/Dockerfile
sed -i "s+# RUN make test+RUN make unittest+g" docker/actinia-core-tests/Dockerfile
- name: Unittests of actinia
id: docker_build
uses: docker/build-push-action@v6
with:
push: false
tags: actinia-core-tests:alpine
context: .
file: docker/actinia-core-tests/Dockerfile
no-cache: true
# pull: true

# Unittests for GRASS 8.4
alpine-unittests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
# with:
# path: "."
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Replace run only unittest command
Expand All @@ -32,13 +54,35 @@ jobs:
no-cache: true
# pull: true

# Integration tests for GRASS 8.3
alpine-integration-tests-G83:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Replace run integration test command
run: |
sed -i "1s+.*+FROM mundialis/actinia-core:4.14.1 AS actinia_test+" docker/actinia-core-tests/Dockerfile
sed -i "s+# RUN make test+RUN make integrationtest+g" docker/actinia-core-tests/Dockerfile
- name: Integration tests of actinia
id: docker_build
uses: docker/build-push-action@v6
with:
push: false
tags: actinia-core-tests:alpine
context: .
file: docker/actinia-core-tests/Dockerfile
no-cache: true
# pull: true

# Integration tests for GRASS 8.4
alpine-integration-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
# with:
# path: "."
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Replace run integration test command
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,28 +63,28 @@ Actinia is also available on [OSGeoLive](https://live.osgeo.org/en/overview/acti

### Data management

- List all locations that are available in the actinia persistent database:
- List all projects that are available in the actinia persistent database:

```bash
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/locations"
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/projects"
```

- List all mapsets in the location latlong_wgs84:
- List all mapsets in the project latlong_wgs84:

```bash
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/locations/latlong_wgs84/mapsets"
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/projects/latlong_wgs84/mapsets"
```

- List all space-time raster datasets (STRDS) in location latlong_wgs84 and mapset Sentinel_timeseries:
- List all space-time raster datasets (STRDS) in project latlong_wgs84 and mapset Sentinel_timeseries:

```bash
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/locations/latlong_wgs84/mapsets/modis_ndvi_global/strds"
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/projects/latlong_wgs84/mapsets/modis_ndvi_global/strds"
```

- List all raster map layers of the STRDS:

```bash
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/locations/latlong_wgs84/mapsets/modis_ndvi_global/strds/ndvi_16_5600m/raster_layers"
curl -u 'demouser:gu3st!pa55w0rd' -X GET "https://actinia.mundialis.de/api/v3/projects/latlong_wgs84/mapsets/modis_ndvi_global/strds/ndvi_16_5600m/raster_layers"
```

### Landsat and Sentinel-2 NDVI computation
Expand Down
4 changes: 2 additions & 2 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ curl http://127.0.0.1:8088/api/v3/version
- Want to __start developing__? Look for [Local dev-setup with docker](#local-dev-setup) below.
- For __production deployment__, see [Production deployment](#production-deployment) below.

On startup, some GRASS GIS locations are created by default but they are still empty. How to get some geodata to start processing, see in [Testing GRASS GIS inside a container](#grass-gis) below.
On startup, some GRASS GIS projects are created by default but they are still empty. How to get some geodata to start processing, see in [Testing GRASS GIS inside a container](#grass-gis) below.

## Adding a user

Expand Down Expand Up @@ -192,7 +192,7 @@ You now have some data which you can access through actinia. To get information
via API, start actinia with gunicorn and run

```bash
curl -u actinia-gdi:actinia-gdi http://127.0.0.1:8088/api/v3/locations/nc_spm_08/mapsets
curl -u actinia-gdi:actinia-gdi http://127.0.0.1:8088/api/v3/projects/nc_spm_08/mapsets
```

The folder where you downloaded the data into (`/actinia_core/grassdb`) is mounted into your docker container via the compose file, so all data is kept, even if your docker container restarts.
Expand Down
8 changes: 4 additions & 4 deletions docker/actinia-core-alpine/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
FROM mundialis/actinia:alpine-dependencies-2023-12-06 as build-base
FROM osgeo/grass-gis:releasebranch_8_3-alpine as grass
FROM mundialis/actinia:alpine-dependencies-2024-08-14 as build-base
FROM osgeo/grass-gis:releasebranch_8_4-alpine as grass

FROM build-base as requirements

Expand All @@ -16,12 +16,12 @@ ENV GISBASE ""
COPY --from=grass /usr/local/bin/grass /usr/local/bin/grass
COPY --from=grass /usr/local/grass* /usr/local/grass/
RUN ln -s /usr/local/grass "$(grass --config path)"
RUN grass --tmp-location EPSG:4326 --exec g.version -rge && \
RUN grass --tmp-project EPSG:4326 --exec g.version -rge && \
pdal --version && \
python --version
anikaweinmann marked this conversation as resolved.
Show resolved Hide resolved

# Install GRASS GIS addon d.rast.multi (needed for STRDS render endpoint)
RUN grass --tmp-location EPSG:4326 --exec g.extension -s \
RUN grass --tmp-project EPSG:4326 --exec g.extension -s \
extension=d.rast.multi url=https://github.com/mundialis/d_rast_multi


Expand Down
2 changes: 1 addition & 1 deletion docker/actinia-core-alpine/start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ mkdir -p /actinia_core/workspace/temp_db
mkdir -p /actinia_core/workspace/tmp
mkdir -p /actinia_core/resources

# Create default location in mounted (!) directory
# Create default project in mounted (!) directory
[ ! -d "/actinia_core/grassdb/nc_spm_08" ] && grass -e -c 'EPSG:3358' /actinia_core/grassdb/nc_spm_08

actinia-user create -u actinia-gdi -w actinia-gdi -r superadmin -g superadmin -c 100000000000 -n 1000 -t 31536000
Expand Down
4 changes: 2 additions & 2 deletions docker/actinia-core-dev/endpoints.csv
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
ListLocationsResource;GET
LocationManagementResourceUser;GET
ListProjectsResource;GET
ProjectManagementResourceUser;GET
ListMapsetsResource;GET
MapsetManagementResourceUser;GET
RasterLayersResource;GET
Expand Down
3 changes: 2 additions & 1 deletion docker/actinia-core-tests/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM mundialis/actinia-core:4.14.1 AS actinia_test
FROM mundialis/actinia-core:grass8.4 AS actinia_test

LABEL authors="Carmen Tawalika,Anika Weinmann"
LABEL maintainer="[email protected],[email protected]"
Expand All @@ -21,6 +21,7 @@ RUN wget --quiet https://grass.osgeo.org/sampledata/north_carolina/nc_spm_mapset
RUN chown -R 1001:1001 nc_spm_08/modis_lst && chmod -R g+w nc_spm_08/modis_lst

# install GRASS addons required for tests
# TODO use --tmp-project if actinia-core uses GRASS 8.4
RUN grass --tmp-location EPSG:4326 --exec g.extension -s extension=r.colors.out_sld

# install things only for tests
Expand Down
16 changes: 8 additions & 8 deletions docs/docs/actinia_concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ The following user-roles are supported:
### 3. user
- Can run computational tasks in ephemeral and user specific
databases
- Can create, modify and delete locations in a user specific
- Can create, modify and delete projects in a user specific
database
- Can create, modify and delete mapsets in user
specific databases
Expand All @@ -81,7 +81,7 @@ Overview table:
|------|------------|-------|------|-------|------|
| amount raster cells is unlimited | y | y | limited, selected via redis | limited, selected via redis | - |
| database access is unlimited | y | only to persistent databases that were granted by a superadmin | limited, defined in redis | limited, defined in redis | - |
| location/mapset access is unlimited | y | y | can create, modify and delete locations/mapsets in user specific databases, defined in redis | has access to persistent databases that were granted by a superadmin, defined in redis | - |
| project/mapset access is unlimited | y | y | can create, modify and delete projects/mapsets in user specific databases, defined in redis | has access to persistent databases that were granted by a superadmin, defined in redis | - |
|module access is unlimited | y | y | can run computational tasks in ephemeral and user specific databases | has very limited access to API calls | - |
| get, create, delete a single user | y | users with the maximum user-role user of the same user group | n | n | Only normal users (role=user can be created) |

Expand All @@ -97,7 +97,7 @@ number_of_workers = 3

## The Actinia databases

Actinia manages GRASS GIS locations in its *persistent database*. User
Actinia manages GRASS GIS projects in its *persistent database*. User
are not permitted to modify data in the actinia persistent database, but
can access all data read-only for processing and visualization. Data in
the persistent database can only be accessed via HTTP GET API calls.
Expand All @@ -113,7 +113,7 @@ database.
**Summary**

### 1. Persistent database
- Read only database with locations and mapsets that can be
- Read only database with projects and mapsets that can be
used as processing environment and data source
- Data can only be accessed using HTTP GET API calls

Expand All @@ -124,16 +124,16 @@ database.
removed after the processing is finished
- Ephemeral databases use persistent databases as processing
environments to access required data from mapsets in
persistent locations
persistent projects

### 3. User specific databases
- Persistent databases that can be created and modified by a
specific user group
- The base for a location in a user specific database can be
a location from a persistent database, however mapsets
- The base for a project in a user specific database can be
a project from a persistent database, however mapsets
names must be unique.
- A user group can only access a single database with any
number of locations
number of projects

**Footnotes**

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ The source code is available here: <https://github.com/actinia-org/actinia-core>
- Installation
- Actinia database access
- Using curl for HTTP requests
- Access to locations and mapsets in the persistent database
- Access to projects and mapsets in the persistent database
- Access to raster layers in the persistent database
- Access to raster time-series in the persistent database
- Time-series sampling
Expand Down
8 changes: 4 additions & 4 deletions docs/docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1
3. Compile and install GRASS GIS and additional modules:

```bash
# or some other location of your choice, /tmp will not keep data between reboots
# or some other project of your choice, /tmp will not keep data between reboots
cd /tmp

# fetch weekly generated latest snapshot of GRASS GIS stable
Expand Down Expand Up @@ -110,7 +110,7 @@ cd ../..
sudo make install
```

4. Download the test datasets ("locations") and place them into a
4. Download the test datasets ("projects") and place them into a
specific directory that will be used by actinia as a persistent database:

```bash
Expand Down Expand Up @@ -179,13 +179,13 @@ ACTINIA_VERSION="v3"
export ACTINIA_URL="http://localhost:5000/api/${ACTINIA_VERSION}"
export AUTH='-u superadmin:abcdefgh'

curl ${AUTH} -X GET ${ACTINIA_URL}/locations
curl ${AUTH} -X GET ${ACTINIA_URL}/projects
```

The `curl` command call should report back:

```bash
{"locations":["nc_spm_08","ECAD","latlong_wgs84"],"status":"success"}
{"projects":["nc_spm_08","ECAD","latlong_wgs84"],"status":"success"}
```

Success and welcome to actinia!
24 changes: 12 additions & 12 deletions docs/docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ provide problem specific services like Sentinel-2 and Landsat NDVI
computation, spatio-temporal statistical analysis and many more.

To use actinia the user must have an understanding of the GRASS GIS
concept[^3] of location, mapsets, raster maps, space-time datasets and
concept[^3] of project, mapsets, raster maps, space-time datasets and
modules. The URLs that provide access to the GRASS database reflect
these concepts. Hence, the location, the mapset and the required raster map
these concepts. Hence, the project, the mapset and the required raster map
are part of the URL to access the service.

## What is REST?
Expand Down Expand Up @@ -48,35 +48,35 @@ export AUTH='-u demouser:gu3st!pa55w0rd'

**Data management**

List all locations that are available in the actinia persistent database:
List all projects that are available in the actinia persistent database:

```bash
curl ${AUTH} -X GET "${ACTINIA_URL}/locations"
curl ${AUTH} -X GET "${ACTINIA_URL}/projects"
```

List all mapsets in the location latlong_wgs84:
List all mapsets in the project latlong_wgs84:

```bash
curl ${AUTH} -X GET "${ACTINIA_URL}/locations/latlong_wgs84/mapsets"
curl ${AUTH} -X GET "${ACTINIA_URL}/projects/latlong_wgs84/mapsets"
```

List all raster layers in location latlong_wgs84 and mapset Sentinel2A:
List all raster layers in project latlong_wgs84 and mapset Sentinel2A:

```bash
curl ${AUTH} -X GET "${ACTINIA_URL}/locations/latlong_wgs84/mapsets/Sentinel2A/raster_layers"
curl ${AUTH} -X GET "${ACTINIA_URL}/projects/latlong_wgs84/mapsets/Sentinel2A/raster_layers"
```

List all space-time raster datasets (STRDS) in location
List all space-time raster datasets (STRDS) in project
ECAD and mapset PERMANENT:

```bash
curl ${AUTH} -X GET "${ACTINIA_URL}/locations/ECAD/mapsets/PERMANENT/strds"
curl ${AUTH} -X GET "${ACTINIA_URL}/projects/ECAD/mapsets/PERMANENT/strds"
```

List all raster map layers of the STRDS precipitation_1950_2013_yearly_mm:

```bash
curl ${AUTH} -X GET "${ACTINIA_URL}/locations/ECAD/mapsets/PERMANENT/strds/precipitation_1950_2013_yearly_mm/raster_layers"
curl ${AUTH} -X GET "${ACTINIA_URL}/projects/ECAD/mapsets/PERMANENT/strds/precipitation_1950_2013_yearly_mm/raster_layers"
```

**Landsat and Sentinel-2A NDVI computation**
Expand All @@ -102,7 +102,7 @@ file in a cloud storage for download.

[^1]: https://grass.osgeo.org/
[^2]: https://en.wikipedia.org/wiki/Representational_State_Transfer
[^3]: https://grass.osgeo.org/grass-stable/manuals/helptext.html#2.-background:-grass-gis-location-structure
[^3]: https://grass.osgeo.org/grass-stable/manuals/helptext.html#2.-background:-grass-gis-project-structure
[^4]: https://en.wikipedia.org/wiki/Representational_State_Transfer
[^5]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
[^6]: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods
Expand Down
Loading
Loading