Merge branch 'development'

.ansible-lint

@@ -1,3 +1,6 @@
+exclude_paths:
+  - ./.github
+  - ./docker
 skip_list:
   - no-changed-when  # Commands should not change things if nothing needs doing
   - no-handler  # Tasks that run when changed should likely be handlers

README.md

@@ -64,7 +64,7 @@ sudo adduser vagrant vboxsf
 
 Next, restart the server:
 ```bash
-sudo restart
+sudo reboot
 ```
 
 After rebooting ssh into epos-msl again and the content of the share should be visible within /media/sf_epos!
@@ -84,6 +84,21 @@ sudo ln -s /media/sf_epos /var/www/msl_api
 Check to see if the login page is accessible by navigating to https://epos-msl.ckan.test/webservice/login. A reboot might be
 needed.
 
+#### CKAN extensions
+
+The same method can be used to work with local folders containing the msl CKAN extensions by creating shared folders 
+within the Virtualbox management program as described above. For example after creating a shared folder containing 
+the 'msl_ckan_core' repo checkout can be used within the epos-msl server by first removing the currently present version 
+on the server with:
+```bash
+sudo rm -r /usr/lib/ckan/msl_ckan_core
+```
+
+Followed by creating a symlink to the shared mount:
+```bash
+sudo ln -s /media/sf_ckanext-msl_ckan /usr/lib/ckan/msl_ckan_core
+```
+
 ### Seeding test admin panel account(s)
 
 The msl_api project contains a specific seeder for adding test admin accounts. Contents can be adjusted to add or
@@ -165,6 +180,16 @@ sudo -u www-data /usr/bin/php8.0 artisan db:seed
 sudo -u www-data /usr/bin/php8.0 artisan storage:link
 ```
 
+## Connecting to the mysql database from local machine
+
+To connect from the host machine to the mysql database used by msl_api use the following command:
+
+```
+vagrant ssh epos-msl -- -L 3306:127.0.0.1:3306 -N epos-msl
+```
+
+While running, you can connect to the database using the default credentials.
+
 ## Configuration
 
 The main configuration settings are:
@@ -179,6 +204,10 @@ The main configuration settings are:
 |msl_api_asset_url         | asset URL for the MSL API web service, e.g. https://epos-catalog.mydomain.nl/webservice |
 |ckan_api_token            | the MSL API uses this value to authenticate to the CKAN API. this should currently be the API key (not API token!) of the ckanadmin account. The current way to use this field is: deploy the catalog using a dummy value for this parameter, log in on CKAN using the ckanadmin account, generate an API key, replace the dummy value in the host\_vars file with the real API key, and run the playbook a second time.
 |msl_api_app_key           | the MSL API application key. The current way to configure this is to deploy the application, generate the app key by running `sudo -u www-data /usr/bin/php8.0 artisan key:generate && sudo -u www-data /usr/bin/php8.0 artisan config:cache` in /var/www/msl\_api. Finally copy the generated key in /var/www/msl\_api/.env to the host\_vars file.
+| ckan_install_spatial_plugin | whether to install the ckanext-spatial plugin (default: false) |
+| ckan_spatial_plugin_repo    | Github repository to use for the ckanext-spatial plugin |
+| ckan_spatial_plugin_version | Branch or tag to use for the ckanext-spatial plugin |
+| ckan_plugins_editable_mode  | Whether to install CKAN plugins in editable mode. This is convenient for development and testing purposes. Enabled on development environment; default value is `false`. |
 
 ## CKAN catalog
 

docker/.env

@@ -0,0 +1,48 @@
+# Variables in this file will be substituted into docker-compose.yml
+# Save a copy of this file as .env and insert your own values.
+# Verify correct substitution with "docker-compose config"
+# If variables are newly added or enabled, please delete and rebuild the images to pull in changes:
+# docker-compose down
+# docker rmi -f docker_ckan docker_db
+# docker rmi $(docker images -f dangling=true -q)
+# docker-compose build
+# docker-compose up -d
+# docker-compose restart ckan # give the db service time to initialize the db cluster on first run
+
+# Image: ckan
+CKAN_SITE_ID=default
+
+EPOS_MSL_HOST=epos-msl.ckan
+#
+# On AWS, your CKAN_SITE_URL is the output of:
+# curl -s http://169.254.169.254/latest/meta-data/public-hostname
+# CKAN_SITE_URL=http://ec2-xxx-xxx-xxx-xxx.ap-southeast-2.compute.amazonaws.com
+# On OSX with Docker for Mac, your CKAN_SITE_URL is
+# CKAN_SITE_URL=http://docker.for.mac.localhost:5000
+# When running locally, CKAN_SITE_URL must contain the port
+CKAN_SITE_URL=http://localhost:5000
+#
+# CKAN_PORT must be available on the host: sudo netstat -na
+# To apply change: docker-compose down && docker rmi docker_ckan && docker-compose build ckan
+CKAN_PORT=5000
+#
+# Email settings
+CKAN_SMTP_SERVER=smtp.corporateict.domain:25
+CKAN_SMTP_STARTTLS=True
+CKAN_SMTP_USER=user
+CKAN_SMTP_PASSWORD=pass
+CKAN_SMTP_MAIL_FROM=ckan@localhost
+#
+# Image: db
+POSTGRES_PASSWORD=ckan
+#
+# POSTGRES_PORT must be available on the host: sudo netstat -na | grep 5432
+# To apply change: docker-compose down && docker rmi docker_db docker_ckan && docker-compose build
+POSTGRES_PORT=5432
+#
+# The datastore database will be created in the db container as docs
+# Readwrite user/pass will be ckan:POSTGRES_PASSWORD
+# Readonly user/pass will be datastore_ro:DATASTORE_READONLY_PASSWORD
+DATASTORE_READONLY_PASSWORD=datastore
+
+MYSQL_ROOT_PASSWORD=testtest

docker/README.md

@@ -0,0 +1,32 @@
+# EPOS-MSL Docker development setup
+
+This Docker setup currently has an experimental status, and is in development.
+
+If you use Windows, ensure that core.autocrlf is set to false in your git client before you clone the EPOS-MSL
+repository: _git config --global core.autocrlf false_ Otherwise the Docker images may not work due to line
+ending changes.
+
+## Building the images
+
+The images of the EPOS-MSL are not yet available in a registry, so you'll have to build them locally first.
+
+```
+./build-local-images.sh
+```
+
+## Using the Docker setup
+
+First add an entry to your `/etc/hosts` file (or equivalent) so that queries for the development setup
+interface resolve to your loopback interface. For example:
+
+```
+127.0.0.1 epos-msl.ckan
+```
+
+Start the Docker Compose setup:
+```
+docker compose up
+```
+
+Wait until CKAN has started. Then navigate to [https://epos-msl.ckan](https://epos-msl.ckan) in your browser. The
+development VM runs with self-signed certificates, so you'll need to accept the security warning.

docker/build-local-images.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+set -e
+
+cd images
+
+for image in ckan nginx solr msl-api
+do cd "$image"
+   echo "Building image $image ..."
+   ./build.sh
+   cd ..
+done
+
+echo "Building images completed."

docker/docker-compose.yml

@@ -0,0 +1,102 @@
+# This is a locally modified version of the upstream CKAN docker compose file.
+# It contains image names for pushing locally built images to a central registry at ECR
+
+# docker-compose build && docker-compose up -d
+# If "docker-compose logs ckan" shows DB not ready, run "docker-compose restart ckan" a few times.
+version: "3"
+
+volumes:
+  ckan_config:
+  ckan_home:
+  ckan_storage:
+  ckan_coveragedata:
+  pg_data:
+  solr_data:
+  nginx_config:
+  nginx_certificates:
+  mslapi_signal:
+  mslapi_storage:
+
+services:
+  ckan:
+     container_name: ckan
+     image: epos-msl-ckan:latest
+     links:
+       - db
+       - solr
+       - redis
+     depends_on:
+       - db
+       - solr
+       - redis
+     ports:
+       - "0.0.0.0:${CKAN_PORT}:5000"
+     environment:
+       - CKAN_SQLALCHEMY_URL=postgresql://ckan:${POSTGRES_PASSWORD}@db/ckan_default
+       - CKAN_SOLR_URL=http://solr:8983/solr/ckan
+       - CKAN_REDIS_URL=redis://redis:6379/1
+       - CKAN_SITE_URL=${CKAN_SITE_URL}
+       - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+     volumes:
+       - ckan_config:/etc/ckan
+       - ckan_home:/usr/lib/ckan
+       - ckan_storage:/var/lib/ckan
+       - ckan_coveragedata:/coverage
+
+  nginx:
+    container_name: nginx
+    image: epos-msl-nginx:latest
+    environment:
+      - EPOS_MSL_HOST=${EPOS_MSL_HOST}
+    ports:
+      - "18443:443"
+    volumes:
+      - nginx_config:/etc/nginx/conf.d
+      - nginx_certificates:/etc/certificates
+
+  db:
+    container_name: db
+    image: postgres:12.20
+    environment:
+      - POSTGRES_DB=ckan_default
+      - POSTGRES_USER=ckan
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+      - PGDATA=/var/lib/postgresql/data/db
+    volumes:
+      - pg_data:/var/lib/postgresql/data
+
+  mslapi_db:
+    container_name: mslapi_db
+    image: mysql:8.0.39-debian
+    environment:
+      -  MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
+
+  mslapi_webserver:
+    container_name: mslapi_web
+    image: epos-msl-api:latest
+    environment:
+      -  MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
+      -  MSLAPI_ROLE=WEBSERVER
+    volumes:
+      -  mslapi_signal:/signal
+      -  mslapi_storage:/storage
+
+  mslapi_worker:
+    container_name: mslapi_worker
+    image: epos-msl-api:latest
+    environment:
+      -  MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
+      -  MSLAPI_ROLE=QUEUE_WORKER
+    volumes:
+      -  mslapi_signal:/signal
+      -  mslapi_storage:/storage
+
+  solr:
+    container_name: solr
+    image: epos-msl-solr:latest
+    volumes:
+      - solr_data:/opt/solr/server/solr/ckan/data
+
+  redis:
+    container_name: redis
+    image: redis:6.2

docker/images/ckan/Dockerfile

@@ -0,0 +1,106 @@
+# This is a locally modified version of the upstream CKAN Dockerfile
+
+# See CKAN docs on installation from Docker Compose on usage
+FROM ubuntu:focal-20210119
+MAINTAINER Yoda team
+
+# Set timezone
+ENV TZ=UTC
+RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
+
+# Setting the locale
+ENV LC_ALL=en_US.UTF-8
+RUN adduser ckan
+RUN apt-get update
+RUN apt-get install --no-install-recommends -y locales
+RUN sed -i "/$LC_ALL/s/^# //g" /etc/locale.gen
+RUN dpkg-reconfigure --frontend=noninteractive locales
+RUN update-locale LANG=${LC_ALL}
+
+# Install required system packages
+RUN apt-get -q -y update \
+    && DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade \
+    && apt-get -q -y install \
+        python3.8 \
+        python3-dev \
+        python3-pip \
+        python3-venv \
+        python3-wheel \
+        libpq-dev \
+        python3-pastescript \
+        python3-virtualenv \
+        libxml2-dev \
+        libxslt-dev \
+        libgeos-dev \
+        libssl-dev \
+        libffi-dev \
+        postgresql-client \
+        build-essential \
+        git-core \
+        vim \
+        wget \
+        curl \
+        nmap \
+        sqlite3 \
+        pwgen \
+        uuid-runtime \
+    && apt-get -q clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Define environment variables
+ENV CKAN_HOME /usr/lib/ckan
+ENV CKAN_VENV $CKAN_HOME/default
+ENV CKAN_CONFIG /etc/ckan
+ENV CKAN_STORAGE_PATH=/ckanstorage
+ENV CKAN_VERSION=2.9.11
+ENV CKAN_SCHEMING_VERSION=release-2.1.0
+ENV CKAN_MSL_CORE_VERSION=1.4.0
+ENV CKAN_MSL_UTIL_VERSION=1.0.0
+
+# Create storage path
+RUN mkdir -p $CKAN_STORAGE_PATH/webassets $CKAN_STORAGE_PATH/storage
+
+# Build-time variables specified by docker-compose.yml / .env
+ARG CKAN_SITE_URL
+
+# Setup virtual environment for CKAN
+RUN mkdir -p $CKAN_VENV $CKAN_CONFIG/default && \
+    python3 -m venv $CKAN_VENV && \
+    ln -s $CKAN_VENV/bin/pip3 /usr/local/bin/ckan-pip3 &&\
+    ln -s $CKAN_VENV/bin/ckan /usr/local/bin/ckan
+ADD ckan.ini /etc/ckan/default/ckan.ini
+ADD who.ini /etc/ckan/default/who.ini
+ADD wsgi.py /etc/ckan/default/wsgi.py
+ADD ckan-uwsgi.ini /etc/ckan/default/ckan-uwsgi.ini
+
+# Virtual environment binaries/scripts to be used first
+ENV PATH=${CKAN_VENV}/bin:${PATH}
+ENV EPOS_MSL_FQDN=epos-msl.local
+ENV CKAN_ADMIN_PASSWORD="testtest"
+
+# Install CKAN and plugins
+RUN ckan-pip3 install -U pip && \
+    ckan-pip3 install setuptools==44.1.0 && \
+    ckan-pip3 install --upgrade pip && \
+    ckan-pip3 install wheel && \
+    ckan-pip3 install -e "git+https://github.com/ckan/ckan@ckan-${CKAN_VERSION}#egg=ckan[requirements]" && \
+    ckan-pip3 install uwsgi && \
+    ckan-pip3 install -e "git+https://github.com/ckan/ckanext-scheming@${CKAN_SCHEMING_VERSION}#egg=ckanext-scheming" && \
+    ckan-pip3 install -e "git+https://github.com/UtrechtUniversity/msl_ckan_core@${CKAN_MSL_CORE_VERSION}#egg=ckanext-msl_ckan" && \
+    ckan-pip3 install -e "git+https://github.com/UtrechtUniversity/msl_ckan_util@${CKAN_MSL_UTIL_VERSION}#egg=ckanext-msl_ckan_util" && \
+    ln -s $CKAN_VENV/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini && \
+    cp -v $CKAN_VENV/src/ckan/contrib/docker/ckan-entrypoint.sh /ckan-entrypoint.sh && \
+    chmod +x /ckan-entrypoint.sh && \
+    chown -R ckan:ckan $CKAN_HOME $CKAN_VENV $CKAN_CONFIG $CKAN_STORAGE_PATH $COVERAGE_DIR && \
+    rm /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml && \
+    ln -sf /usr/lib/ckan/default/src/ckanext-msl-ckan/ckanext/msl_ckan/config/solr/schema.xml /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml && \
+    perl -n -i.bak -e 'print unless /defaultSearchField/ or /solrQueryParser/' /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml
+
+ADD ./ckan-entrypoint.sh /ckan-entrypoint.sh
+ENTRYPOINT ["/ckan-entrypoint.sh"]
+RUN chmod +x /ckan-entrypoint.sh
+
+USER ckan
+EXPOSE 8080
+
+CMD ["/ckan-entrypoint.sh"]

docker/images/ckan/build.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+docker build -t epos-msl-ckan:latest .