Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add integrationtest-sda-doa and build-sda-doa to Makefile #1305

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

add integrationtest-sda-doa and build-sda-doa to Makefile #1305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/integration/scripts/make_certs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ openssl x509 -req -in "$out_dir/mq.csr" -days 1200 -CA "$out_dir/ca.crt" -CAkey

# Create client certificate
openssl req -config "$script_dir/ssl.cnf" -new -nodes -newkey rsa:4096 -keyout "$out_dir/client.key" -out "$out_dir/client.csr" -extensions client_cert -subj "/CN=admin"
openssl pkcs8 -topk8 -inform PEM -outform DER -in "$out_dir/client.key" -out "$out_dir/client.der" -nocrypt
openssl x509 -req -in "$out_dir/client.csr" -days 1200 -CA "$out_dir/ca.crt" -CAkey "$out_dir/ca-key.pem" -set_serial 01 -out "$out_dir/client.crt" -extensions client_cert -extfile "$script_dir/ssl.cnf"

if [ -n "$KEYSTORE_PASSWORD" ]; then
Expand Down Expand Up @@ -68,7 +69,9 @@ chmod 600 /certs/*.key
cp -p "$out_dir/ca.crt" /client_certs/ca.crt
cp -p "$out_dir/client.crt" /client_certs/
cp -p "$out_dir/client.key" /client_certs/
cp -p "$out_dir/client.der" /client_certs/
chmod 600 /client_certs/*.key
chmod 644 /client_certs/*.der

# needed if testing locally
mkdir -p /temp/certs
Expand Down
132 changes: 132 additions & 0 deletions .github/integration/sda-doa-posix-outbox.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
services:
certfixer:
command:
- /bin/sh
- /scripts/make_certs.sh
container_name: certfixer
image: alpine:latest
volumes:
- ./scripts:/scripts
- certs:/certs
- client_certs:/client_certs
- /tmp:/temp

postgres:
build:
context: ../../postgresql
container_name: postgres
image: ghcr.io/neicnordic/sensitive-data-archive:PR${PR_NUMBER}-postgres
depends_on:
certfixer:
condition: service_completed_successfully
ports:
- 5432:5432
environment:
- LEGA_IN_PASSWORD=password
- LEGA_OUT_PASSWORD=password
- POSTGRES_PASSWORD=rootpasswd
- POSTGRES_SERVER_CACERT=/certs/ca.crt
- POSTGRES_SERVER_CERT=/certs/db.crt
- POSTGRES_SERVER_KEY=/certs/db.key

healthcheck:
test: [ "CMD", "pg_isready", "-h", "localhost", "-U", "lega_out" ]
interval: 5s
timeout: 20s
retries: 3
volumes:
- certs:/certs

mockauth:
container_name: mockauth
image: python:3.11-slim
ports:
- 8000:8000
volumes:
- ../../sda-doa/test/mock_auth.py:/mock_auth.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should us the same mock-auth as the pother services, update to match if needed.

- client_certs:/client_certs
command: >
sh -c "pip install --upgrade pip && pip install aiohttp Authlib && python -u /mock_auth.py 0.0.0.0 8000"



rabbitmq:
build:
context: ../../rabbitmq
image: ghcr.io/neicnordic/sensitive-data-archive:PR${PR_NUMBER}-rabbitmq
container_name: rabbitmq
depends_on:
certfixer:
condition: service_completed_successfully
environment:
- RABBITMQ_SERVER_CACERT=/etc/rabbitmq/ssl/ca.crt
- RABBITMQ_SERVER_CERT=/etc/rabbitmq/ssl/mq.crt
- RABBITMQ_SERVER_KEY=/etc/rabbitmq/ssl/mq.key
- RABBITMQ_SERVER_VERIFY=verify_none
healthcheck:
test:
[
"CMD",
"bash",
"-c",
"rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms",
]
interval: 5s
timeout: 20s
retries: 3
restart: always
volumes:
- certs:/etc/rabbitmq/ssl/
- rabbitmq_data:/var/lib/rabbitmq
ports:
- 5671:5671
- 25671:15671

doa:
container_name: doa
build:
context: ../../sda-doa
dockerfile: ../../sda-doa/Dockerfile
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dockerfile: ../../sda-doa/Dockerfile

Implicit and should not be needed

image: ghcr.io/neicnordic/sensitive-data-archive:PR${PR_NUMBER}-sda-doa
depends_on:
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
certfixer:
condition: service_completed_successfully
links:
- rabbitmq
- postgres
Comment on lines +98 to +100
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
links:
- rabbitmq
- postgres

Handled by proper configuration.

ports:
- "8080:8080"
environment:
- SSL_MODE=verify-ca
- POSTGRES_USER=postgres
- POSTGRES_DB=sda
- POSTGRES_PASSWORD=rootpasswd
- DB_INSTANCE=postgres
- SSL_ENABLED=false
- BROKER_HOST=rabbitmq
- BROKER_VALIDATE=false
- OPENID_CONFIGURATION_URL=http://mockauth:8000/openid-configuration
- USERINFO_ENDPOINT_URL=http://mockauth:8000/userinfo
Comment on lines +112 to +113
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- OPENID_CONFIGURATION_URL=http://mockauth:8000/openid-configuration
- USERINFO_ENDPOINT_URL=http://mockauth:8000/userinfo
- OPENID_CONFIGURATION_URL=http://localhost:8000/openid-configuration
- USERINFO_ENDPOINT_URL=http://localhost:8000/userinfo

Or this will not be usable in an interactive case.

- CRYPT4GH_PRIVATE_KEY_PATH=test/crypt4gh.sec.pem
- CRYPT4GH_PRIVATE_KEY_PASSWORD_PATH=test/crypt4gh.pass
- OUTBOX_TYPE=POSIX
- OUTBOX_LOCATION=/outbox/%s/files/
- SSL_ENABLED=false
- ROOT_CERT_PATH=/certs/ca.crt
- CERT_PATH=/certs/client.crt
- CERT_KEY=/certs/client.der
- BROKER_USERNAME=guest

volumes:
- client_certs:/certs
- ../../sda-doa/test/body.enc:/test/body.enc
- ../../sda-doa/outbox:/outbox
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't write data to the repository, always use the system temp for this.


volumes:
certs:
client_certs:
rabbitmq_data:
163 changes: 163 additions & 0 deletions .github/integration/sda-doa-s3-outbox.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
services:
certfixer:
command:
- /bin/sh
- /scripts/make_certs.sh
container_name: certfixer
image: alpine:latest
volumes:
- ./scripts:/scripts
- certs:/certs
- client_certs:/client_certs
- /tmp:/temp

postgres:
build:
context: ../../postgresql
container_name: postgres
image: ghcr.io/neicnordic/sensitive-data-archive:PR${PR_NUMBER}-postgres
depends_on:
certfixer:
condition: service_completed_successfully
ports:
- 5432:5432
environment:
- LEGA_IN_PASSWORD=password
- LEGA_OUT_PASSWORD=password
- POSTGRES_PASSWORD=rootpasswd
- POSTGRES_SERVER_CACERT=/certs/ca.crt
- POSTGRES_SERVER_CERT=/certs/db.crt
- POSTGRES_SERVER_KEY=/certs/db.key

healthcheck:
test: [ "CMD", "pg_isready", "-h", "localhost", "-U", "lega_out" ]
interval: 5s
timeout: 20s
retries: 3
volumes:
- certs:/certs

mockauth:
container_name: mockauth
image: python:3.11-slim
ports:
- 8000:8000
volumes:
- ../../sda-doa/test/mock_auth.py:/mock_auth.py
- client_certs:/client_certs
command: >
sh -c "pip install --upgrade pip && pip install aiohttp Authlib && python -u /mock_auth.py 0.0.0.0 8000"

rabbitmq:
build:
context: ../../rabbitmq
image: ghcr.io/neicnordic/sensitive-data-archive:PR${PR_NUMBER}-rabbitmq
container_name: rabbitmq
depends_on:
certfixer:
condition: service_completed_successfully
environment:
- RABBITMQ_SERVER_CACERT=/etc/rabbitmq/ssl/ca.crt
- RABBITMQ_SERVER_CERT=/etc/rabbitmq/ssl/mq.crt
- RABBITMQ_SERVER_KEY=/etc/rabbitmq/ssl/mq.key
- RABBITMQ_SERVER_VERIFY=verify_none
healthcheck:
test:
[
"CMD",
"bash",
"-c",
"rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms",
]
interval: 5s
timeout: 20s
retries: 3
restart: always
volumes:
- certs:/etc/rabbitmq/ssl/
- rabbitmq_data:/var/lib/rabbitmq
ports:
- 5671:5671
- 25671:15671

outbox:
container_name: outbox
image: minio/minio
ports:
- 9000:9000
environment:
- MINIO_ACCESS_KEY=minio
- MINIO_SECRET_KEY=miniostorage
command: server /data
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 10s
timeout: 5s
retries: 5

init-bucket:
container_name: init-bucket
image: minio/mc
depends_on:
outbox:
condition: service_healthy
entrypoint: >
/bin/sh -c "
/usr/bin/mc config host add s3 http://outbox:9000 minio miniostorage;
/usr/bin/mc mb s3/lega;
exit 0;
"

doa:
container_name: doa
build:
context: ../../sda-doa
dockerfile: ../../sda-doa/Dockerfile
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dockerfile: ../../sda-doa/Dockerfile

image: ghcr.io/neicnordic/sensitive-data-archive:PR${PR_NUMBER}-sda-doa
depends_on:
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
certfixer:
condition: service_completed_successfully
links:
- rabbitmq
- postgres
Comment on lines +124 to +126
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
links:
- rabbitmq
- postgres

ports:
- "8080:8080"
environment:
- SSL_MODE=verify-ca
- POSTGRES_USER=postgres
- POSTGRES_DB=sda
- POSTGRES_PASSWORD=rootpasswd
- DB_INSTANCE=postgres
- SSL_ENABLED=false
- BROKER_HOST=rabbitmq
- BROKER_VALIDATE=false
- OPENID_CONFIGURATION_URL=http://mockauth:8000/openid-configuration
- USERINFO_ENDPOINT_URL=http://mockauth:8000/userinfo
Comment on lines +138 to +139
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- OPENID_CONFIGURATION_URL=http://mockauth:8000/openid-configuration
- USERINFO_ENDPOINT_URL=http://mockauth:8000/userinfo
- OPENID_CONFIGURATION_URL=http://localhost:8000/openid-configuration
- USERINFO_ENDPOINT_URL=http://localhost:8000/userinfo

- CRYPT4GH_PRIVATE_KEY_PATH=test/crypt4gh.sec.pem
- CRYPT4GH_PRIVATE_KEY_PASSWORD_PATH=test/crypt4gh.pass
- OUTBOX_TYPE=S3
- SSL_ENABLED=false
- ROOT_CERT_PATH=/certs/ca.crt
- CERT_PATH=/certs/client.crt
- CERT_KEY=/certs/client.der
- BROKER_USERNAME=guest
- S3_OUT_ENDPOINT=outbox
- S3_OUT_PORT=9000
- S3_OUT_SECURE=false

volumes:
- client_certs:/certs
- ../../sda-doa/test/body.enc:/test/body.enc


volumes:
postgres:
certs:
client_certs:
postgres_data:
rabbitmq_data:
shared:
4 changes: 4 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ build-sda-sftp-inbox:
@cd sda-sftp-inbox && docker build -t ghcr.io/neicnordic/sensitive-data-archive:PR$$(date +%F)-sftp-inbox .
build-sda-admin:
@cd sda-admin && go build
build-sda-doa:
@cd sda-doa && docker build -t ghcr.io/neicnordic/sensitive-data-archive:PR$$(date +%F)-sda-doa .
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@cd sda-doa && docker build -t ghcr.io/neicnordic/sensitive-data-archive:PR$$(date +%F)-sda-doa .
@cd sda-doa && docker build -t ghcr.io/neicnordic/sensitive-data-archive:PR$$(date +%F)-doa .

In order to match the rest of the artifacts generated.



go-version-check: SHELL:=/bin/bash
Expand Down Expand Up @@ -127,6 +129,8 @@ integrationtest-sda-sync-run:
@PR_NUMBER=$$(date +%F) docker compose -f .github/integration/sda-sync-integration.yml run integration_test
integrationtest-sda-sync-down:
@PR_NUMBER=$$(date +%F) docker compose -f .github/integration/sda-sync-integration.yml down -v --remove-orphans
integrationtest-sda-doa:
@cd sda-doa/test/run && sh run_sda_doa_tests.sh
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rework this to match the rest of the test suite i.e.

integrationtest-sda-doa: build-all
	@PR_NUMBER=$$(date +%F) docker compose -f .github/integration/sda-doa-integration.yml run integration_test
	@PR_NUMBER=$$(date +%F) docker compose -f .github/integration/sda-doa-integration.yml down -v --remove-orphans
integrationtest-sda-doa-run:
	@PR_NUMBER=$$(date +%F) docker compose -f .github/integration/sda-doa-integration.yml run integration_test
integrationtest-sda-doa-down:
	@PR_NUMBER=$$(date +%F) docker compose -f .github/integration/sda-doa-integration.yml down -v --remove-orphans


# lint go code
lint-all: lint-sda lint-sda-download lint-sda-admin
Expand Down
2 changes: 1 addition & 1 deletion sda-doa/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM maven:3-eclipse-temurin-21-alpine as builder
FROM maven:3-eclipse-temurin-21-alpine AS builder

COPY pom.xml .

Expand Down
Loading
Loading