Service WMTS

Branch	Status
develop
master

Table of content

• Table of content
• Summary of the project
• How to run locally
  • dependencies
  • Setting up to work
  • Running the server locally
  • Unittesting
  • Testing locally
  • Docker helpers
  • Linting and formatting your work
• Service configuration
  • General configuration
  • Cache Configuration
  • WMS configuration
    • WMS Backend Connection settings
  • S3 2nd level caching settings
  • Get Capabilities settings
• GetTile
  • WMTS Config Cache
  • Query Parameters
    • mode - Operation Mode
      • default
      • preview
    • nodata
  • S3 2nd level caching
• GetCapabilities
• OpenAPI
  • Redoc Renderer
  • Swagger UI Renderer
  • Spec linting

Summary of the project

This application translates WMTS requests into WMS requests and optionally caches the responses in S3 (by default this second level caching is disable). It uses gatilegrid module to perform this translation. It also serve the WMTS GetCapabilities as defined in the OGC Standard.

How to run locally

dependencies

The Make targets assume you have

• bash
• python3.7
• python3.7-venv
• python3.7-dev
• pipenv
• gcc
• libpq-dev
• curl
• docker
• docker-compose-plugin

installed.

Setting up to work

First, you'll need to clone the repo

git clone [email protected]:geoadmin/service-wmts

Then, you can run the setup target to ensure you have everything needed to develop, test and serve locally

make setup

Then you need to run some local containers (DB, WMS-BOD)

docker compose up

That's it, you're ready to work.

Running the server locally

To run locally enter

make serve

or the following to use gunicorn as web server:

make gunicornserve

or the following to run the service in a docker image locally:

make dockerrun

Unittesting

To run the unittest enter:

make test

To run only a single test module/class/method enter:

pipenv shell # activate the virtual environment
nose2 -c tests/unittest.cfg -t ${PWD} tests.unit_tests.test_functional

Testing locally

To test the localhost server you can use

curl http://localhost:5000/1.0.0/inline_points/default/current/4326/15/34136/7882.png

You can then check with the minio browser that a file has been saved on the S3 cache; http://localhost:9001 (use the credentials from minio.env)

Docker helpers

To build a local docker image tagged as service-wmts:local-${USER}-${GIT_HASH_SHORT} you can use

make dockerbuild

To push the image on the ECR repository use the following two commands

make dockerlogin
make dockerpush

Linting and formatting your work

In order to have a consistent code style the code should be formatted using yapf. Also to avoid syntax errors and non pythonic idioms code, the project uses the pylint linter. Both formatting and linter can be manually run using the following command:

make format-lint

The openapi spec has also a linter, and the CI will check for spec errors.

make lint-spec

Formatting and linting should be at best integrated inside the IDE, for this look at Integrate yapf and pylint into IDE

NOTE: CI will failed if the code is not properly formatted

Service configuration

All settings can be found in app/settings.py but here below you have the most important one described.

General configuration

Variable	Default	Description
ENV_FILE		Configuration environment file. Note that environment variable will overwrite values from environment file. This is especially used for local development
WMTS_PORT	9000	Port of the service
LOGGING_CFG	./config/logging-cfg-local.yml	Logging configuration file
LOGS_DIR	./logs	Logging output directory. Only used by local logging configuration file.
DEFAULT_MODE	default	Default operation mode see Operation Mode
UNITTEST_SKIP_XML_VALIDATION	False	Validating Get Capabilities XML output in Unittest takes time (~32s), therefore with this variable you can skip this test.
FORWARED_ALLOW_IPS	*	Sets the gunicorn forwarded_allow_ips. See Gunicorn Doc. This setting is required in order to secure_scheme_headers to work.
FORWARDED_PROTO_HEADER_NAME	X-Forwarded-Proto	Sets gunicorn secure_scheme_headers parameter to {${FORWARDED_PROTO_HEADER_NAME}: 'https'}. This settings is required in order to generate correct URLs in the service responses. See Gunicorn Doc.
SCRIPT_NAME	''	If the service is behind a reverse proxy and not served at the root, the route prefix must be set in SCRIPT_NAME.
WMTS_WORKERS	0	WMTS service number of workers. 0 or negative value means that the number of worker are computed from the number of cpu.
WSGI_TIMEOUT	45	WSGI timeout.
SQLALCHEMY_POOL_PRE_PING	False	True will enable the connection pool “pre-ping” feature that tests connections for liveness upon each checkout. This will trigger a recycle of outdated, stale connections. Activating this option will help to get rid of idle connection timeout errors but has a slight influence on the performance.
SQLALCHEMY_ISOLTATION_LEVEL	READ COMMITTED	affects the transaction isolation level of the database connection.
SQLALCHEMY_POOL_RECYCLE	20	this setting causes the pool to recycle connections after the given number of seconds has passed
SQLALCHEMY_POOL_SIZE	20	the number of connections to keep open inside the connection pool
SQLALCHEMY_MAX_OVERFLOW	-1	the number of connections to allow in connection pool “overflow”, -1 will disable overflow.
GUNICORN_WORKER_TMP_DIR	None	This should be set to an tmpfs file system for better performance. See https://docs.gunicorn.org/en/stable/settings.html#worker-tmp-dir.

Cache Configuration

NOTE: max-age is usually used by the Browser, while s-maxage by the server cache (e.g. CloudFront cache, see CloudFront - Managing how long content stays in the cache (expiration)).

Variable	Default	Description
GET_TILE_DEFAULT_CACHE	'public, max-age={browser_cache_ttl}, s-maxage=5184000'	Default cache settings for GetTile requests (default to 2 months). browser_cache_ttl is set to GET_TILE_BROWSER_CACHE_MAX_TTL. Note the s-maxage directive is usually overridden by the cache_ttl value from BOD.
GET_TILE_ERROR_DEFAULT_CACHE	'public, max-age=3600'	Default cache settings for GetTile error responses (default to 1 hour).
ERROR_5XX_DEFAULT_CACHE	public, max-age=5	Default cache settings for 5xx HTTP errors
GET_TILE_CACHE_TEMPLATE	'public, max-age={browser_cache_ttl}, s-maxage={cf_cache_ttl}'	GetTile cache-control header template used with the cache_ttl value if present for the layer in the BOD. The browser_cache_ttl value will be set cache_ttl or to the GET_TILE_BROWSER_CACHE_MAX_TTL value if the later is bigger.
GET_TILE_BROWSER_CACHE_MAX_TTL	3600	Maximum value used for the GetTile Cache-Control max-age header in case of cache_ttl configured in BOD.
GET_CAP_DEFAULT_CACHE	'public, max-age=3600, s-maxage=5184000'	GetCapabilities cache-control header value (default to 2 months).
CHECKER_DEFAULT_CACHE	'public, max-age=120'	Checker cache-control header value (default to 2 minutes)

WMS configuration

Variable	Default	Description
WMS_HOST	localhost	Host name of the WMS service
WMS_PORT	80	Port of the WMS service
BOD_DB_HOST		WMS Postgresql database hostname
BOD_DB_PORT	5432	WMS Postgresql database port
BOD_DB_NAME		WMS database name
BOD_DB_USER		WMS database user name
BOD_DB_PASSWD		WMS database user password

WMS Backend Connection settings

See https://requests.readthedocs.io/en/latest/api/#requests.adapters.HTTPAdapter for description of the following variables.

Variable	Default
WMS_BACKEND_POOL_CONNECTION	10
WMS_BACKEND_POOL_MAXSIZE	10
WMS_BACKEND_POOL_BLOCK	False
WMS_BACKEND_CONNECTION_MAX_RETRY	0

S3 2nd level caching settings

Variable	Default	Description
AWS_ACCESS_KEY_ID		AWS access key for S3
AWS_SECRET_ACCESS_KEY		AWS access secret for S3
AWS_S3_BUCKET_NAME	service-wmts-cache	S3 bucket name used for 2nd level caching
AWS_S3_REGION_NAME		AWS Region
AWS_S3_ENDPOINT_URL		AWS endpoint url if not standard. This allow to use a local S3 instance with minio
HTTP_CLIENT_TIMEOUT	1	HTTP client timeout in seconds for AWS S3 GetTile requests

Get Capabilities settings

Variable	Default	Description
APP_STAGING	'prod'	Filter the capabilities for this staging
LEGENDS_BASE_URL	"https://api3.geo.admin.ch/static/images/legends"	Legend base url used in GetCapabilities

GetTile

Get Tile endpoint; /1.0.0/{layer_id}/default/{time}/{srid}/{zoom}/{col}/{row}.{extension}, returns a Web Map Tile. It uses a WMTS Configuration taken from BOD that is cached upon service starts.

WMTS Config Cache

For performance reason the WMTS Config needed to return a Web Map Tile, is cached once locally in Memory during the startup of the service. To update this cache, you need to restart the service.

Query Parameters

mode - Operation Mode

default

mode=default

This mode is the default mode for the WMTS service. It is meant to be integrated with the full stack.

The steps are:

1. Check if the tile is present on S3
2. If yes return the S3 tile to the client
3. Otherwise request the tile from the WMS server image
4. If needed puts the tile in S3 after closing the TCP connection with client
5. Return the WMS image to the client

preview

mode=preview

This mode is meant to be used to test the configuration of the wms server (skipping caching in S3).

The steps are:

1. Request the WMS Server image
2. Return the WMS image to the client

nodata

No data parameter can be used for tile creation.

nodata=true

Returns OK if the image was successfully fetched and created. Can be used for tile generation.

Note the Tile will be put into 2nd level S3 cache if needed.

S3 2nd level caching

Because some tiles are very slow to generates; up to 30 seconds, those ones are also cached into a 2nd level cache on S3. Tiles are saved on S3 based on the BOD configuration; s3_resolution_max. This cache is more deterministic as any other CDN cache (e.g. CloudFront cache).

GetCapabilities

The following endpoint alias for GetCapabilities are implemented:

• /EPSG/<int:epsg>/<string:lang>/1.0.0/WMTSCapabilities.xml
• /EPSG/<int:epsg>/1.0.0/WMTSCapabilities.xml (lang=de)
• /1.0.0/WMTSCapabilities.EPSG.<int:epsg>.xml (lang=de)
• /1.0.0/WMTSCapabilities.xml?lang=<string:lang>&epsg=<int:epsg> (default query: lan=de&epsg=21781)

Those endpoints are using the view from the BOD service-wmts schema.

OpenAPI

The service uses OpenAPI Specification to document its endpoints. This documentation is in the openapi.yml file. This file can be used with either Redoc or Swagger UI renderer.

Redoc Renderer

To render the spec using Redoc do as follow

make serve-spec-redoc

Then open localhost:8080 in your Browser.

Swagger UI Renderer

To render the spec using Swagger UI do as follow

make serve-spec-swagger

Then open localhost:8080/swagger in your Browser.

See also Swagger UI Installation for more info on the Swagger UI docker image.

Spec linting

The OpenAPI follow some rules and must be validated using make lint-spec