dis-wagtail

The Wagtail CMS for managing and publishing content for the Office for National Statistics (ONS)

---

Table of Contents

• Getting Started
  • Pre-requisites
  • Setup
    • Using Docker
    • Running locally with supporting services in Docker
• Development
  • Front-end tooling
  • Adding Python packages
  • Run Tests with Coverage
  • Linting and Formatting
    • Python
    • Front-end
    • pre-commit
    • Megalinter
  • Django Migrations
• Contributing
• License

For further developer documentation see docs

Getting Started

To get a local copy up and running, follow the steps below.

Pre-requisites

Ensure you have the following installed:

1. Python: Version specified in .python-version. We recommend using pyenv for managing Python versions.
2. Poetry: This is used to manage package dependencies and virtual environments.
3. Colima for running the project in Docker containers.
4. PostgreSQL for the database. Provided as container via docker-compose.yml when using the Docker setup.
5. Node and nvm (Node Version Manager) for front-end tooling.
6. JQ for the step in the build that installs the design system templates
7. Operation System: Ubuntu/MacOS

Setup

1. Clone the repository

   git clone [email protected]:ONSdigital/dis-wagtail.git

2. Install dependencies

   Poetry is used to manage dependencies in this project. For more information, read the Poetry documentation.

   To install all dependencies, including development dependencies, run:

   make install-dev

   To install only production dependencies, run:

   make install

Using Docker

Follow these steps to set up and run the project using Docker.

1. Build and Start the Containers

   # pull the supporting containers
   make compose-pull
   
   # build the main application image
   make compose-build
   
   # start the containers
   make compose-up

2. Migrations and Superuser Creation

   If this is your first time setting up the project, you’ll need to run migrations to set up the database schema and create an administrative user.

   # ssh into the web container
   make docker-shell
   
   # Run database migrations
   make migrate
   
   # Create a superuser for accessing the admin interface
   make createsuperuser

3. Start Django Inside the Container

   Once the containers are running, you need to manually start Django from within the web container. This allows for running both the Django server and any additional background services (e.g., schedulers).

   ⚠️ WARNING The honcho command will pick up your local mounted .env file when running via docker-compose. Ensure that you comment out any variables in the .env file which might cause clashes in the container context as they will take precedence when running honcho start.

   # Start both Django and the scheduler using Honcho
   honcho start
   
   # This is not needed if you used `honcho start`
   make runserver

You can then access the admin at http://0.0.0.0:8000/admin/ or http://localhost:8000/admin/.

Running locally with supporting services in Docker

You can also run the main application locally with the supporting backend services such as the Postgres and Redis running in Docker. This can be useful when you want to make changes that require the app to be restarted in order to be picked up.

For this method you can specify the runtime configuration either in your IDE (for PyCharm see here), or copy the .development.env and rename it to .env which will allow Django to pick up the config.

Once you create the .env file, and you'd like to switch back to running the application in a container, the .env file will be accessible inside the containers and it will be picked up by the honcho command. In order to avoid conflicts you may need to comment out some variables (such as DATABASE_URL and REDIS_URL) in the .env file.

Note

When running the application locally in a virtual environment via Poetry the .env file will not be picked up automatically. For this to work you'll need to install the poetry-plugin-dotenv. However, if you installed Poetry with brew rather than pip that currently isn't going to work (see the issue) and you'll need to install an older and seemingly no longer maintained poetry-dotenv-plugin.

In order to run it:

1. Pull the images of the supporting services.

   make compose-dev-pull

2. Start the supporting services in Docker.

   make compose-dev-up

3. Run the below command to apply the necessary pre-run steps, which include:

   • loading design system templates,
   • collecting the static files,
   • generating and applying database migrations,
   • creating a superuser with:
     • username: admin
     • password: changeme # pragma: allowlist secret

   make dev-init

4. Run the Django server locally via your IDE or with the following command:

   make runserver

Development

Get started with development by running the following commands. Before proceeding, make sure you have the development dependencies installed using the make install-dev command.

A Makefile is provided to simplify common development tasks. To view all available commands, run:

make

Front-end tooling

While the end goal is to have all front-end elements in the Design System, the new design introduces a number of components that we need to build and contributed to the DS. In order to aid development and avoid being blocked by the DS, we will use modern front-end tooling for that.

Here are the common commands:

# Install front-end dependencies.
npm install
# Start the Webpack build in watch mode, without live-reload.
npm run start
# Start the Webpack server build on port 3000 only with live-reload.
npm run start:reload
# Do a one-off Webpack development build.
npm run build
# Do a one-off Webpack production build.
npm run build:prod

Adding Python packages

Python packages can be installed using poetry in the web container:

make docker-shell
poetry add wagtailmedia

Run Tests with Coverage

To run the tests and check coverage, run:

make test

During tests, the cms.settings.test settings module is used. When running test without using make test, ensure this settings module is used.

Linting and Formatting

Various tools are used to lint and format the code in this project.

Python

The project uses Ruff and pylint for linting and formatting of the Python code.

The tools are configured using the pyproject.toml file.

To lint the Python code, run:

make lint

To auto-format the Python code, and correct fixable linting issues, run:

make format

Front-end

# lint and format custom CSS/JS
npm run lint
# only CSS
npm run lint:css
# only JS
npm run lint:js
# check css, js, markdown and yaml formatting
npm run lint:format
# format
npm run format

pre-commit

Note that this project has configuration for pre-commit. To set up locally:

# if you don't have it yet, globally
pip install pre-commit

# in the project directory, initialize pre-commit
pre-commit install

# Optional, run all checks once for this, then the checks will run only on the changed files
pre-commit run --all-files

The detect-secrets pre-commit hook requires a baseline secrets file to be included. If you need to,
you can update this file, e.g. when adding dummy secrets for unit tests:

poetry run detect-secrets scan > .secrets.baseline

MegaLinter (Lint/Format non-python files)

MegaLinter is utilised to lint the non-python files in the project. It offers a single interface to execute a suite of linters for multiple languages and formats, ensuring adherence to best practices and maintaining consistency across the repository without the need to install each linter individually.

MegaLinter examines various file types and tools, including GitHub Actions, Shell scripts, Dockerfile, etc. It is configured using the .mega-linter.yml file.

To run MegaLinter, ensure you have Docker installed on your system.

Note: The initial run may take some time to download the Docker image. However, subsequent executions will be considerably faster due to Docker caching. 🚀

To start the linter and automatically rectify fixable issues, run:

make megalint

Django Migrations

Wagtail is built on Django and changes to its models may require generating and running schema migrations. For full details see the Django documentation on migrations

Below are the commands you will most commonly use, note that these have to be run inside the container.

# Check if you need to generate any new migrations after changes to the model
make makemigrations-check

# Generate migrations
make makemigrations

# Apply migrations. Needed if new migrations have been generated (either by you, or via upstream code)
make migrate

Contributing

See CONTRIBUTING.md for details.

License

See LICENSE for details.