Rendering CAMIS in Docker

[michaelwalshe]

I'm not sure if anyone else has had this problem, but recently I ran into some issues developing a new comparison where although everything ran fine locally (tale as old as time - works on my machine!) when it came time to create a PR the GitHub Action that tests rendering the site errored.

This led to a somewhat torturous debug cycle, where I would make some changes locally, push a commit, wait the ~11 minutes to build the site, only for it to error somewhere else 😭 . To fix this, I setup my development environment in WSL so I could effectively work in a much more similar environment. However, this can still lead to various issues with a non-reproducible environment (e.g. having an R package installed outside of the renv, which renv doesn't realise is a dependency but which is being used in the render process).

So as a more complete fix, I rewrote a Dockerfile I've used previously to setup an R/Python environment. This will create the entire environment from scratch, explicitly installing any dependencies, in a very similar way to the PR on GH actions, all without using up the allocation of free resources on GH. To further speed up the process, I've used a couple of tricks to be able to use a renv and pip cache in a docker image, and I would also set freeze: auto in the quarto settings to not re-render previously rendered pages. This gets the total render time down to a few minutes.

See below for the 3 files I added to the root directory of the project. To fully render it, you just have to run render_docker.ps1 with PowerShell, this then creates a local site in your _site directory.

I'm not advocating that this should be added to the main CAMIS repo, using Docker is a bit overkill and would likely discourage potential contributors. However, I wanted the solution I used documented somewhere here so that if anyone needs to fix bugs with the rendering process locally, they don't have to repeat the problem solving I did.

Dockerfile

# syntax=docker.io/docker/dockerfile:1.7-labs
FROM debian:bookworm-slim

# Setup directory and a volume for the output site files
WORKDIR /project
VOLUME /project/_site

# Set volumes to use as the renv, pip, and quarto execution caches. If provided, this will
# speed up builds that use the same dependencies.
VOLUME /project/_freeze
VOLUME /project/renv/library/_cache
VOLUME /project/renv/python/_cache
ENV RENV_PATHS_CACHE="/project/renv/library/_cache"
ENV PIP_CACHE_DIR="/project/renv/python/_cache"

# Install system deps
RUN apt-get -y update; apt-get -y install curl libcurl4-openssl-dev cmake libv8-dev sqlite3

# Install Quarto
ENV QUARTO_VERSION="1.5.54"
RUN mkdir -p /opt/quarto/${QUARTO_VERSION} \
    && curl -o quarto.tar.gz -L \
        "https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.tar.gz" \
    && tar -zxvf quarto.tar.gz \
        -C "/opt/quarto/${QUARTO_VERSION}" \
        --strip-components=1 \
    && rm quarto.tar.gz \
    && ln -s /opt/quarto/${QUARTO_VERSION}/bin/quarto /usr/local/bin/quarto

# Install Python
ENV PYTHON_VERSION="3.12.5"
RUN curl -O https://cdn.rstudio.com/python/debian-12/pkgs/python-${PYTHON_VERSION}_1_amd64.deb \
    && apt-get -y install ./python-${PYTHON_VERSION}_1_amd64.deb \
    && rm ./python-${PYTHON_VERSION}_1_amd64.deb \
    && /opt/python/"${PYTHON_VERSION}"/bin/pip install --upgrade pip setuptools wheel \
    && echo 'PATH=/opt/python/'${PYTHON_VERSION}'/bin:$PATH' > /etc/profile.d/python.sh

# Install R
ENV R_VERSION=4.4.1
RUN curl -O https://cdn.rstudio.com/r/debian-12/pkgs/r-${R_VERSION}_1_amd64.deb \
    && apt-get install -y ./r-${R_VERSION}_1_amd64.deb \
    && ln -s /opt/R/${R_VERSION}/bin/R /usr/local/bin/R \
    && ln -s /opt/R/${R_VERSION}/bin/Rscript /usr/local/bin/Rscript \
    && echo "options(repos = c(CRAN = 'https://packagemanager.posit.co/cran/__linux__/bookworm/latest'))" >> /opt/R/${R_VERSION}/lib/R/etc/Rprofile.site \
    && R -e 'install.packages("renv")'

# Ensure we use the project venv - note that this won't exist until we run
# renv::restore() in the project, which we keep out of the build stage to let
# us mount a renv cache volume
ENV PATH="/project/renv/python/virtualenvs/renv-python-3.12/bin:$PATH"

# Copy across the workspace, will skip patterns in .dockerignore
COPY . .

# Run our build script - currently just a shell and rendering is done via docker run commands
# CMD ["/bin/bash", "quarto", "render", "--output-dir", "/app/_site"]
CMD ["/bin/bash"]

render_docker.ps1

# Build our docker image, using the Dockerfile in the current directory
docker build -t camis .

# Command to render our site. Start by restoring R packages - use binaries to speed up
# and avoid compilation dependencies (will be even faster if a cache is available). This
# also creates the Python venv. Then render an _site2 dir, and move the contents to the mounted
# volume (as otherwise it would error due to the quarto trying to delete _site)
$cmd1 = @'
    R -e \"renv::restore(repos='https://packagemanager.posit.co/cran/__linux__/bookworm/latest')\" \
    && quarto render --output-dir _site2 \
    && rm -r _site/* \
    && mv _site2/* _site
'@

# Run the above command via docker, with mounted volumes for the site, and R/Python/Quarto caches
docker run --rm `
    -v "${pwd}/_site:/project/_site" `
    -v "${pwd}/renv/library/_cache:/project/renv/library/_cache" `
    -v "${pwd}/renv/python/_cache:/project/renv/python/_cache" `
    -v "${pwd}/_freeze:/project/_freeze" `
    camis `
    /bin/bash -c "${cmd1}"

.dockerignore

.Rproj.user
.Rhistory
.RData
.Ruserdata
/_site/
/.quarto/
.DS_Store
**/renv/library/
**/renv/local/
**/renv/cellar/
**/renv/lock/
**/renv/python/
**/renv/sandbox/
**/renv/staging/