Slow Python Docker Image Build when installingnumpy or pandas

[mwakaba2]

Hi! I have a detection bot that utilizes python data science related packages pandas and numpy, and it took ~2 hours to build the docker image.

$ time npm run push

> [email protected] push
> forta-agent push

building agent image...
pushing agent image to repository...
successfully pushed image with reference XXXXX
npm run push  12.92s user 8.96s system 0% cpu 2:05:33.56 total

It took the most time installing pandas and numpy because alpine was rebuilding from source code for these packages instead of copying the wheel packages over which is a much faster and reliable python package installation method. However, Alpine platform doesn't support wheel packages, so we may need to switch to a different base builder image, e.g. debian based for python bots.

[haseebrabbani]

yes, there are plans to fix this by decoupling the nodeJS CLI from bot execution inside the container (which is currently why we use a node-alpine base image). the dependent issue is tracked here: #145

an alternative solution in the meantime could be to just use a python base image, and then install node to it so the CLI can be invoked.

[mwakaba2]

This alternative solution provided by vvlovsky works!

# Build stage: compile Python dependencies
FROM ubuntu:focal as builder
RUN apt-get update -y && apt-get upgrade -y
RUN apt-get install -y python3 pip
RUN python3 -m pip install --upgrade pip
COPY requirements.txt ./
RUN python3 -m pip install --user -r requirements.txt

# Final stage: copy over Python dependencies and install production Node dependencies
FROM ubuntu:focal
# this python version should match the build stage python version
RUN apt-get update
RUN apt-get -y install curl gnupg
RUN curl -sL https://deb.nodesource.com/setup_14.x  | bash -
RUN apt-get -y install nodejs
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local:$PATH
ENV NODE_ENV=production
# Uncomment the following line to enable agent logging
LABEL "network.forta.settings.agent-logs.enable"="true"
WORKDIR /app
COPY ./src ./src
COPY package*.json ./
RUN npm ci --production
CMD [ "npm", "run", "start:prod" ]