Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create container-build.yml #200

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
7fe5a6d
Create container-build.yml
aryanbhosale Sep 13, 2024
f749532
Update Dockerfile
aryanbhosale Sep 13, 2024
46f3722
updated docker
aryanbhosale Sep 14, 2024
3a4d998
qsf
aryanbhosale Sep 14, 2024
b0bb165
full app
aryanbhosale Sep 14, 2024
f053d0c
build
aryanbhosale Sep 14, 2024
839d39e
rm poetry
aryanbhosale Sep 14, 2024
3f55772
q
aryanbhosale Sep 14, 2024
3354c7d
not everything
aryanbhosale Sep 14, 2024
b70e84e
Merge branch 'openclimatefix:main' into main
aryanbhosale Sep 17, 2024
b025399
copy pyproj
aryanbhosale Sep 17, 2024
ccd9030
Merge branch 'main' of https://github.com/aryanbhosale/open-source-qu…
aryanbhosale Sep 17, 2024
441c002
namespace
aryanbhosale Sep 17, 2024
2e95d0b
verbose
aryanbhosale Sep 17, 2024
4c7c284
explicit
aryanbhosale Sep 17, 2024
96de732
additional deps
aryanbhosale Sep 17, 2024
669ab43
find
aryanbhosale Sep 17, 2024
eb92585
pandas add
aryanbhosale Sep 17, 2024
d33923a
rm unnc
aryanbhosale Sep 17, 2024
07e9d63
h5py
aryanbhosale Sep 17, 2024
abe105e
revert frontend dockerisation
aryanbhosale Sep 26, 2024
0f273a0
line
aryanbhosale Sep 26, 2024
81c4a29
rm healthcheck
aryanbhosale Sep 26, 2024
c68a872
numcodecs
aryanbhosale Sep 26, 2024
00f8952
explicit numcodecs
aryanbhosale Sep 26, 2024
50b4623
no deps flag
aryanbhosale Sep 26, 2024
48804e6
contraints
aryanbhosale Sep 26, 2024
5fc13e7
from uk pvnet
aryanbhosale Sep 26, 2024
cd14498
conda
aryanbhosale Sep 26, 2024
2f973d1
rm pytorch install numcodecs conda
aryanbhosale Sep 26, 2024
0de6465
Merge branch 'openclimatefix:main' into main
aryanbhosale Nov 12, 2024
6f7db85
Update Dockerfile
aryanbhosale Nov 12, 2024
aa5d012
Update docker-compose.yaml
aryanbhosale Nov 12, 2024
1c8de2f
Update Dockerfile
aryanbhosale Nov 12, 2024
2916fb8
arm and amd
aryanbhosale Nov 12, 2024
b548079
disable env
aryanbhosale Nov 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Base URL for the API
FASTAPI_BASE_URL = 1. With Docker: "http://web:8000", 2. Local: "http://localhost:8000"
LOG_LEVEL = "info"

# User needs to add their Enphase API details
ENPHASE_SYSTEM_ID = 'user_enphase_system_id'
ENPHASE_CLIENT_ID = 'user_enphase_client_id'
Expand Down
60 changes: 60 additions & 0 deletions .github/workflows/container-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Create and publish Container image

on:
push:
branches:
- main
pull_request:
branches:
- main

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up QEMU
uses: docker/setup-qemu-action@v2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Login to Container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=ref,event=branch
type=ref,event=pr
type=edge
type=sha

- name: Build and push
uses: docker/build-push-action@v4
with:
platforms: linux/amd64,linux/arm64
context: .
file: Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
33 changes: 24 additions & 9 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,20 +1,35 @@
# Use an official Python runtime as a parent image
FROM python:3.11-slim

# Set the working directory in the container
WORKDIR /app
# Install necessary build tools and curl
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
curl \
gcc \
libzstd-dev \
&& rm -rf /var/lib/apt/lists/*

# Copy the requirements.txt file
COPY requirements.txt .
# Install Rust using rustup
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# Add Rust to PATH
ENV PATH="/root/.cargo/bin:${PATH}"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to copy the pyproject.toml file over

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to copy the pyproject.toml file over

Aren't we copying all the files in the directory anyway?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not currently. In general we try to only copy the files we need.
COPY pyproject.toml . should do it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah see, below we do, but we copy them to app

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah see, below we do, but we copy them to app

So simply COPY pyproject.toml . Should work?

Copy link
Member Author

@aryanbhosale aryanbhosale Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#18 2.100 configuration error: tool.setuptools.packages must be valid exactly by one definition (0 matches found):

this is the error, it means the packages section of pyproject.toml isnt correctly set

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#18 2.100 configuration error: tool.setuptools.packages must be valid exactly by one definition (0 matches found):

this is the error, it means the packages section of pyproject.toml isnt correctly set

this seems to be the point of failure:
Failed to build numcodecs h5py
220.5 ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (numcodecs, h5py)

seems that h5py requires some additional dependencies , trying that

Copy link
Member Author

@aryanbhosale aryanbhosale Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay i tried to do a workaround here, tried installing h5py which was creating issues with numcodecs as seen in the error logs,
but i see this:

Dockerfile:14

12 |
13 | # Install h5py with no-binary flag
14 | >>> RUN pip install --no-binary h5py h5py
15 |
16 | # Copy the pyproject.toml file

ERROR: failed to solve: process "/bin/sh -c pip install --no-binary h5py h5py" did not complete successfully: exit code: 1

apparently everything fails at the first RUN pip install xyz(whatever dependency), are we missing anything @peterdudfield ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need pip install --no-binary=h5py h5py, I admit this is a weird problem though

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need pip install --no-binary=h5py h5py, I admit this is a weird problem though

Not really no, was just trying to see if installing it explicitly fixes the h5py, numcodec issue we see on docker, but rather it led us to another issue that the first pip install whatever always fails

# Copy the entire project directory (including quartz_solar_forecast)
# Set environment variables to disable SSE2 and AVX2 for numcodecs
ENV BLOSC_DISABLE_AVX2=1
ENV BLOSC_DISABLE_SSE2=1

# Set the working directory in the container
WORKDIR /app

# Copy the project files to the container
COPY . /app

# Install the quartz_solar_forecast package in editable mode
RUN pip install -e .
# Upgrade pip and setuptools
RUN pip install --upgrade pip setuptools wheel

# Install the project and its dependencies
RUN pip install .

# Expose port 8000 to the outside world
EXPOSE 8000
Expand Down
2 changes: 1 addition & 1 deletion dashboards/dashboard_2/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
st.session_state.redirect_url = ""

# Set up the base URL for the FastAPI server
FASTAPI_BASE_URL = "http://localhost:8000"
FASTAPI_BASE_URL = os.getenv("FASTAPI_BASE_URL")

# Get the directory of the current script
script_dir = os.path.dirname(os.path.abspath(__file__))
Expand Down
2 changes: 0 additions & 2 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,6 @@ services:
- "8000:8000"
env_file:
- .env
volumes:
- .:/app
depends_on:
- open-meteo-api
- open-meteo-sync
Expand Down
Loading