Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create container-build.yml #200

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
7fe5a6d
Create container-build.yml
aryanbhosale Sep 13, 2024
f749532
Update Dockerfile
aryanbhosale Sep 13, 2024
46f3722
updated docker
aryanbhosale Sep 14, 2024
3a4d998
qsf
aryanbhosale Sep 14, 2024
b0bb165
full app
aryanbhosale Sep 14, 2024
f053d0c
build
aryanbhosale Sep 14, 2024
839d39e
rm poetry
aryanbhosale Sep 14, 2024
3f55772
q
aryanbhosale Sep 14, 2024
3354c7d
not everything
aryanbhosale Sep 14, 2024
b70e84e
Merge branch 'openclimatefix:main' into main
aryanbhosale Sep 17, 2024
b025399
copy pyproj
aryanbhosale Sep 17, 2024
ccd9030
Merge branch 'main' of https://github.com/aryanbhosale/open-source-qu…
aryanbhosale Sep 17, 2024
441c002
namespace
aryanbhosale Sep 17, 2024
2e95d0b
verbose
aryanbhosale Sep 17, 2024
4c7c284
explicit
aryanbhosale Sep 17, 2024
96de732
additional deps
aryanbhosale Sep 17, 2024
669ab43
find
aryanbhosale Sep 17, 2024
eb92585
pandas add
aryanbhosale Sep 17, 2024
d33923a
rm unnc
aryanbhosale Sep 17, 2024
07e9d63
h5py
aryanbhosale Sep 17, 2024
abe105e
revert frontend dockerisation
aryanbhosale Sep 26, 2024
0f273a0
line
aryanbhosale Sep 26, 2024
81c4a29
rm healthcheck
aryanbhosale Sep 26, 2024
c68a872
numcodecs
aryanbhosale Sep 26, 2024
00f8952
explicit numcodecs
aryanbhosale Sep 26, 2024
50b4623
no deps flag
aryanbhosale Sep 26, 2024
48804e6
contraints
aryanbhosale Sep 26, 2024
5fc13e7
from uk pvnet
aryanbhosale Sep 26, 2024
cd14498
conda
aryanbhosale Sep 26, 2024
2f973d1
rm pytorch install numcodecs conda
aryanbhosale Sep 26, 2024
0de6465
Merge branch 'openclimatefix:main' into main
aryanbhosale Nov 12, 2024
6f7db85
Update Dockerfile
aryanbhosale Nov 12, 2024
aa5d012
Update docker-compose.yaml
aryanbhosale Nov 12, 2024
1c8de2f
Update Dockerfile
aryanbhosale Nov 12, 2024
2916fb8
arm and amd
aryanbhosale Nov 12, 2024
b548079
disable env
aryanbhosale Nov 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Base URL for the API
FASTAPI_BASE_URL = 1. With Docker: "http://web:8000", 2. Local: "http://localhost:8000"
LOG_LEVEL = "info"

# User needs to add their Enphase API details
ENPHASE_SYSTEM_ID = 'user_enphase_system_id'
ENPHASE_CLIENT_ID = 'user_enphase_client_id'
Expand Down
57 changes: 57 additions & 0 deletions .github/workflows/container-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Create and publish Container image

on:
push:
branches:
- main
pull_request:
branches:
- main

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Login to Container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=ref,event=branch
type=ref,event=pr
type=edge
type=sha

- name: Build and push
uses: docker/build-push-action@v4
with:
platforms: linux/amd64,linux/arm64
context: .
file: Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
16 changes: 5 additions & 11 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,14 @@ FROM python:3.11-slim
# Set the working directory in the container
WORKDIR /app

# Copy the requirements.txt file
COPY requirements.txt .

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to copy the pyproject.toml file over

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to copy the pyproject.toml file over

Aren't we copying all the files in the directory anyway?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not currently. In general we try to only copy the files we need.
COPY pyproject.toml . should do it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah see, below we do, but we copy them to app

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah see, below we do, but we copy them to app

So simply COPY pyproject.toml . Should work?

Copy link
Member Author

@aryanbhosale aryanbhosale Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#18 2.100 configuration error: tool.setuptools.packages must be valid exactly by one definition (0 matches found):

this is the error, it means the packages section of pyproject.toml isnt correctly set

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#18 2.100 configuration error: tool.setuptools.packages must be valid exactly by one definition (0 matches found):

this is the error, it means the packages section of pyproject.toml isnt correctly set

this seems to be the point of failure:
Failed to build numcodecs h5py
220.5 ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (numcodecs, h5py)

seems that h5py requires some additional dependencies , trying that

Copy link
Member Author

@aryanbhosale aryanbhosale Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay i tried to do a workaround here, tried installing h5py which was creating issues with numcodecs as seen in the error logs,
but i see this:

Dockerfile:14

12 |
13 | # Install h5py with no-binary flag
14 | >>> RUN pip install --no-binary h5py h5py
15 |
16 | # Copy the pyproject.toml file

ERROR: failed to solve: process "/bin/sh -c pip install --no-binary h5py h5py" did not complete successfully: exit code: 1

apparently everything fails at the first RUN pip install xyz(whatever dependency), are we missing anything @peterdudfield ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need pip install --no-binary=h5py h5py, I admit this is a weird problem though

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need pip install --no-binary=h5py h5py, I admit this is a weird problem though

Not really no, was just trying to see if installing it explicitly fixes the h5py, numcodec issue we see on docker, but rather it led us to another issue that the first pip install whatever always fails

# Copy the entire project directory (including quartz_solar_forecast)
COPY . /app

# Install the quartz_solar_forecast package in editable mode
RUN pip install -e .
RUN pip install .

# Expose port 8000 to the outside world
EXPOSE 8000
# Expose port 8000 and 8501 to the outside world
EXPOSE 8000 8501

# Run the application using python main.py
CMD ["python", "api/main.py"]
# The CMD will be provided by docker-compose.yml
CMD ["sh", "-c", "$CMD"]
4 changes: 4 additions & 0 deletions api/app/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@
allow_headers=["*"]
)

@app.get("/health")
async def root():
return {"status": "ok"}

@app.post("/forecast/")
def forecast(forecast_request: ForecastRequest):
site = forecast_request.site
Expand Down
2 changes: 1 addition & 1 deletion dashboards/dashboard_2/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
st.session_state.redirect_url = ""

# Set up the base URL for the FastAPI server
FASTAPI_BASE_URL = "http://localhost:8000"
FASTAPI_BASE_URL = os.getenv("FASTAPI_BASE_URL")

# Get the directory of the current script
script_dir = os.path.dirname(os.path.abspath(__file__))
Expand Down
37 changes: 31 additions & 6 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
x-shared_environment: &shared_environment
LOG_LEVEL: ${LOG_LEVEL:-info}
LOG_LEVEL: ${LOG_LEVEL}

services:
open-meteo-api:
Expand Down Expand Up @@ -28,17 +28,42 @@ services:
build:
context: .
dockerfile: Dockerfile
container_name: fastapi-app
container_name: quartz-solar-forecast-api
ports:
- "8000:8000"
env_file:
- .env
volumes:
- .:/app
depends_on:
- open-meteo-api
- open-meteo-sync
restart: always
environment:
<<: *shared_environment
command: python api/main.py
healthcheck:
test: ["CMD", "python", "-c", "import requests; requests.get('http://localhost:8000/health')"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s

streamlit:
build:
context: .
dockerfile: Dockerfile
container_name: quartz-solar-forecast-frontend
ports:
- "8501:8501"
volumes:
- .:/app
working_dir: /app
restart: always
environment:
<<: *shared_environment
FASTAPI_BASE_URL: ${FASTAPI_BASE_URL}
command: streamlit run dashboards/dashboard_2/app.py
depends_on:
web:
condition: service_healthy

volumes:
data:
data:
Loading