Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(upgrade): upgrade to v3 of the mlops-python-package, replacing p… #1

Merged
merged 5 commits into from
Dec 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# These are supported funding model platforms

# github: ["MLOps-Courses"]
custom: ["https://donate.stripe.com/4gw8xT9oVbCc98s7ss"]
16 changes: 10 additions & 6 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Tests
on:
pull_request:
branches:
- main
- '*'
permissions:
contents: read
id-token: write
Expand All @@ -15,9 +15,13 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install
run: |
pipx install poetry
poetry install
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version-file: .python-version
- name: Test
run: poetry run pytest tests/
run: uv run invoke test
3 changes: 0 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@
/.idea/
/.vscode/

# Poetry
poetry.lock

# Python
.venv/
*.py[cod]
Expand Down
8 changes: 2 additions & 6 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,14 @@
# https://pre-commit.com/hooks.html

default_language_version:
python: python3
python: python3.12
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
rev: v5.0.0
hooks:
- id: check-added-large-files
- id: check-case-conflict
- id: check-merge-conflict
- id: end-of-file-fixer
- id: mixed-line-ending
- id: trailing-whitespace
- repo: https://github.com/python-poetry/poetry
rev: 1.8.3
hooks:
- id: poetry-check
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
## v3.0.0 (2024-12-15)

### Fix

- **project**: switch from mlflow to uv

### Refactor

- **cicd**: fix ci/cd bug

## v2.0.0 (2024-12-15)

### Feat

- **upgrade**: upgrade to v3 of the mlops-python-package, replacing poetry with uv

## v1.0.0 (2024-07-28)
2 changes: 0 additions & 2 deletions LICENSE.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
Copyright 2024 Médéric HURIER (Fmind)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
Expand Down
13 changes: 6 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# MLOps Package Cookiecutter
# Cookiecutter - MLOps Package

[![Release](https://img.shields.io/github/v/release/fmind/cookiecutter-mlops-package)](https://github.com/fmind/cookiecutter-mlops-package/releases)
[![License](https://img.shields.io/github/license/fmind/cookiecutter-mlops-package)](https://github.com/fmind/cookiecutter-mlops-package/blob/main/LICENSE.txt)
Expand All @@ -10,22 +10,22 @@ The template provides a robust foundation for building, testing, packaging, and
**Related resources**:
- **[MLOps Coding Course (Learning)](https://mlops-coding-course.fmind.dev/)**: Learn how to create, develop, and maintain a state-of-the-art MLOps code base.
- **[MLOps Python Package (Example)](https://github.com/fmind/mlops-python-package)**: Kickstart your MLOps initiative with a flexible, robust, and productive Python package.
- **[LLMOps Coding Package (Example)](https://github.com/callmesora/llmops-python-package/)**: Example with best practices and tools to support your LLMOps projects.

## Philosophy

This [Cookiecutter](https://cookiecutter.readthedocs.io/) is designed to be a common ground for diverse MLOps environments. Whether you're working with [Kubernetes](https://www.kubeflow.org/), [Vertex AI](https://cloud.google.com/vertex-ai), [Databricks](https://www.databricks.com/), [Azure ML](https://azure.microsoft.com/en-us/products/machine-learning), or [AWS SageMaker](https://aws.amazon.com/sagemaker/), the core principles of using Python packages and Docker images remain consistent.

This template equips you with the essentials for creating, testing, and packaging your code, providing a solid base for integration into your chosen platform. To fully leverage its capabilities within a specific environment, you might need to combine it with external tools like [Airflow](https://airflow.apache.org/) for orchestration or platform-specific SDKs for deployment.
This template equips you with the essentials for creating, testing, and packaging your AI/ML code, providing a solid base for integration into your chosen MLOps platform. To fully leverage its capabilities within a specific environment, you might need to combine it with external tools like [Airflow](https://airflow.apache.org/) for orchestration or platform-specific SDKs for deployment.

You have the freedom to structure your `src/` and `tests/` directories according to your preferences. Alternatively, you can draw inspiration from the structure used in the [MLOps Python Package](https://github.com/fmind/mlops-python-package) project for a ready-made implementation.

## Key Features

* **Streamlined Project Structure:** A well-defined directory layout for source code, tests, documentation, tasks, and Docker configurations.
* **Poetry Integration:** Effortless dependency management and packaging with [Poetry](https://python-poetry.org/).
* **Uv Integration:** Effortless dependency management and packaging with [uv](https://docs.astral.sh/uv/).
* **Automated Testing and Checks:** Pre-configured workflows using [Pytest](https://docs.pytest.org/), [Ruff](https://docs.astral.sh/ruff/), [Mypy](https://mypy.readthedocs.io/), [Bandit](https://bandit.readthedocs.io/), and [Coverage](https://coverage.readthedocs.io/) to ensure code quality, style, security, and type safety.
* **Pre-commit Hooks:** Automatic code formatting and linting with [Ruff](https://docs.astral.sh/ruff/) and other pre-commit hooks to maintain consistency.
* **MLflow Project Ready:** An MLproject file for executing jobs using [MLflow](https://mlflow.org/), allowing for easy experimentation and tracking.
* **Dockerized Deployment:** Dockerfile and docker-compose.yml for building and running the package within a containerized environment ([Docker](https://www.docker.com/)).
* **Invoke Task Automation:** [PyInvoke](https://www.pyinvoke.org/) tasks to simplify development workflows such as cleaning, installing, formatting, checking, building, documenting, and running MLflow projects.
* **Comprehensive Documentation:** [pdoc](https://pdoc.dev/) generates API documentation, and Markdown files provide clear usage instructions.
Expand All @@ -51,7 +51,7 @@ You'll be prompted for the following variables:
- `version`: The initial version of your project.
- `description`: A brief description of your project.
- `python_version`: The Python version to use (e.g., 3.12).
- `mlflow_version`: The MLflow version to use (e.g., 2.14.3).
- `mlflow_version`: The MLflow version to use (e.g., 2.19.0).

2. **Initialize a git repository:**

Expand All @@ -73,7 +73,6 @@ git init
- `tasks/`: PyInvoke tasks for automation.
- `Dockerfile`: Configuration for building your Docker image.
- `docker-compose.yml`: Orchestration file for running MLflow and your project.
- `MLproject`: MLflow project definition.

5. **Start developing!**

Expand All @@ -97,7 +96,7 @@ After installing dependencies and setting up MLflow:
invoke projects
```

This will execute the "main" job defined in your [`MLproject`](https://github.com/fmind/cookiecutter-mlops-package/blob/main/%7B%7Bcookiecutter.repository%7D%7D/MLproject) file. You can specify different jobs using the `-P job=your_job_name` flag.
This will execute the job with the configuration file in your `confs` folder.

### Building and Running Your Docker Image

Expand Down
3 changes: 2 additions & 1 deletion cookiecutter-mlops-package.code-workspace
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
},
"extensions": {
"recommendations": [
"ms-python.python",
"dchanco.vsc-invoke",
"ms-python.python"
]
}
}
2 changes: 1 addition & 1 deletion cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"version": "0.1.0",
"description": "TODO",
"python_version": "3.12",
"mlflow_version": "2.14.3",
"mlflow_version": "2.19.0",
"__prompts__": {
"user": "GitHub User",
"name": "Project Name",
Expand Down
5 changes: 0 additions & 5 deletions poetry.toml

This file was deleted.

51 changes: 25 additions & 26 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,45 +1,44 @@
# https://python-poetry.org/docs/pyproject/
# https://docs.astral.sh/uv/reference/settings/
# https://packaging.python.org/en/latest/guides/writing-pyproject-toml/

# PACKAGE
# PROJECT

[tool.poetry]
[project]
name = "cookiecutter-mlops-package"
version = "1.0.0"
description = "Build and deploy Python packages and Docker images for MLOps tasks."
repository = "https://github.com/fmind/cookiecutter-mlops-package"
authors = ["Médéric HURIER <[email protected]>"]
version = "3.0.0"
description = "Build and deploy Python packages and Docker images for MLOps projects."
authors = [{ name = "Médéric HURIER", email = "[email protected]" }]
readme = "README.md"
license = "MIT"
package-mode = false
requires-python = ">=3.12"
license = { file = "LICENSE.txt" }

# DEPENDENCIES

[tool.poetry.dependencies]
python = "^3.12"
[dependency-groups]
dev = [
"commitizen>=4.1.0",
"invoke>=2.2.0",
"pre-commit>=4.0.1",
"pytest>=8.3.4",
"pytest-cookies>=0.7.0",
"pytest-shell-utilities>=1.9.7",
]

[tool.poetry.group.dev.dependencies]
commitizen = "^3.28.0"
invoke = "^2.2.0"
pre-commit = "^3.7.1"
pytest = "^8.3.2"
pytest-cookies = "^0.7.0"
pytest-shell-utilities = "^1.9.0"
# TOOLS

# CONFIGURATIONS
[tool.uv]
package = false

[tool.bandit]
targets = ["src"]

[tool.commitizen]
name = "cz_conventional_commits"
tag_format = "v$version"
version_scheme = "pep440"
version_provider = "poetry"
version_provider = "pep621"
update_changelog_on_bump = true

[tool.pytest.ini_options]
log_cli = true
log_cli_level = "INFO"

# SYSTEMS

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
17 changes: 14 additions & 3 deletions tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,30 @@

# %% TASKS


@task
def clean(ctx: Context) -> None:
"""Clean the project."""
ctx.run("rm -rf .venv/")
ctx.run("rm -rf .mypy_cache/")
ctx.run("rm -rf .pytest_cache/")
ctx.run("find . -type f -name '*.py[co]' -delete")
ctx.run(r"find . -type d -name __pycache__ -exec rm -r {} \+")


@task
def install(ctx: Context) -> None:
"""Install the project."""
ctx.run("poetry install")
ctx.run("uv sync --all-groups")


@task
def hooks(ctx: Context) -> None:
"""Setup the project hooks."""
ctx.run("poetry run pre-commit install")
ctx.run("uv run pre-commit install")


@task
def test(ctx: Context) -> None:
"""Run the project unit tests."""
ctx.run("poetry run pytest tests/")
ctx.run("uv run pytest tests/")
23 changes: 12 additions & 11 deletions tests/test_cookiecutter.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,15 @@

COMMANDS = [
"git init",
"invoke cleans.reset",
"invoke installs",
"invoke formats",
"invoke checks",
"invoke docs",
"invoke projects",
"invoke packages",
"invoke containers",
"uv run invoke cleans.reset",
"uv run invoke installs",
"uv run invoke formats",
"uv run invoke checks",
"uv run invoke docs",
"uv run invoke projects",
"uv run invoke packages",
"uv run invoke containers",
"uv run invoke mlflow.doctor",
]

# %% TESTS
Expand All @@ -26,13 +27,13 @@ def test_project_generation(cookies: Cookies) -> None:
"""Test the generation of the project."""
# given
context = {
"user": "test",
"user": "tester",
"name": "MLOps 123",
"license": "apache-2",
"version": "1.0.0",
"description": "DONE",
"description": "A test project.",
"python_version": "3.12",
"mlflow_version": "2.14.3",
"mlflow_version": "2.19.0",
}
repository = context['name'].lower().replace(' ', '-')
package = repository.replace('-', '_')
Expand Down
Loading
Loading