Review and Document Best Practices for Package Creation and Management

[Polichinel]

Issues that must be solved first:

• Create ADR for Naming Conventions #157
• Create ADR for Class Structure of Utilities #158

Review and Document Best Practices for Package Creation and Management

Description:
Conduct a thorough review of Python and MLOps best practices for creating and managing packages, ensuring alignment with the project’s structure and future scalability needs. The developer should consider the project’s unique requirements, such as modularity, maintainability, and ease of CI/CD integration, to create a guide or ADR (Architectural Decision Record) on package management within the project.

(Most likely this is comprehensive enough that a guide is preferable)

This document should provide clear guidelines for creating, structuring, and managing packages, ensuring consistency across all (future) project packages pertaining to the views_pipeline. Additionally, it must align with the md&d_python_style_guide as well as all decision derived from these issues:

• Create ADR for Naming Conventions #157
• Create ADR for Class Structure of Utilities #158

Key Areas to Cover

• Package Structure: Outline the recommended directory structure for each package, including:

  • Core Components: Main modules, submodules, and a clear organization for primary functions (e.g., data_processing, models, evaluation).
  • Supporting Directories: Guidelines for adding tests/, docs/, examples/, and any other supporting directories within each package.
  • Configuration Files: Best practices for including and managing essential files like setup.py, setup.cfg, requirements.txt, and any package-specific configuration.

• Naming Conventions: Ensure package names and module names are consistent, descriptive, and aligned with best practices, avoiding ambiguity and redundancy in accordance with Create ADR for Class Structure of Utilities #158

• Dependency Management: Recommendations for managing dependencies within each package to ensure compatibility across packages and reproducibility in different environments. Should address:

  • Using requirements.txt and requirements-dev.txt for dependencies.
  • Optional guidance on using lock files for version pinning (e.g., requirements.lock).
  • Handling internal dependencies between packages.

• Documentation Standards: Define standards for documenting each package:

  • README: Required for each package, with a clear description, usage examples, and installation instructions.
  • Additional Documentation: Guidelines for documenting complex functionality or APIs within a docs/ folder.
  • Docstrings: Consistent docstring format for all modules and functions.

• Testing and Validation: Establish standards for package-level testing:

  • Unit Testing: Required for each function and class, ensuring comprehensive test coverage.
  • Integration Testing: Necessary for modules interacting with other packages.
  • CI Integration: Guidelines for setting up continuous integration to run tests on each package, ensuring they’re maintained in a deployable state.

• CI/CD and Versioning: Outline versioning practices for each package (e.g., Semantic Versioning) and guidelines for automating CI/CD workflows for package deployment.

Objective:
To create a guide or ADR that documents best practices for creating and managing packages within the project. This document will ensure consistency, readability, and maintainability across packages, aligning with MLOps standards and the md&d_python_style_guide.

Tasks:

• Review best practices for Python package creation and management, with a focus on MLOps requirements.
• Draft the package creation guide or ADR, including guidelines on structure, dependencies, documentation, testing, and CI/CD.
• Review the draft with at least two team members to gather input and align with project needs.
• Finalize and publish the guide or ADR, linking it in relevant project documentation for future reference.

Labels: documentation, ADR, enhancement, research