diff --git a/README.md b/README.md index 879f969..d08b271 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,14 @@ Virtual Environment Stacks for Python ===================================== +Machine learning and AI libraries for Python are big. Really big. Nobody wants to download +and install multiple copies of [PyTorch](https://pypi.org/project/torch/) or +[CUDA](https://pypi.org/project/cuda-python/) if they can reasonably avoid it. + +``venvstacks`` allows you to package Python applications and all their dependencies into a +portable, deterministic format, *without* needing to include copies of these large Python +frameworks in every application archive. + The `venvstacks` project uses Python's `sitecustomize.py` environment setup feature to chain together three layers of Python virtual environments: diff --git a/changelog.d/20241031_004930_ncoghlan_further_rc1_docs_review.rst b/changelog.d/20241031_004930_ncoghlan_further_rc1_docs_review.rst new file mode 100644 index 0000000..c42d34d --- /dev/null +++ b/changelog.d/20241031_004930_ncoghlan_further_rc1_docs_review.rst @@ -0,0 +1,35 @@ +.. A new scriv changelog fragment. +.. +.. Uncomment the header that is right (remove the leading dots). +.. +.. Removed +.. ------- +.. +.. - A bullet item for the Removed category. +.. +.. Added +.. ----- +.. +.. - A bullet item for the Added category. +.. +Changed +------- + +- Further documentation fixes and improvements + +.. +.. Deprecated +.. ---------- +.. +.. - A bullet item for the Deprecated category. +.. +.. Fixed +.. ----- +.. +.. - A bullet item for the Fixed category. +.. +.. Security +.. -------- +.. +.. - A bullet item for the Security category. +.. diff --git a/docs/development/index.rst b/docs/development/index.rst index 73422d5..e27e772 100644 --- a/docs/development/index.rst +++ b/docs/development/index.rst @@ -31,35 +31,13 @@ Development Environment ----------------------- In order to work on venvstacks, you need to install -:pypi:`pdm`, :pypi:`tox`, and :pypi:`tox-pdm`. +:pypi:`pdm`, :pypi:`tox`, :pypi:`tox-pdm`, and :pypi:`scriv` +(everything else can be executed via ``tox`` environments). Given these tools, the default development environment can be set up and other commands executed as described below. -Changelog Entries ------------------ - -The ``venvstacks`` changelog is managed with :pypi:`scriv`. - -Entries are written in ``.rst`` format by default, so they -can use semantic references to the rest of the documentation. -However, ``.md`` fragments are entirely fine if internal -semantic links aren't needed. - -All changes which may affect ``venvstacks`` users should be -given a user facing changelog entry with ``scriv create``. - -Refer to the -`"per-user" settings `__ -in the ``scriv`` documentation for details on how to customise the -local behaviour of ``scriv create``. - -The project level ``scriv`` settings are stored in -``pyproject.toml`` (but the project largely relies on the default -settings) - - Running from the source tree ---------------------------- @@ -70,11 +48,11 @@ to set up an editable install in the default venv: $ pdm sync --dev -venvstacks can then be executed via the ``-m`` switch: +venvstacks can then be executed with ``pdm run``: .. code-block:: console - $ .venv/bin/venvstacks --help + $ pdm run venvstacks --help Usage: venvstacks [OPTIONS] COMMAND [ARGS]... @@ -90,6 +68,7 @@ venvstacks can then be executed via the ``-m`` switch: │ publish Publish layer archives for Python virtual environment stacks. │ ╰─────────────────────────────────────────────────────────────────────────────────╯ + Building Documentation ---------------------- @@ -104,6 +83,30 @@ To build it locally, run: The built documentation can be found in the ``docs/_build`` folder. + +Changelog Entries +----------------- + +The ``venvstacks`` changelog is managed with :pypi:`scriv`. + +All changes which may affect ``venvstacks`` users should be +given a user facing changelog entry with ``scriv create``. + +Entries are written in ``.rst`` format by default, so they +can use semantic references to the rest of the documentation. +However, ``.md`` fragments are entirely fine if internal +semantic links aren't needed. + +Refer to the +`"per-user" settings `__ +in the ``scriv`` documentation for details on how to customise the +local behaviour of ``scriv create``. + +The project level ``scriv`` settings are stored in +``pyproject.toml`` (but the project largely relies on the default +settings) + + Automated Testing ================= @@ -171,7 +174,7 @@ environment directly: $ tox -e py3.11 -There are also additional labels defined for running the oldest test environment, +There are additional labels defined for running the oldest test environment, the latest test environment, and all test environments: .. code-block:: console @@ -223,7 +226,7 @@ should only affect the hashes and sizes of the application layer archives that include those launch modules). If the original PR is not correct, then it can be retriggered by -closing and reopening the PR once the relevants fixes have been +closing and reopening the PR once the relevant fixes have been implemented. @@ -260,13 +263,15 @@ Prior to release: * Update the version in ``pyproject.toml`` to remove the pre-release suffix * Run ``scriv collect`` to update ``CHANGELOG.rst`` -* Create a PR for the collected change log updates +* Commit and push the updated version number and collected change log updates * Check the updated docs after the PR has been merged -Release (requires ``pandoc``): +Release (requires ``pandoc`` and a GitHub access token with release permissions): +* Use `misc/tag-release.sh`_ to create an annotated tag for the current version +* Push the tag to the remote repo * Run ``scriv github-release --dry-run`` to check what would be published -* Run ``scriv github-release`` to make the release tag +* Run ``scriv github-release`` to make the release from the annotated tag After release: @@ -278,3 +283,4 @@ After release: .. _`rich CLI`: https://docs.pytest.org/en/stable/how-to/usage.html#specifying-which-tests-to-run .. _`GitHub`: https://github.com/lmstudio/venvstacks .. _`testing README file`: https://github.com/lmstudio-ai/venvstacks/blob/main/tests/README.md +.. _`misc/tag-release.sh`: https://github.com/lmstudio-ai/venvstacks/blob/main/misc/tag-release.sh diff --git a/docs/glossary.rst b/docs/glossary.rst index c31e2a4..0d9275a 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -51,7 +51,7 @@ Essential Terms and Concepts requirements specified as part of a base runtime layer are installed directly into the base runtime (there is no virtual environment defined). * *framework layers*: these layers primarily contain large dependencies - (such as :pypi:`pytorch`) which should not be published multiple times, + (such as :pypi:`PyTorch `) which should not be published multiple times, even when they are used by multiple applications. Applications are constrained to use the versions of any packages installed in the framework layers they depend on. Each framework layer depends on a diff --git a/docs/index.rst b/docs/index.rst index 7acdf1a..7de0992 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -2,6 +2,14 @@ Virtual Environment Stacks ========================== +Machine learning and AI libraries for Python are big. Really big. Nobody wants to download +and install multiple copies of :pypi:`PyTorch ` or :pypi:`CUDA ` if +they can reasonably avoid it. + +``venvstacks`` allows you to package Python applications and all their dependencies into a +portable, deterministic format, *without* needing to include copies of these large Python +frameworks in every application archive. + ``venvstacks`` uses Python's ``sitecustomize.py`` environment setup feature to chain together three layers of Python virtual environments: @@ -10,7 +18,7 @@ to chain together three layers of Python virtual environments: * "Application" layers: environments containing components to be launched directly Application layer environments may include additional unpackaged Python launch modules or -packages for invocation with ``python``'s :external+py:option:`-m` switch. +packages for invocation with ``python``'s :py:option:`-m` switch. While the layers are archived and published separately, their dependency locking is integrated, allowing the application layers to share dependencies installed in the framework layers, diff --git a/docs/overview.rst b/docs/overview.rst index c164dca..ef3097c 100644 --- a/docs/overview.rst +++ b/docs/overview.rst @@ -38,7 +38,7 @@ The environment layers to be published are defined in a ``venvstacks.toml`` stac with a separate array of tables for each kind of layer definition. For example, the following specification defines a pair of applications which use -`scikit-learn `__` as a shared framework layer with +`scikit-learn `__ as a shared framework layer with `numpy `__ preinstalled in the runtime layer, all running in a controlled Python 3.11 base runtime: @@ -136,8 +136,8 @@ Locally exporting environment stacks $ venvstacks local-export --output-dir demo_export sklearn_demo/venvstacks.toml Given that even considering the use of ``venvstacks`` implies that some layer archives may be of -significant size (a fully built `pytorch` archive weighs in at multiple gigabytes, for example), -packing and unpacking the layer archives can take a substantial amount of time. +significant size (a fully built :pypi:`PyTorch ` archive weighs in at multiple gigabytes, +for example), packing and unpacking the layer archives can take a substantial amount of time. To avoid that overhead when iterating on layer definitions and launch module details, the ``local-export`` subcommand allows the built environments to be copied to a different