Skip to content

Commit

Permalink
Further post-rc1 docs fixes and improvements (#60)
Browse files Browse the repository at this point in the history
  • Loading branch information
ncoghlan authored Oct 30, 2024
1 parent 115c842 commit 76afccf
Show file tree
Hide file tree
Showing 6 changed files with 93 additions and 36 deletions.
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
Virtual Environment Stacks for Python
=====================================

Machine learning and AI libraries for Python are big. Really big. Nobody wants to download
and install multiple copies of [PyTorch](https://pypi.org/project/torch/) or
[CUDA](https://pypi.org/project/cuda-python/) if they can reasonably avoid it.

``venvstacks`` allows you to package Python applications and all their dependencies into a
portable, deterministic format, *without* needing to include copies of these large Python
frameworks in every application archive.

The `venvstacks` project uses Python's `sitecustomize.py` environment setup feature to
chain together three layers of Python virtual environments:

Expand Down
35 changes: 35 additions & 0 deletions changelog.d/20241031_004930_ncoghlan_further_rc1_docs_review.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
.. A new scriv changelog fragment.
..
.. Uncomment the header that is right (remove the leading dots).
..
.. Removed
.. -------
..
.. - A bullet item for the Removed category.
..
.. Added
.. -----
..
.. - A bullet item for the Added category.
..
Changed
-------

- Further documentation fixes and improvements

..
.. Deprecated
.. ----------
..
.. - A bullet item for the Deprecated category.
..
.. Fixed
.. -----
..
.. - A bullet item for the Fixed category.
..
.. Security
.. --------
..
.. - A bullet item for the Security category.
..
68 changes: 37 additions & 31 deletions docs/development/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,35 +31,13 @@ Development Environment
-----------------------

In order to work on venvstacks, you need to install
:pypi:`pdm`, :pypi:`tox`, and :pypi:`tox-pdm`.
:pypi:`pdm`, :pypi:`tox`, :pypi:`tox-pdm`, and :pypi:`scriv`
(everything else can be executed via ``tox`` environments).

Given these tools, the default development environment can be set up
and other commands executed as described below.


Changelog Entries
-----------------

The ``venvstacks`` changelog is managed with :pypi:`scriv`.

Entries are written in ``.rst`` format by default, so they
can use semantic references to the rest of the documentation.
However, ``.md`` fragments are entirely fine if internal
semantic links aren't needed.

All changes which may affect ``venvstacks`` users should be
given a user facing changelog entry with ``scriv create``.

Refer to the
`"per-user" settings <https://scriv.readthedocs.io/en/1.5.1/configuration.html#per-user-git-settings>`__
in the ``scriv`` documentation for details on how to customise the
local behaviour of ``scriv create``.

The project level ``scriv`` settings are stored in
``pyproject.toml`` (but the project largely relies on the default
settings)


Running from the source tree
----------------------------

Expand All @@ -70,11 +48,11 @@ to set up an editable install in the default venv:
$ pdm sync --dev
venvstacks can then be executed via the ``-m`` switch:
venvstacks can then be executed with ``pdm run``:

.. code-block:: console
$ .venv/bin/venvstacks --help
$ pdm run venvstacks --help
Usage: venvstacks [OPTIONS] COMMAND [ARGS]...
Expand All @@ -90,6 +68,7 @@ venvstacks can then be executed via the ``-m`` switch:
│ publish Publish layer archives for Python virtual environment stacks. │
╰─────────────────────────────────────────────────────────────────────────────────╯
Building Documentation
----------------------

Expand All @@ -104,6 +83,30 @@ To build it locally, run:
The built documentation can be found in the ``docs/_build`` folder.


Changelog Entries
-----------------

The ``venvstacks`` changelog is managed with :pypi:`scriv`.

All changes which may affect ``venvstacks`` users should be
given a user facing changelog entry with ``scriv create``.

Entries are written in ``.rst`` format by default, so they
can use semantic references to the rest of the documentation.
However, ``.md`` fragments are entirely fine if internal
semantic links aren't needed.

Refer to the
`"per-user" settings <https://scriv.readthedocs.io/en/1.5.1/configuration.html#per-user-git-settings>`__
in the ``scriv`` documentation for details on how to customise the
local behaviour of ``scriv create``.

The project level ``scriv`` settings are stored in
``pyproject.toml`` (but the project largely relies on the default
settings)


Automated Testing
=================

Expand Down Expand Up @@ -171,7 +174,7 @@ environment directly:
$ tox -e py3.11
There are also additional labels defined for running the oldest test environment,
There are additional labels defined for running the oldest test environment,
the latest test environment, and all test environments:

.. code-block:: console
Expand Down Expand Up @@ -223,7 +226,7 @@ should only affect the hashes and sizes of the application layer
archives that include those launch modules).

If the original PR is not correct, then it can be retriggered by
closing and reopening the PR once the relevants fixes have been
closing and reopening the PR once the relevant fixes have been
implemented.


Expand Down Expand Up @@ -260,13 +263,15 @@ Prior to release:

* Update the version in ``pyproject.toml`` to remove the pre-release suffix
* Run ``scriv collect`` to update ``CHANGELOG.rst``
* Create a PR for the collected change log updates
* Commit and push the updated version number and collected change log updates
* Check the updated docs after the PR has been merged

Release (requires ``pandoc``):
Release (requires ``pandoc`` and a GitHub access token with release permissions):

* Use `misc/tag-release.sh`_ to create an annotated tag for the current version
* Push the tag to the remote repo
* Run ``scriv github-release --dry-run`` to check what would be published
* Run ``scriv github-release`` to make the release tag
* Run ``scriv github-release`` to make the release from the annotated tag

After release:

Expand All @@ -278,3 +283,4 @@ After release:
.. _`rich CLI`: https://docs.pytest.org/en/stable/how-to/usage.html#specifying-which-tests-to-run
.. _`GitHub`: https://github.com/lmstudio/venvstacks
.. _`testing README file`: https://github.com/lmstudio-ai/venvstacks/blob/main/tests/README.md
.. _`misc/tag-release.sh`: https://github.com/lmstudio-ai/venvstacks/blob/main/misc/tag-release.sh
2 changes: 1 addition & 1 deletion docs/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Essential Terms and Concepts
requirements specified as part of a base runtime layer are installed
directly into the base runtime (there is no virtual environment defined).
* *framework layers*: these layers primarily contain large dependencies
(such as :pypi:`pytorch`) which should not be published multiple times,
(such as :pypi:`PyTorch <torch>`) which should not be published multiple times,
even when they are used by multiple applications. Applications are
constrained to use the versions of any packages installed in the
framework layers they depend on. Each framework layer depends on a
Expand Down
10 changes: 9 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@
Virtual Environment Stacks
==========================

Machine learning and AI libraries for Python are big. Really big. Nobody wants to download
and install multiple copies of :pypi:`PyTorch <torch>` or :pypi:`CUDA <cuda-python>` if
they can reasonably avoid it.

``venvstacks`` allows you to package Python applications and all their dependencies into a
portable, deterministic format, *without* needing to include copies of these large Python
frameworks in every application archive.

``venvstacks`` uses Python's ``sitecustomize.py`` environment setup feature
to chain together three layers of Python virtual environments:

Expand All @@ -10,7 +18,7 @@ to chain together three layers of Python virtual environments:
* "Application" layers: environments containing components to be launched directly

Application layer environments may include additional unpackaged Python launch modules or
packages for invocation with ``python``'s :external+py:option:`-m` switch.
packages for invocation with ``python``'s :py:option:`-m` switch.

While the layers are archived and published separately, their dependency locking is integrated,
allowing the application layers to share dependencies installed in the framework layers,
Expand Down
6 changes: 3 additions & 3 deletions docs/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ The environment layers to be published are defined in a ``venvstacks.toml`` stac
with a separate array of tables for each kind of layer definition.

For example, the following specification defines a pair of applications which use
`scikit-learn <https://scikit-learn.org/>`__` as a shared framework layer with
`scikit-learn <https://scikit-learn.org/>`__ as a shared framework layer with
`numpy <https://numpy.org/>`__ preinstalled in the runtime layer,
all running in a controlled Python 3.11 base runtime:

Expand Down Expand Up @@ -136,8 +136,8 @@ Locally exporting environment stacks
$ venvstacks local-export --output-dir demo_export sklearn_demo/venvstacks.toml
Given that even considering the use of ``venvstacks`` implies that some layer archives may be of
significant size (a fully built `pytorch` archive weighs in at multiple gigabytes, for example),
packing and unpacking the layer archives can take a substantial amount of time.
significant size (a fully built :pypi:`PyTorch <torch>` archive weighs in at multiple gigabytes,
for example), packing and unpacking the layer archives can take a substantial amount of time.

To avoid that overhead when iterating on layer definitions and launch module details,
the ``local-export`` subcommand allows the built environments to be copied to a different
Expand Down

0 comments on commit 76afccf

Please sign in to comment.