From 089cb1164a19b9dab9b07edf14141992780e2239 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 22 Oct 2024 21:05:57 +1000 Subject: [PATCH] Add overview doc --- README.md | 2 + docs/conf.py | 3 +- docs/development/index.rst | 2 + docs/overview.rst | 195 ++++++++++++++++++++++++++++++++++++- 4 files changed, 199 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index eed1c5b..7f86e11 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,8 @@ Caveats and Limitations declared as stable in the documentation may be renamed or relocated without a deprecation period. API stabilisation (mostly splitting up the overly large `venvstacks.stacks` namespace) will be the trigger for the 1.0 milestone release. +* while the `venvstacks` CLI is broadly stable, there are still some specific areas + where changes may occur (such as in the handling of relative paths). * dynamic library dependencies across layers currently only work on Windows. There is a [proposal](https://github.com/lmstudio-ai/venvstacks/issues/1) in place for resolving that limitation, but it has not yet been implemented. diff --git a/docs/conf.py b/docs/conf.py index 1a4d755..9bae84c 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -47,8 +47,7 @@ # -- Options for intersphinx ---------------------------------------------------------- intersphinx_mapping = { - "python": ("https://docs.python.org/3", None), - "pypug": ("https://packaging.python.org", None), + "py": ("https://docs.python.org/3", None), } diff --git a/docs/development/index.rst b/docs/development/index.rst index df93041..b71ae0c 100644 --- a/docs/development/index.rst +++ b/docs/development/index.rst @@ -1,3 +1,5 @@ +.. _dev-guide: + Development =========== diff --git a/docs/overview.rst b/docs/overview.rst index 8fb5b59..7440a1c 100644 --- a/docs/overview.rst +++ b/docs/overview.rst @@ -1,4 +1,197 @@ Layered Virtual Environment Stacks ================================== -TODO: Overview text... \ No newline at end of file +``venvstacks`` uses Python's ``sitecustomize.py`` environment setup feature +to chain together three layers of Python virtual environments: + +* "Runtime" layers: environments containing the desired version of a specific Python interpreter +* "Framework" layers: environments containing desired versions of key Python frameworks +* "Application" layers: environments containing components to be launched directly + +While the layers are archived and published separately, their dependency locking is integrated, +allowing the application layers to share dependencies installed in the framework layers, +and the framework layers to share dependencies installed in the runtime layers. + + +Installing +---------- + +``venvstacks`` is available from the :pypi:`Python Package Index `, +and can be installed with :pypi:`pip`: + +.. code-block:: console + + $ pip install --user venvstacks + +The command line help also provides additional usage information: + +.. code-block:: console + + $ .venv/bin/python -m venvstacks --help + + Usage: python -m venvstacks [OPTIONS] COMMAND [ARGS]... + + Lock, build, and publish Python virtual environment stacks. + + ╭─ Options ───────────────────────────────────────────────────────────────────────╮ + │ --help Show this message and exit. │ + ╰─────────────────────────────────────────────────────────────────────────────────╯ + ╭─ Commands ──────────────────────────────────────────────────────────────────────╮ + │ build Build (/lock/publish) Python virtual environment stacks. │ + │ local-export Export layer environments for Python virtual environment stacks. │ + │ lock Lock layer requirements for Python virtual environment stacks. │ + │ publish Publish layer archives for Python virtual environment stacks. │ + ╰─────────────────────────────────────────────────────────────────────────────────╯ + +.. note:: + + The ``venvstacks`` command is currently only available via the Python interpreter's + :py:option:`-m` switch, so Python command execution utilities like ``pipx run`` + and ``uvx`` don't work yet. + However, direct execution as ``venvstacks`` is expected to be + :issue:`supported in a future release <26>`. + + +Defining environment stacks +--------------------------- + +The environment layers to be published are defined in a ``venvstacks.toml`` stack specification, +with a separate array of tables for each kind of layer definition. + +For example, the following specification defines a pair of applications which use +`scikit-learn `__` as a shared framework layer with +`numpy `__ preinstalled in the runtime layer, +all running in a controlled Python 3.11 base runtime: + +.. code-block:: toml + + [[runtimes]] + name = "cpython@3.11" + fully_versioned_name = "cpython@3.11.10" + requirements = [ + "numpy", + ] + + [[frameworks]] + name = "sklearn" + runtime = "cpython@3.11" + requirements = [ + "scikit-learn", + ] + + [[applications]] + name = "classification-demo" + launch_module = "launch_modules/sklearn_classification.py" + frameworks = ["sklearn"] + requirements = [ + "scikit-learn", + ] + + [[applications]] + name = "clustering-demo" + launch_module = "launch_modules/sklearn_clustering.py" + frameworks = ["sklearn"] + requirements = [ + "scikit-learn", + ] + +Locking environment stacks +-------------------------- + +.. code-block:: console + + $ python -m venvstacks lock sklearn_demo/venvstacks.toml + +The ``lock`` subcommand takes the defined layer requirements from the specification, +and uses them to perform a complete combined resolution of all of the environment stacks +that ensures the different layers can be published separately, +but still work as expected when deployed to a target system. + +The locking mechanism is defined such that only changes to modules a given layer +uses from lower layers affect them, +rather than upper layers needing to be rebuilt for *every* change to a lower layer. + +Building environment stacks +--------------------------- + +.. code-block:: console + + $ python -m venvstacks build sklearn_demo/venvstacks.toml + +The ``build`` subcommand performs the step of converting the layer specifications +and their locked requirements into a working Python environment +(either a base runtime environment, +or a layered virtual environment based on one of the defined runtime environments). +If the environments have not already been explicitly locked, +the build step will lock them as necessary. + +This command is also a "build pipeline" command that allows locking, building, +and publishing to be performed in a single step (see the command line help for details). + +Publishing environment layer archives +------------------------------------- + +.. code-block:: console + + $ python -m venvstacks publish --tag-outputs --output-dir demo_artifacts sklearn_demo/venvstacks.toml + +Once the environments have been successfully built, +the ``publish`` command allows each layer to be converted to a separate +`reproducible `__` binary archive suitable +for transferring to another system, unpacking, and using the unpacked environments +to run the included applications (needing only a small post-installation step using +a Python script embedded in the built layer archives to correctly relink the deployed +environments with each other in their deployed location on the target system). + +Metadata regarding the layer definitions and the published artifacts is published +alongside the published archives (to ``demo_artifacts/__venvstacks__/`` in the given example). +This metadata captures both input details (such as the hashes of the locked requirements +and the included launch modules) and output details +(such as the exact size and exact hash of the built layer archive). + +Locally exporting environment stacks +------------------------------------ + +.. code-block:: console + + $ python -m venvstacks local-export --output-dir demo_export sklearn_demo/venvstacks.toml + +Given that even considering the use of ``venvstacks`` implies that some layer archives may be of +significant size (a fully built `pytorch` archive weighs in at multiple gigabytes, for example), +packing and unpacking the layer archives can take a substantial amount of time. + +To avoid that overhead when iterating on layer definitions and launch module details, +the ``local-export`` subcommand allows the built environments to be copied to a different +location on the same system, with most of the same filtering steps applied as would be +applied when performing the archive pack-and-unpack steps (the omissions are details +related to reproducible builds, like clamping the maximum file modification times to known values). + +Locally exporting environments produces much of the same metadata as publishing layer archives, +but the details related specifically to the published archive (such as its size and expected +contents hash) are necessarily omitted. + +Contributing to ``venvstacks`` development +------------------------------------------ + +``venvstacks`` is MIT Licensed and `developed on GitHub `__. + +If you have a suitable use case, +the easiest way to contribute to ``venvstacks`` development is just to try it out, +and let us know how that goes. What did you like, what did you dislike, what just plain broke? + +If anything does break, +then please `open an issue `__ +(if the problem hasn't already been reported). +If you're not sure if some behaviour is a bug or not, +or would just like to provide general feedback rather than file specific issues or suggestions, +the following Discord channels are the best way to get directly in touch with the developers: + +* Discuss ``venvstacks`` in general in the ``#venvstacks`` channel on the + `PyPA Discord Server `__. +* Discuss the use of ``venvstacks`` in LM Studio in the ``#dev-chat`` channel on the + `LM Studio Discord Server `__. + +The `"Packaging" category `__ on +`discuss.python.org `__ is also a reasonable place to provide feedback. + +For additional information, consult the :ref:`developer documentation `