Skip to content

Commit

Permalink
Post-rc1 docs and metadata updates
Browse files Browse the repository at this point in the history
  • Loading branch information
ncoghlan committed Oct 30, 2024
1 parent 4bcfee3 commit e0fa288
Show file tree
Hide file tree
Showing 7 changed files with 93 additions and 73 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Changelog
Until the Python API has stabilised, ``venvstacks`` is using
`ZeroVer <https://0ver.org/>`__ (starting from 0.1.0).

Refer to the :ref:`version-numbering` for additional details
Refer to :ref:`version-numbering` for additional details
on the way releases are versioned.


Expand Down
56 changes: 31 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Layered Virtual Environment Stacks for Python
=============================================
Virtual Environment Stacks for Python
=====================================

The `venvstacks` project uses Python's `sitecustomize.py` environment setup feature to
chain together three layers of Python virtual environments:
Expand All @@ -11,29 +11,26 @@ chain together three layers of Python virtual environments:
Application layer environments may include additional unpackaged Python launch modules or
packages for invocation with `python`'s `-m` switch.

This project does NOT support combining arbitrary virtual environments with each other.
Instead, it allows larger integrated applications to split up their Python dependencies into
distinct layers, without needing to download and install multiple copies of large
dependencies (such as the `pytorch` ML/AI framework). The environment stack specification
and build process helps ensure that shared dependencies are kept consistent across layers,
while unshared dependencies are free to vary across the application components that need them.
While the layers are archived and published separately, their dependency locking is integrated,
allowing the application layers to share dependencies installed in the framework layers,
and the framework layers to share dependencies installed in the runtime layers.

As an example, the main sample project used in the test suite defines the following layers:
Refer to the [Project Overview](https://lmstudio-ai.github.io/venvstacks/overview/) for an
example of specifying, locking, building, and publishing a set of environment stacks.

* `[email protected]`: CPython 3.11 base runtime
* `[email protected]`: CPython 3.12 base runtime
* `framework-scipy`: example framework layer (based on 3.11 runtime)
* `framework-sklearn` example framework layer (based on 3.12 runtime)
* `framework-http-client`: example framework layer (based on 3.11 runtime)
* `app-scipy-import`: example app layer with a single framework and a simple launch module
* `app-scipy-client`: example app layer with two frameworks and a multi-file launch package
* `app-sklearn-import`: example of defining a platform specific app layer

Refer to `tests\sample_project\venvstacks.toml` for the full definition of this example.
Installing
----------

To avoid relying on the Python ecosystem's still limited support for cross-platform
component installation, the stack build processes need to be executed on the target
platform (for example, by using an OS matrix in GitHub Actions).
`venvstacks` is available from the [Python Package Index](https://pypi.org/project/venvstacks/),
and can be installed with [pipx](https://pypi.org/project/pipx/):

$ pipx install venvstacks

Alternatively, it can be installed as a user level package (although this may
make future Python version upgrades more irritating):

$ pip install --user venvstacks


Interactions with other packaging tools
Expand Down Expand Up @@ -63,18 +60,27 @@ may be useful in generating the required input archives.
Caveats and Limitations
-----------------------

* the `venvstacks` Python API is *not yet stable*. Any interface not specifically
* This project does NOT support combining arbitrary virtual environments with each other.
Instead, it allows larger integrated applications to split up their Python dependencies into
distinct layers, without needing to download and install multiple copies of large
dependencies (such as the `pytorch` ML/AI framework). The environment stack specification
and build process helps ensure that shared dependencies are kept consistent across layers,
while unshared dependencies are free to vary across the application components that need them.
* The `venvstacks` Python API is *not yet stable*. Any interface not specifically
declared as stable in the documentation may be renamed or relocated without a
deprecation period. API stabilisation (mostly splitting up the overly large
`venvstacks.stacks` namespace) will be the trigger for the 1.0 milestone release.
* while the `venvstacks` CLI is broadly stable, there are still some specific areas
* While the `venvstacks` CLI is broadly stable, there are still some specific areas
where changes may occur (such as in the handling of relative paths).
* dynamic library dependencies across layers currently only work on Windows.
* Dynamic library dependencies across layers currently only work on Windows.
There is a [proposal](https://github.com/lmstudio-ai/venvstacks/issues38) in
place for resolving that limitation, but it has not yet been implemented.
* local exports to filesystems which do not support symlinks (such as `VFAT` and
* Local exports to filesystems which do not support symlinks (such as `VFAT` and
`FAT32`) are nominally supported (with symlinks being replaced by the files
they refer to), but this support is *not* currently tested.
* To avoid relying on the Python ecosystem's still limited support for cross-platform
component installation, the stack build processes need to be executed on the target
platform (for example, by using an OS matrix in GitHub Actions).


Project History
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Changed
-------

- Included project URLs in project metadata

- Added installation instructions to README.md

- Linked to main documentation from README.md

- Improved the content split between the project
overview page and the top level docs landing page
2 changes: 1 addition & 1 deletion docs/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,4 @@ Essential Terms and Concepts
An application layer with its supporting framework and base runtime layers.

stack specification
A ``venvstacks.toml`` file that definez one or more environment stacks.
A ``venvstacks.toml`` file that defines one or more environment stacks.
41 changes: 35 additions & 6 deletions docs/index.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,42 @@
.. venvstacks documentation master file, created by
sphinx-quickstart on Mon Oct 21 20:08:31 2024.
==========================
Virtual Environment Stacks
==========================

==========================================
Welcome to ``venvstacks``'s documentation!
==========================================
``venvstacks`` uses Python's ``sitecustomize.py`` environment setup feature
to chain together three layers of Python virtual environments:

* "Runtime" layers: environments containing the desired version of a specific Python interpreter
* "Framework" layers: environments containing desired versions of key Python frameworks
* "Application" layers: environments containing components to be launched directly

Application layer environments may include additional unpackaged Python launch modules or
packages for invocation with ``python``'s :external+py:option:`-m` switch.

While the layers are archived and published separately, their dependency locking is integrated,
allowing the application layers to share dependencies installed in the framework layers,
and the framework layers to share dependencies installed in the runtime layers.

Refer to the :ref:`overview` for an example of
specifying, locking, building, and publishing a set of environment stacks.

.. _installing:

``venvstacks`` is available from the :pypi:`Python Package Index <venvstacks>`,
and can be installed with :pypi:`pipx` (or similar tools):

.. code-block:: console
$ pipx install venvstacks
Alternatively, it can be installed as a user level package (although this may
make future Python version upgrades more irritating):

.. code-block:: console
$ pip install --user venvstacks
.. toctree::
:maxdepth: 2
:caption: Contents:

overview
glossary
Expand Down
46 changes: 7 additions & 39 deletions docs/overview.rst
Original file line number Diff line number Diff line change
@@ -1,37 +1,13 @@
----------------------------------
Layered Virtual Environment Stacks
----------------------------------
.. _overview:

``venvstacks`` uses Python's ``sitecustomize.py`` environment setup feature
to chain together three layers of Python virtual environments:
----------------
Project Overview
----------------

* "Runtime" layers: environments containing the desired version of a specific Python interpreter
* "Framework" layers: environments containing desired versions of key Python frameworks
* "Application" layers: environments containing components to be launched directly
Command line interface
======================

While the layers are archived and published separately, their dependency locking is integrated,
allowing the application layers to share dependencies installed in the framework layers,
and the framework layers to share dependencies installed in the runtime layers.


Installing
==========

``venvstacks`` is available from the :pypi:`Python Package Index <venvstacks>`,
and can be installed with :pypi:`pipx` (or similar tools):

.. code-block:: console
$ pipx install venvstacks
Alternatively, it can be installed as a user level package (although this may
make future Python version upgrades more irritating):

.. code-block:: console
$ pip install --user venvstacks
The command line help also provides additional usage information:
The command line interface is the recommended interface for working with ``venvstacks``:

.. code-block:: console
Expand All @@ -51,14 +27,6 @@ The command line help also provides additional usage information:
│ publish Publish layer archives for Python virtual environment stacks. │
╰─────────────────────────────────────────────────────────────────────────────────╯
.. note::

The ``venvstacks`` command is currently only available via the Python interpreter's
:py:option:`-m` switch, so Python command execution utilities like ``pipx run``
and ``uvx`` don't work yet.
However, direct execution as ``venvstacks`` is expected to be
:issue:`supported in a future release <26>`.

Working with environment stacks
===============================
Expand Down
8 changes: 7 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "venvstacks"
version = "0.1.0rc1"
version = "0.1.0rc1.post0"
description = "Use layered Python virtual environment stacks to share large dependencies"
authors = [
{name = "Alyssa Coghlan", email = "[email protected]"},
Expand Down Expand Up @@ -49,6 +49,12 @@ rich-cli = [
[project.scripts]
venvstacks = "venvstacks.cli:main"

[project.urls]
Homepage = "https://github.com/lmstudio-ai/venvstacks"
Documentation = "https://lmstudio-ai.github.io/venvstacks/"
Issues = "https://github.com/lmstudio-ai/venvstacks/issues"
Changelog = "https://lmstudio-ai.github.io/venvstacks/changelog/"

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"
Expand Down

0 comments on commit e0fa288

Please sign in to comment.