diff --git a/NOTES_FOR_MAINTAINERS.md b/NOTES_FOR_MAINTAINERS.md index 03b4ede4e..c377f53c9 100644 --- a/NOTES_FOR_MAINTAINERS.md +++ b/NOTES_FOR_MAINTAINERS.md @@ -27,11 +27,62 @@ The script output is designed to be deterministic; if the vega-lite version is not changed, then running the script should overwrite the schema wrappers with identical copies. -## Updating the Vega-Lite version +## Updating Vega versions +All versions are maintained in [pyproject.toml](pyproject.toml). -The vega & vega-lite versions for the Python code can be updated by manually -changing the ``SCHEMA_VERSION`` definition within -``tools/generate_schema_wrapper.py``, and then re-running the script. +### Python Packages + +Projects which publish a package to PyPI are listed with a version bound in one of the following tables: + +- [`project.dependencies`](https://packaging.python.org/en/latest/specifications/pyproject-toml/#dependencies-optional-dependencies): Published dependencies. +- [`project.optional-dependencies`](https://packaging.python.org/en/latest/specifications/pyproject-toml/#dependencies-optional-dependencies): Published optional dependencies, or "extras". +- [`dependency-groups`](https://peps.python.org/pep-0735/): Local dependencies for development. + +> [!NOTE] +> All are currently declared in sub-tables of `project.optional-dependencies` + +#### `vl-convert` + +We need to ensure that [vl-convert](https://github.com/vega/vl-convert) includes support for the new Vega-Lite version. +Check the [vl-convert releases](https://github.com/vega/vl-convert/releases) to find the minimum +version of `vl-convert` that includes support for the desired version of Vega-Lite (and [open +an issue](https://github.com/vega/vl-convert/issues) if this version hasn't been +included in a released yet). + +#### `vegafusion` + + +> [!WARNING] +> *TODO: Find out how the constraint for `vegafusion` is decided* + +### Javascript/other + +Additional version constraints, including for [`Vega-Lite`](https://github.com/vega/vega-lite) itself are declared in `[tool.altair.vega]`. + +Whereas the [previous dependencies](#python-packages) are used at *install-time*; these are embedded into `altair` for use at *runtime* or when [generating the python code](#auto-generating-the-python-code): + +```toml +[tool.altair.vega] +vega-datasets = "..." # https://github.com/vega/vega-datasets +vega-embed = "..." # https://github.com/vega/vega-embed +vega-lite = "..." # https://github.com/vega/vega-lite +vegafusion = "..." # https://github.com/vega/vegafusion +vl-convert-python = "..." # https://github.com/vega/vl-convert +``` + +Some examples of where these propagate to: +- [altair/jupyter/js/index.js](altair/jupyter/js/index.js) +- [altair/utils/_importers.py](altair/utils/_importers.py) +- [tools/generate_schema_wrapper.py](tools/generate_schema_wrapper.py) +- [tools/versioning.py](tools/versioning.py) +- [altair/utils/schemapi.py](https://github.com/vega/altair/blob/0e23fd33e9a755bab0ef73a856340c48c14897e6/altair/utils/schemapi.py#L1619-L1640) + +> [!IMPORTANT] +> When updating **any** of these versions, be sure to [re-generate the python code](#auto-generating-the-python-code). + +#### Updating the Vega-Lite version + +The Vega-Lite version for the Python code propagates to `tools.generate_schema_wrapper.SCHEMA_VERSION`. This will update all of the automatically-generated files in the ``schema`` directory for each version, but please note that it will *not* update other @@ -50,30 +101,6 @@ of some docstrings. Major version updates (e.g. Vega-Lite 1.X->2.X) have required substantial rewrites, because the internal structure of the schema changed appreciably. -### Updating the Vega-Lite in JupyterChart -To update the Vega-Lite version used in JupyterChart, update the version in the -esm.sh URL in `altair/jupyter/js/index.js`. - -For example, to update to Vega-Lite 5.15.1, Vega 5 and Vega-Embed 6, the URL -should be: - -```javascript -import embed from "https://esm.sh/vega-embed@6?deps=vega@5&deps=vega-lite@5.15.1"; -``` - -### Updating vl-convert version bound - -When updating the version of Vega-Lite, it's important to ensure that -[vl-convert](https://github.com/vega/vl-convert) includes support for the new Vega-Lite version. -Check the [vl-convert releases](https://github.com/vega/vl-convert/releases) to find the minimum -version of vl-convert that includes support for the desired version of Vega-Lite (and [open -an issue](https://github.com/vega/vl-convert/issues) if this version hasn't been -included in a released yet.). Update the vl-convert version check in `altair/utils/_importers.py` -with the new minimum required version of vl-convert. - -Also, the version bound of the `vl-convert-python` package should be updated in the -`[project.optional-dependencies]/all` dependency group in `pyproject.toml`. - ## Releasing the Package To cut a new release of Altair, follow the steps outlined in diff --git a/RELEASING.md b/RELEASING.md index 6582af76a..264081ebd 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,16 +1,18 @@ -1. Make sure to have an environment set up with `hatch` installed. See `CONTRIBUTING.md`. +1. Check all [Vega project](https://github.com/orgs/vega/repositories?type=source) versions are up-to-date. See [NOTES_FOR_MAINTAINERS.md](NOTES_FOR_MAINTAINERS.md) + +2. Make sure to have an environment set up with `hatch` installed. See [CONTRIBUTING.md](CONTRIBUTING.md). Remove any existing environments managed by `hatch` so that it will create new ones with the latest dependencies when executing the commands further below: hatch env prune -2. Make certain your branch is in sync with head, and that you have no uncommitted modifications. If you work on a fork, replace `origin` with `upstream`: +3. Make certain your branch is in sync with head, and that you have no uncommitted modifications. If you work on a fork, replace `origin` with `upstream`: git checkout main git pull origin main git status # Should show "nothing to commit, working tree clean" -3. Do a clean doc build: +4. Do a clean doc build: hatch run doc:clean-all hatch run doc:build-html @@ -19,62 +21,62 @@ Navigate to http://localhost:8000 and ensure it looks OK (particularly do a visual scan of the gallery thumbnails). -4. Create a new release branch: +5. Create a new release branch: git switch -c version_5.0.0 -5. Update version to, e.g. 5.0.0: +6. Update version to, e.g. 5.0.0: - in ``altair/__init__.py`` - in ``doc/conf.py`` -6. Commit changes and push: +7. Commit changes and push: git add . -u git commit -m "chore: Bump version to 5.0.0" git push -7. Merge release branch into main, make sure that all required checks pass +8. Merge release branch into main, make sure that all required checks pass -8. On main, build source & wheel distributions. If you work on a fork, replace `origin` with `upstream`: +9. On main, build source & wheel distributions. If you work on a fork, replace `origin` with `upstream`: git switch main git pull origin main hatch clean # clean old builds & distributions hatch build # create a source distribution and universal wheel -9. publish to PyPI (Requires correct PyPI owner permissions): +10. publish to PyPI (Requires correct PyPI owner permissions): hatch publish -10. build and publish docs (Requires write-access to altair-viz/altair-viz.github.io): +11. build and publish docs (Requires write-access to altair-viz/altair-viz.github.io): hatch run doc:publish-clean-build -11. On main, tag the release. If you work on a fork, replace `origin` with `upstream`: +12. On main, tag the release. If you work on a fork, replace `origin` with `upstream`: git tag -a v5.0.0 -m "Version 5.0.0 release" git push origin v5.0.0 -12. Create a new branch: +13. Create a new branch: git switch -c maint_5.1.0dev -13. Update version and add 'dev' suffix, e.g. 5.1.0dev: +14. Update version and add 'dev' suffix, e.g. 5.1.0dev: - in ``altair/__init__.py`` - in ``doc/conf.py`` -14. Commit changes and push: +15. Commit changes and push: git add . -u git commit -m "chore: Bump version to 5.1.0dev" git push -15. Merge maintenance branch into main +16. Merge maintenance branch into main -16. Double-check that a conda-forge pull request is generated from the updated +17. Double-check that a conda-forge pull request is generated from the updated pip package by the conda-forge bot (may take up to several hours): https://github.com/conda-forge/altair-feedstock/pulls -17. Publish a new release in https://github.com/vega/altair/releases/ +18. Publish a new release in https://github.com/vega/altair/releases/ diff --git a/altair/utils/__init__.py b/altair/utils/__init__.py index bfcb52eff..68697ce33 100644 --- a/altair/utils/__init__.py +++ b/altair/utils/__init__.py @@ -12,10 +12,18 @@ from .deprecation import AltairDeprecationWarning, deprecated, deprecated_warn from .html import spec_to_html from .plugin_registry import PluginRegistry -from .schemapi import Optional, SchemaBase, SchemaLike, Undefined, is_undefined +from .schemapi import ( + VERSIONS, + Optional, + SchemaBase, + SchemaLike, + Undefined, + is_undefined, +) __all__ = ( "SHORTHAND_KEYS", + "VERSIONS", "AltairDeprecationWarning", "Optional", "PluginRegistry", diff --git a/altair/utils/_importers.py b/altair/utils/_importers.py index efe48df8b..d2b601804 100644 --- a/altair/utils/_importers.py +++ b/altair/utils/_importers.py @@ -5,12 +5,14 @@ from packaging.version import Version +from altair.utils.schemapi import VERSIONS + if TYPE_CHECKING: from types import ModuleType def import_vegafusion() -> ModuleType: - min_version = "1.5.0" + min_version = VERSIONS["vegafusion"] try: import vegafusion as vf @@ -45,7 +47,7 @@ def import_vegafusion() -> ModuleType: def import_vl_convert() -> ModuleType: - min_version = "1.6.0" + min_version = VERSIONS["vl-convert-python"] try: version = importlib_version("vl-convert-python") if Version(version) < Version(min_version): diff --git a/altair/utils/schemapi.py b/altair/utils/schemapi.py index 0416c3405..c6420307a 100644 --- a/altair/utils/schemapi.py +++ b/altair/utils/schemapi.py @@ -1614,3 +1614,27 @@ def with_property_setters(cls: type[TSchemaBase]) -> type[TSchemaBase]: for prop, propschema in schema.get("properties", {}).items(): setattr(cls, prop, _PropertySetter(prop, propschema)) return cls + + +VERSIONS: Mapping[ + Literal[ + "vega-datasets", "vega-embed", "vega-lite", "vegafusion", "vl-convert-python" + ], + str, +] = { + "vega-datasets": "v2.11.0", + "vega-embed": "6", + "vega-lite": "v5.20.1", + "vegafusion": "1.5.0", + "vl-convert-python": "1.7.0", +} +""" +Version pins for non-``python`` `vega projects`_. + +Notes +----- +When cutting a new release, make sure to update ``[tool.altair.vega]`` in ``pyproject.toml``. + +.. _vega projects: + https://github.com/vega +""" diff --git a/pyproject.toml b/pyproject.toml index 7bb051b35..a6012cd1f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -84,6 +84,7 @@ dev = [ "types-setuptools", "geopandas", "polars>=0.20.3", + "tomli; python_version<\"3.11\"" ] doc = [ "sphinx", @@ -99,6 +100,14 @@ doc = [ "scipy", ] +[tool.altair.vega] +# Minimum/exact versions, for projects under the `vega` organization +vega-datasets = "v2.11.0" # https://github.com/vega/vega-datasets +vega-embed = "6" # https://github.com/vega/vega-embed +vega-lite = "v5.20.1" # https://github.com/vega/vega-lite +vegafusion = "1.5.0" # https://github.com/vega/vegafusion +vl-convert-python = "1.7.0" # https://github.com/vega/vl-convert + [tool.hatch.version] path = "altair/__init__.py" diff --git a/tools/__init__.py b/tools/__init__.py index 46fc97553..00ce52b6d 100644 --- a/tools/__init__.py +++ b/tools/__init__.py @@ -4,6 +4,7 @@ markup, schemapi, update_init_file, + versioning, ) __all__ = [ @@ -12,4 +13,5 @@ "markup", "schemapi", "update_init_file", + "versioning", ] diff --git a/tools/generate_schema_wrapper.py b/tools/generate_schema_wrapper.py index f68771e52..4964f27b6 100644 --- a/tools/generate_schema_wrapper.py +++ b/tools/generate_schema_wrapper.py @@ -41,6 +41,7 @@ spell_literal, ) from tools.vega_expr import write_expr_module +from tools.versioning import VERSIONS if TYPE_CHECKING: from collections.abc import Iterable, Iterator @@ -50,7 +51,7 @@ T = TypeVar("T", bound="str | Iterable[str]") -SCHEMA_VERSION: Final = "v5.20.1" +SCHEMA_VERSION: Final = VERSIONS["vega-lite"] HEADER_COMMENT = """\ @@ -632,9 +633,8 @@ def copy_schemapi_util() -> None: destination_fp.open("w", encoding="utf8") as dest, ): dest.write(HEADER_COMMENT) - dest.writelines(source.readlines()) - if sys.platform == "win32": - ruff.format(destination_fp) + dest.writelines(chain(source.readlines(), VERSIONS.iter_inline_literal())) + ruff.format(destination_fp) def recursive_dict_update(schema: dict, root: dict, def_dict: dict) -> None: @@ -1384,13 +1384,10 @@ def main() -> None: "--skip-download", action="store_true", help="skip downloading schema files" ) args = parser.parse_args() + VERSIONS.update_all() copy_schemapi_util() vegalite_main(args.skip_download) - write_expr_module( - vlc.get_vega_version(), - output=EXPR_FILE, - header=HEADER_COMMENT, - ) + write_expr_module(VERSIONS.vlc_vega, output=EXPR_FILE, header=HEADER_COMMENT) # The modules below are imported after the generation of the new schema files # as these modules import Altair. This allows them to use the new changes diff --git a/tools/versioning.py b/tools/versioning.py new file mode 100644 index 000000000..7df96466b --- /dev/null +++ b/tools/versioning.py @@ -0,0 +1,222 @@ +""" +Versioning utils, specfic to `vega projects`_. + +Includes non-`python` projects. + +.. _vega projects: + https://github.com/vega + +Examples +-------- +>>> from tools.versioning import VERSIONS # doctest: +SKIP +>>> VERSIONS["vega-lite"] # doctest: +SKIP +'v5.20.1' + +>>> VERSIONS # doctest: +SKIP +{'vega-datasets': 'v2.11.0', + 'vega-embed': '6', + 'vega-lite': 'v5.20.1', + 'vegafusion': '1.5.0', + 'vl-convert-python': '1.7.0'} +""" + +from __future__ import annotations + +import sys +from collections import deque +from pathlib import Path +from typing import TYPE_CHECKING, Any, ClassVar, Literal + +if sys.version_info >= (3, 11): + import tomllib +else: + # NOTE: See https://github.com/hukkin/tomli?tab=readme-ov-file#building-a-tomlitomllib-compatibility-layer + import tomli as tomllib # type: ignore +from packaging.version import parse as parse_version + +import vl_convert as vlc +from tools.schemapi.utils import spell_literal + +if TYPE_CHECKING: + from collections.abc import ( + ItemsView, + Iterable, + Iterator, + KeysView, + Mapping, + Sequence, + ) + + if sys.version_info >= (3, 11): + from typing import LiteralString + else: + from typing_extensions import LiteralString + if sys.version_info >= (3, 10): + from typing import TypeAlias + else: + from typing_extensions import TypeAlias + +__all__ = ["VERSIONS"] + +_REPO_ROOT: Path = Path(__file__).parent.parent +_JUPYTER_INDEX = "altair/jupyter/js/index.js" +_PYPROJECT: Literal["pyproject.toml"] = "pyproject.toml" + +VegaProject: TypeAlias = Literal[ + "vega-datasets", "vega-embed", "vega-lite", "vegafusion", "vl-convert-python" +] + +VERSIONS: _Versions +"""Singleton ``_Versions`` instance.""" + + +def _read_pyproject_toml(fp: Path | None = None, /) -> dict[str, Any]: + source = fp or Path(__file__).parent.parent / _PYPROJECT + return tomllib.loads(source.read_text("utf-8")) + + +def _keypath(mapping: Mapping[str, Any], path: Iterable[str], /) -> Any: + """Get a nested table from ``mapping`` by following ``path``.""" + mut = dict[str, Any](**mapping) + for key in path: + mut = mut[key] + return mut + + +class _Versions: + _TABLE_PATH: ClassVar[Sequence[LiteralString]] = "tool", "altair", "vega" + """ + The table header path split by ``"."``:: + + [tool.altair.vega] -> "tool", "altair", "vega" + """ + + _CONST_NAME: ClassVar[Literal["VERSIONS"]] = "VERSIONS" + """Variable name for the exported literal.""" + + _mapping: Mapping[VegaProject, str] + + def __init__(self) -> None: + self._mapping = _keypath(_read_pyproject_toml(), self._TABLE_PATH) + + def __getitem__(self, key: VegaProject) -> str: + return self._mapping[key] + + def __repr__(self) -> str: + return repr(self._mapping) + + def projects(self) -> KeysView[VegaProject]: + return self._mapping.keys() + + def items(self) -> ItemsView[VegaProject, str]: + return self._mapping.items() + + @property + def vlc_vega(self) -> str: + """ + Returns version of `Vega`_ bundled with `vl-convert`_. + + .. _Vega: + https://github.com/vega/vega + .. _vl-convert: + https://github.com/vega/vl-convert + """ + return vlc.get_vega_version() + + @property + def vlc_vega_embed(self) -> str: + """ + Returns version of `Vega-Embed`_ bundled with `vl-convert`_. + + .. _Vega-Embed: + https://github.com/vega/vega-embed + .. _vl-convert: + https://github.com/vega/vl-convert + """ + return vlc.get_vega_embed_version() + + @property + def vlc_vega_themes(self) -> str: + """ + Returns version of `Vega-Themes`_ bundled with `vl-convert`_. + + .. _Vega-Themes: + https://github.com/vega/vega-themes + .. _vl-convert: + https://github.com/vega/vl-convert. + """ + return vlc.get_vega_themes_version() + + @property + def vlc_vegalite(self) -> list[str]: + """ + Returns versions of `Vega-Lite`_ bundled with `vl-convert`_. + + .. _Vega-Lite: + https://github.com/vega/vega-lite + .. _vl-convert: + https://github.com/vega/vl-convert + """ + return vlc.get_vegalite_versions() + + @property + def _annotation(self) -> str: + return f"Mapping[{spell_literal(self.projects())}, str]" + + @property + def _header(self) -> str: + return f"[{'.'.join(self._TABLE_PATH)}]" + + def iter_inline_literal(self) -> Iterator[str]: + """ + Yields the ``[tool.altair.vega]`` table as an inline ``dict``. + + Includes a type annotation and docstring. + + Notes + ----- + - Write at the bottom of ``altair.utils.schemapi``. + - Used in ``altair.utils._importers``. + """ + yield f"{self._CONST_NAME}: {self._annotation} = {self!r}\n" + yield '"""\n' + yield ( + "Version pins for non-``python`` `vega projects`_.\n\n" + "Notes\n" + "-----\n" + f"When cutting a new release, make sure to update ``{self._header}`` in ``pyproject.toml``.\n\n" + ".. _vega projects:\n" + " https://github.com/vega\n" + ) + yield '"""\n' + + def update_all(self) -> None: + """Update all static version pins.""" + print("Updating Vega project pins") + self.update_vega_embed() + + def update_vega_embed(self) -> None: + """Updates the **Vega-Lite** version used in ``JupyterChart``.""" + fp = _REPO_ROOT / _JUPYTER_INDEX + embed = self["vega-embed"] + vega = parse_version(self.vlc_vega).major + vegalite = self["vega-lite"].lstrip("v") + stmt = f'import vegaEmbed from "https://esm.sh/vega-embed@{embed}?deps=vega@{vega}&deps=vega-lite@{vegalite}";\n' + + with fp.open("r", encoding="utf-8", newline="\n") as f: + lines = deque(f.readlines()) + lines.popleft() + print(f"Updating import in {fp.as_posix()!r}, to:\n {stmt!r}") + lines.appendleft(stmt) + with fp.open("w", encoding="utf-8", newline="\n") as f: + f.writelines(lines) + + +def __getattr__(name: str) -> _Versions: + if name == "VERSIONS": + global VERSIONS + VERSIONS = _Versions() + return VERSIONS + else: + msg = f"module {__name__!r} has no attribute {name!r}" + raise AttributeError(msg)