Skip to content

Commit

Permalink
Merge pull request #457 from lsst/tickets/DM-47673-main
Browse files Browse the repository at this point in the history
DM-47673: v28 release notes (cherry-picked)
  • Loading branch information
timj authored Nov 22, 2024
2 parents 79fcafa + 969d39f commit 269e412
Show file tree
Hide file tree
Showing 21 changed files with 86 additions and 58 deletions.
8 changes: 7 additions & 1 deletion .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ jobs:
- name: Run tests
shell: bash -l {0}
run: |
pytest -r a -v -n 3 --cov=lsst.pipe.base --cov=tests --cov-report=xml --cov-report=term --cov-branch
pytest -r a -v -n 3 --cov=lsst.pipe.base --cov=tests --cov-report=xml --cov-report=term --cov-branch \
--junitxml=junit.xml -o junit_family=legacy
butler register-instrument -h
butler transfer-from-graph -h
butler retrieve-artifacts-for-quanta -h
Expand All @@ -66,6 +67,11 @@ jobs:
with:
files: ./coverage.xml
token: ${{ secrets.CODECOV_TOKEN }}
- name: Upload test results to Codecov
if: ${{ !cancelled() }}
uses: codecov/test-results-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}

pypi:
runs-on: ubuntu-latest
Expand Down
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
rev: v5.0.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- id: check-toml
- repo: https://github.com/psf/black
rev: 24.4.2
rev: 24.10.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
Expand All @@ -22,10 +22,10 @@ repos:
name: isort (python)
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.5.1
rev: v0.7.4
hooks:
- id: ruff
- repo: https://github.com/numpy/numpydoc
rev: "v1.7.0"
rev: "v1.8.0"
hooks:
- id: numpydoc-validation
3 changes: 0 additions & 3 deletions doc/changes/DM-38041.feature.md

This file was deleted.

9 changes: 0 additions & 9 deletions doc/changes/DM-40443.removal.rst

This file was deleted.

7 changes: 0 additions & 7 deletions doc/changes/DM-41605.feature.md

This file was deleted.

11 changes: 0 additions & 11 deletions doc/changes/DM-41711.feature.md

This file was deleted.

2 changes: 0 additions & 2 deletions doc/changes/DM-43020.misc.rst

This file was deleted.

2 changes: 0 additions & 2 deletions doc/changes/DM-43960.misc.rst

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-44091.bugfix.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-44368.feature.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-44583.feature.md

This file was deleted.

6 changes: 0 additions & 6 deletions doc/changes/DM-44647.feature.md

This file was deleted.

3 changes: 0 additions & 3 deletions doc/changes/DM-45457.feature.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-45536.bugfix.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-45701.api.rst

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-45722.bugfix.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-45722.misc.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-46064.feature.md

This file was deleted.

1 change: 0 additions & 1 deletion doc/changes/DM-46351.misc.md

This file was deleted.

74 changes: 74 additions & 0 deletions doc/lsst.pipe.base/CHANGES.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,77 @@
lsst-pipe-base v28.0.0 (2024-11-21)
===================================

New Features
------------

- Added support for initializing processing output runs with just a pipeline graph, not a quantum graph.

This also moves much of the logic for initializing output runs from ``lsst.ctrl.mpexec.PreExecInit`` to ``PipelineGraph`` and ``QuantumGraph`` methods. (`DM-38041 <https://rubinobs.atlassian.net/browse/DM-38041>`_)
- Added functionality to aggregate multiple ``QuantumProvenanceGraph.Summary`` objects into one ``Summary`` for a holistic report.

While the ``QuantumProvenanceGraph`` was designed to resolve processing over dataquery-identified groups, ``QuantumProvenanceGraph.aggregate`` is designed to combine multiple group-level reports into one which totals the successes,issues, and failures over the same section of pipeline. (`DM-41605 <https://rubinobs.atlassian.net/browse/DM-41605>`_)
- Created a ``QuantumProvenanceGraph`` class, which details the status of every quantum and dataset over multiple attempts at executing graphs, noting when quanta have been recovered.

Steps through all the quantum graphs associated with certain tasks or
processing steps.
For each graph/attempt, the status of each quantum and dataset is recorded in ``QuantumProvenanceGraph.add_new_graph`` and outcomes of quanta over multiple runs are resolved in ``QuantumProvenanceGraph.resolve_duplicates``.
At the end of this process, we can combine all attempts into a summary.
This serves to answer the question "What happened to this data ID?" in a holistic sense. (`DM-41711 <https://rubinobs.atlassian.net/browse/DM-41711>`_)
- Included the number of expected instances in ``pipetask report`` task-level summary for the `QuantumGraphExecutionReport`. (`DM-44368 <https://rubinobs.atlassian.net/browse/DM-44368>`_)
- Added mocking support for tasks that write regular datasets with config, log, or metadata storage classes. (`DM-44583 <https://rubinobs.atlassian.net/browse/DM-44583>`_)
- Added new ``show_dot`` functionality.

Refactored the existing pipeline graph ``show`` function, implementing a new ``parse_display_args`` function to handle the parsing of the ``--dot`` argument.
The ``show_dot`` function is then implemented to display the pipeline graph as a dot file.
A notable user-visible change is that output dataset types with common dimensions and storage classes will now be grouped together in dot files.
This change was implemented in order to save space for otherwise very large dot files. (`DM-44647 <https://rubinobs.atlassian.net/browse/DM-44647>`_)
- Removed the prohibition on optional regular (i.e., non-prerequisite) input connections.

Optional inputs can be declared by passing ``minimum=0`` in the connection definition. (`DM-45457 <https://rubinobs.atlassian.net/browse/DM-45457>`_)
- Storage class conversions of component dataset types are now supported in pipelines. (`DM-46064 <https://rubinobs.atlassian.net/browse/DM-46064>`_)


API Changes
-----------

- Relocated the "dot tools" from ``lsst.ctrl.mpexec.dotTools`` to ``lsst.pipe.base.dot_tools`` unchanged. (`DM-45701 <https://rubinobs.atlassian.net/browse/DM-45701>`_)


Bug Fixes
---------

- Appended failed quanta to a list and then return for ``pipetask report``.
The previous version of the human-readable report only reported the first failed quantum by exiting the loop upon finding it. (`DM-44091 <https://rubinobs.atlassian.net/browse/DM-44091>`_)
- Fixed support for task metadata as inputs in the ``PipelineTask`` mocking system. (`DM-45536 <https://rubinobs.atlassian.net/browse/DM-45536>`_)
- Explanatory logs for "initial data ID query returned no rows" now appear as a single log message instead of one entry per line.
This improves display in log aggregators, but there is no change to console behavior. (`DM-45722 <https://rubinobs.atlassian.net/browse/DM-45722>`_)


Other Changes and Additions
---------------------------

- Added ``pipe.base.utils.RegionTimeInfo``, a container for serializing pairs of sky region and timespan.
It's intended for several specific applications when running the AP pipeline. (`DM-43020 <https://rubinobs.atlassian.net/browse/DM-43020>`_)
- Added an optional parameter to ``PipelineStepTester`` that lets configs be tweaked before testing.
This is needed for AP pipelines, whose APDB config cannot be defaulted, and is not intended for wide adoption. (`DM-43960 <https://rubinobs.atlassian.net/browse/DM-43960>`_)
- Explanatory logs for "initial data ID query returned no rows" are now reported at ``ERROR``, not ``CRITICAL``, level. (`DM-45722 <https://rubinobs.atlassian.net/browse/DM-45722>`_)
- Added a DEBUG-level log message into ``_pipeline_graph.py`` to signify which task is being run. (`DM-46351 <https://rubinobs.atlassian.net/browse/DM-46351>`_)


An API Removal or Deprecation
-----------------------------

- Removed deprecated code scheduled to be removed after v27:

* Removed ``lsst.pipe.base.graphBuilder``.
* Removed ``lsst.pipe.base.pipeTools``.
* Removed ``lsst.pipe.base.BaseConnection.makeDatasetType``
* Removed ``Pipeline.toExpandedPipeline`` (replaced by ``to_graph``).
* Removed ``PipelineDatasetTypes`` and ``TaskDatasetTypes``.
* Removed ``QuantumGraphBuilderError``.
* APIs no longer accept ``TaskDef``. (`DM-40443 <https://rubinobs.atlassian.net/browse/DM-40443>`_)


lsst-pipe-base 27.0.0 (2024-05-29)
==================================

Expand Down
2 changes: 1 addition & 1 deletion doc/lsst.pipe.base/creating-a-pipeline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,7 @@ These bits of information allow campaign management / batch production software
sharding_dimensions: visit, detector
- label: step2
sharding_dimensions: tract, patch, skymap
.. _pipeline_creating_imports:

Expand Down

0 comments on commit 269e412

Please sign in to comment.