DM-47328: Experimental command to copy files based on a graph

[timj]

Requiers lsst/daf_butler#1111

cc/ @tgoldina I think this might do what you need. Please give it a try and let me know. I might change the name of the command etc.

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes

[codecov]

Codecov Report

Attention: Patch coverage is 85.48387% with 18 lines in your changes missing coverage. Please review.

Project coverage is 83.22%. Comparing base (d7cbcf4) to head (9465cbd).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
.../pipe/base/script/retrieve_artifacts_for_quanta.py	50.00%	7 Missing ⚠️
python/lsst/pipe/base/graph/graph.py	91.17%	2 Missing and 4 partials ⚠️
python/lsst/pipe/base/cli/cmd/commands.py	86.66%	2 Missing ⚠️
...ython/lsst/pipe/base/script/transfer_from_graph.py	33.33%	2 Missing ⚠️
python/lsst/pipe/base/script/zip_from_graph.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #454      +/-   ##
==========================================
+ Coverage   83.00%   83.22%   +0.21%     
==========================================
  Files         102      103       +1     
  Lines       12135    12227      +92     
  Branches     1535     1546      +11     
==========================================
+ Hits        10073    10176     +103     
+ Misses       1655     1641      -14     
- Partials      407      410       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[timj]

@andy-slac it looks like I should add QuantumGraph.get_inputs() and QuantumGraph.get_outputs() methods. Does that seem right to you? Not sure whether they even need parameters -- just return all of them, and return datastore records for inputs and update the storage classes for the outputs.

[andy-slac]

it looks like I should add QuantumGraph.get_inputs() and QuantumGraph.get_outputs() methods

That looks natural, indeed. Maybe generalize it a bit and add options to include init inputs/outputs and intermediates? Something like

def get_input_refs(self, *, include_init=True, include_intermediates=True) -> Iterable[DatasetRef]

?

[timj]

Looking at the loops that are involved with the same loop over quanta running for both inputs and outputs, I'm edging towards a single get_refs(self, *, include_inputs=True, include_initInputs=True, include_outputs=True, include_initOutputs=True) approach.

[tgoldina]

@timj Thank you very much!

Indeed, I was able to retrieve all inputs used in a quantum graph with

$ pip install git+https://github.com/lsst/daf_butler@tickets/DM-47328
$ pip install git+https://github.com/lsst/pipe_base/@tickets/DM-47328
$ butler retrieve-artifacts-for-quanta --preserve-path --include-inputs --no-include-outputs tst.qgraph DATA tst_outdir

[andy-slac]

Looks great, I do not have an opinion on command name for CLI, retrieve-artifacts-for-quanta looks reasonable.