Add basic CLI around get_benchmark()

[ColoredCarrot]

Mainly to be used automatically by tooling.
Options are simple CLI options for now, but should probably become more dynamic in the future (does argparse support dynamic/unregistered options?).

[codecov]

Codecov Report

Attention: Patch coverage is 94.44444% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 92.0%. Comparing base (60e1fa7) to head (2bc92c8).

Files	Patch %	Lines
src/mqt/bench/cli.py	94.4%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #321   +/-   ##
=====================================
  Coverage   91.9%   92.0%           
=====================================
  Files         46      47    +1     
  Lines       2406    2441   +35     
=====================================
+ Hits        2213    2246   +33     
- Misses       193     195    +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[burgholzer]

Hey 👋🏼
Many thanks for this first contribution! As discussed in other channels, this seems like a useful feature and the current PR is a solid basis for that.
I added some inline suggestions on some small improvements and some todos to make this feature complete. Those shouldn't really take long to incorporate.

(If you are not aware of that: if you go to the "Files changed" tab on GitHub, you can batch up suggestions into a single commit and apply them in one go directly from the web. This also resolves the related comments)

As already hinted at by the CLI, this PR is missing one more thing in addition to these small comments: test (coverage). There is a convenient way to test CLIs with unittests:

• you add the pytest-console-scripts dependency as illustrated here: https://github.com/cda-tum/mqt-core/blob/4d6f788df95513b9c14db938bae69f1683e26f3d/pyproject.toml#L39
• you write tests similar to https://github.com/cda-tum/mqt-core/blob/main/test/python/test_cli.py

Those tests need not be massive, but they should cover the most common scenarios in which a user would run the CI.

Once those changes are implemented, this should be good to go and a really nice addition to the library. Many thanks 🙏🏼

[ColoredCarrot]

Re. tests: dumps(get_benchmark(...)) seems to yield non-deterministic results. I'm lacking the domain knowledge here; is there some easy way to verify that the printed QASM string corresponds to specific options?
If not, I'll just do some basic sanity checks (like these inputs should work and the result should contain "OPENQASM 2.0;" + these inputs shouldn't work).

[burgholzer]

Re. tests: dumps(get_benchmark(...)) seems to yield non-deterministic results. I'm lacking the domain knowledge here; is there some easy way to verify that the printed QASM string corresponds to specific options?

If not, I'll just do some basic sanity checks (like these inputs should work and the result should contain "OPENQASM 2.0;" + these inputs shouldn't work).

Yeah. Some of the benchmarks involve steps that aren't very easy to seed for reproducibility. Thus, sometimes a check like comparing the result of the CLI with the result of the same get_benchmark call might fail.
However there should be some
Benchmarks that are fairly deterministic. Just at the top of my head:

• the "ghz" benchmark should work deterministically under almost all settings
• "shor" as a non-scalable example should be deterministic as well
• given that the benchmarks with ancillary settings are handled similar to regular instances, there should be no need to test them here.

As stated in the comment at the top, the tests here really don't need to be too exhaustive. If the above helps to create a working set of tests, that should be fine 😌

[burgholzer]

Many thanks for the additional changes and the quick workaround.
Everything looks good to me now!
Would really appreciate another PR resolving the special casing as discussed in the comment above!