Skip to content

Commit

Permalink
Merge pull request #34 from karlaspuldaro/postprocess-automation
Browse files Browse the repository at this point in the history 
Automate workflow using github actions
  • Loading branch information
@karlaspuldaro
karlaspuldaro authored Sep 29, 2023
2 parents 91cd0b4 + 0285b21 commit b39d23f
Show file tree
Hide file tree
Showing 13 changed files with 618 additions and 258 deletions.
50 changes: 50 additions & 0 deletions .github/workflows/run_experiments.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Run benchmark experiments and submit results to metriq.info

on:
schedule:
- cron: '0 0 1 * *' # Schedule to run on the 1st day of each month at 12:00 AM UTC

env:
FORCE_COLOR: true

jobs:
run_experiment_steps:
runs-on: ubuntu-latest

env:
METRIQ_TOKEN: ${{ secrets.METRIQ_TOKEN}}

steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.8

- name: Install tox
run: pip install tox

- name: Run experiment steps in tox
run: tox -ve py38

# TODO: Create Github Bot account
# - name: Configure Git with Bot Git Config
# run: |
# git config user.name "${{ secrets.BOT_USERNAME }}"
# git config user.email "${{ secrets.BOT_EMAIL }}"

# - name: Check for new CSV files and commit with Bot user
# run: |
# untracked_csv_files=$(find ./benchmarking/results/ -name '*.csv' -type f)
# if [ -n "$untracked_csv_files" ]; then
# git add $untracked_csv_files
# git commit -m "Add new result files"
# git push -u origin main
# fi
# env:
# BOT_USERNAME: ${{ secrets.BOT_USERNAME }}
# BOT_EMAIL: ${{ secrets.BOT_EMAIL }}
# BOT_PAT: ${{ secrets.BOT_PAT}}

26 changes: 13 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# Qiskit compilation benchmark for Quantum circuits
# Welcome to Qiskit Metriq

Execute quantum circuit compilation [benchmark task 25](https://metriq.info/Task/25) and submit performance results for each version of Qiskit to [metriq.info](https://metriq.info/), a community-driven platform for hosting quantum benchmarks.
This repository contains a collection of scripts designed to execute `Qiskit` compilation experiments on quantum circuits and upload benchmark results to [metriq.info](https://metriq.info/), a community-driven platform for hosting quantum benchmarks.

## Archictecture
At its core, `Qiskit Metriq` tracks essential metrics such as circuit depth and gate count for these experiments, automatically running jobs for every version of `Qiskit`, streamlining the process of evaluating the performance of quantum circuits compiled with `Qiskit`.

This project creates a tox environment for each qiskit-terra version, starting from v0.13.0 to [latest](https://github.com/Qiskit/qiskit/releases).
## Archictecture

The benchmark circuit compilation is batch processed, and the results are submitted to [metriq.info](https://metriq.info/).
This project creates a [tox](https://pypi.org/project/tox/) environment for each `Qiskit` version, starting from `qiskit-terra v0.13.0` to [latest](https://github.com/Qiskit/qiskit/releases). The `Qiskit` circuit compilation is batch processed, and the results are automatically submitted to [metriq.info](https://metriq.info/).

Benchmark tasks for quantum computer compilers:
- ex1_226.qasm benchmark circuit
- [Task 25](https://metriq.info/Task/25)
- [Task 26](https://metriq.info/Task/26) - Aspen architecture
- [Task 27](https://metriq.info/Task/27) - IBMQ Rochester architecture
- [ex1_226.qasm](https://github.com/CQCL/pytket/blob/2afee82a46cee18d618b7adf4b2b810a8a9da4e8/examples/benchmarking/ibmq/ex1_226.qasm) quantum circuit:
- [Metriq Task 25](https://metriq.info/Task/25) - Parent task
- [Metriq Task 26](https://metriq.info/Task/26) - Child task: specific to Aspen architecture
- [Metriq Task 27](https://metriq.info/Task/27) - Child task: specific to IBMQ Rochester architecture

## Results
## Qiskit compilation results
**Circuit depth distribution**

Compiled for the Rigetti 16Q Aspen architecture
Expand All @@ -36,13 +36,13 @@ Compiled for the IBMQ Rochester architecture
* Python 3.8+

## Run locally
### To run benchmark task using the current stable version of `qiskit-terra`:
### To run a benchmark experiment using the current stable version of `Qiskit`:
```bash
tox -e py38
tox -e qiskit-compilation
```
**Note:**
To run a specific version of `qiskit-terra`, you can manually update it in the `tox.ini` file.
Versions >=0.13,<=0.15 require numpy<1.20. You can run the tox environments `terra13`, `terra14` or `terra15` as:
```bash
tox -e py8-terra13
tox -e qiskit-compilation-terra13
```
67 changes: 0 additions & 67 deletions benchmarking/qiskit_versions.json
View file

This file was deleted.

101 changes: 101 additions & 0 deletions benchmarking/results/ex1_226-qiskit0.44.1-ibm_rochester.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
|Qasm file|Method|Date|Opt level|Platform|Seed|Circuit depth|Gate count
0|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|0|11|15
1|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|1|11|16
2|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|2|9|17
3|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|3|8|12
4|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|4|11|16
5|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|5|10|15
6|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|6|11|15
7|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|7|13|20
8|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|8|9|13
9|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|9|10|17
10|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|10|11|15
11|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|11|10|17
12|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|12|9|13
13|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|13|9|12
14|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|14|11|17
15|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|15|12|22
16|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|16|11|17
17|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|17|11|16
18|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|18|12|20
19|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|19|10|16
20|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|20|11|18
21|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|21|8|12
22|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|22|9|17
23|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|23|10|17
24|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|24|11|18
25|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|25|11|16
26|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|26|10|17
27|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|27|10|16
28|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|28|13|19
29|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|29|13|19
30|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|30|11|17
31|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|31|11|15
32|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|32|9|13
33|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|33|11|16
34|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|34|10|17
35|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|35|13|21
36|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|36|11|16
37|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|37|10|16
38|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|38|13|22
39|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|39|11|17
40|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|40|11|16
41|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|41|10|17
42|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|42|10|17
43|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|43|9|13
44|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|44|13|18
45|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|45|8|13
46|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|46|13|18
47|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|47|11|17
48|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|48|9|17
49|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|49|11|18
50|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|50|8|12
51|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|51|11|15
52|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|52|8|12
53|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|53|10|16
54|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|54|7|13
55|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|55|10|17
56|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|56|11|16
57|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|57|10|16
58|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|58|9|17
59|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|59|13|19
60|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|60|9|13
61|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|61|11|15
62|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|62|10|17
63|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|63|10|17
64|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|64|10|17
65|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|65|9|12
66|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|66|11|15
67|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|67|10|15
68|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|68|9|17
69|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|69|10|17
70|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|70|8|12
71|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|71|9|17
72|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|72|11|16
73|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|73|10|17
74|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|74|9|12
75|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|75|11|17
76|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|76|11|17
77|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|77|8|12
78|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|78|11|16
79|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|79|11|15
80|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|80|9|17
81|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|81|12|21
82|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|82|10|15
83|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|83|9|17
84|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|84|11|17
85|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|85|13|20
86|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|86|12|21
87|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|87|11|16
88|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|88|13|19
89|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|89|11|16
90|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|90|11|16
91|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|91|9|17
92|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|92|13|19
93|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|93|13|22
94|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|94|13|19
95|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|95|9|12
96|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|96|9|17
97|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|97|9|17
98|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|98|7|12
99|ex1_226.qasm|Qiskit 0.44.1 compilation|2023-08-17|3|ibm_rochester|99|11|15
Loading

0 comments on commit b39d23f

Please sign in to comment.