test_experiment_data_integration hangs

[mriedem]

Information

• Qiskit IBM Provider version:

$ git log -1 --oneline
ad10492 (HEAD -> main, origin/main, origin/HEAD) `IBMProvider.run_circuits` now takes a backend_name instead of Backend object (#173)

• Python version:

$ python -V
Python 3.8.10

• Operating system:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

What is the current behavior?

I can run the tests and they pass but then hang on what I'm assuming is cleanup/teardown. We have a CI system running these and Travis times out after 10 minutes with no output (been going on for at least a week now):

$ LOG_LEVEL=INFO python -m unittest -v test/ibm/experiment/test_experiment_data_integration.py
test_add_analysis_results (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test adding an analysis result. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/a1dcfc50-ee5e-4823-8698-1574b5b20499
You can view the experiment online at https://quantum-computing.ibm.com/experiments/a1dcfc50-ee5e-4823-8698-1574b5b20499
ok
test_add_data_job (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test add job to experiment data. ... ok
test_add_figures (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test adding a figure to the experiment data. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/13190491-a7a5-441a-b642-9f3b2d688a3a
ok
test_add_figures_file (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test adding a figure file. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/4ba6a432-b255-4a94-b1c6-0ee88be4ce39
ok
test_add_figures_plot (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test adding a matplotlib figure. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/5a949843-abe5-4a4a-9df9-5b44d180e4cb
ok
test_auto_save_analysis_result (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test auto saving analysis result. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/14bbca62-3ded-48fb-944e-5f188064e286
You can view the experiment online at https://quantum-computing.ibm.com/experiments/14bbca62-3ded-48fb-944e-5f188064e286
ok
test_auto_save_analysis_result_update (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test auto saving analysis result updates. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/87f3d623-dbce-4379-8dbf-61e3c067c352
You can view the experiment online at https://quantum-computing.ibm.com/experiments/87f3d623-dbce-4379-8dbf-61e3c067c352
ok
test_auto_save_experiment (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test auto save. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/95e076c5-5817-4ede-b9a8-6daf5f0621d4
You can view the experiment online at https://quantum-computing.ibm.com/experiments/95e076c5-5817-4ede-b9a8-6daf5f0621d4
ok
test_auto_save_figure (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test auto saving figure. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/8c33b13e-5cea-4165-bcae-e13a08cf281d
You can view the experiment online at https://quantum-computing.ibm.com/experiments/8c33b13e-5cea-4165-bcae-e13a08cf281d
ok
test_block_for_results (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test blocking for jobs ... ok
test_delete_analysis_result (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test deleting an analysis result. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/97fb2d9b-730e-4581-9ca4-50258b2e75dc
You can view the experiment online at https://quantum-computing.ibm.com/experiments/97fb2d9b-730e-4581-9ca4-50258b2e75dc
You can view the experiment online at https://quantum-computing.ibm.com/experiments/97fb2d9b-730e-4581-9ca4-50258b2e75dc
ibm_experiment_service.delete_analysis_result:WARNING:2021-10-28 11:45:36,137: Analysis result 37c90b4e-0143-49ea-b71c-3f968d9547f1 not found.
ok
test_delete_figure (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test deleting a figure. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/28688265-558c-45a6-9be1-1d319005c71e
You can view the experiment online at https://quantum-computing.ibm.com/experiments/28688265-558c-45a6-9be1-1d319005c71e
ok
test_new_experiment_data (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test creating a new experiment data. ... Not all post-processing has finished. Consider calling save() again after all post-processing is done to save any newly generated data.
You can view the experiment online at https://quantum-computing.ibm.com/experiments/2797bc13-c03e-48f7-b17f-97f037a218fb
Job 617ac5a3060289c5090ed93b failed:
Traceback (most recent call last):
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/db_experiment_data.py", line 344, in _add_jobs_data
    job_result, timeout = combined_timeout(job.result, timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/utils.py", line 123, in combined_timeout
    ret = func(timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/qiskit_ibm/job/ibm_circuit_job.py", line 251, in result
    raise IBMJobInvalidStateError('Unable to retrieve result for job {}. '
qiskit_ibm.job.exceptions.IBMJobInvalidStateError: 'Unable to retrieve result for job 617ac5a3060289c5090ed93b. Job was cancelled.'

Job 617ac5a406028935200ed93c failed:
Traceback (most recent call last):
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/db_experiment_data.py", line 344, in _add_jobs_data
    job_result, timeout = combined_timeout(job.result, timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/utils.py", line 123, in combined_timeout
    ret = func(timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/qiskit_ibm/job/ibm_circuit_job.py", line 251, in result
    raise IBMJobInvalidStateError('Unable to retrieve result for job {}. '
qiskit_ibm.job.exceptions.IBMJobInvalidStateError: 'Unable to retrieve result for job 617ac5a406028935200ed93c. Job was cancelled.'

ok
test_save_all (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test saving all. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/6d1a7104-879b-4588-8820-c4dafbe9ba5b
You can view the experiment online at https://quantum-computing.ibm.com/experiments/6d1a7104-879b-4588-8820-c4dafbe9ba5b
You can view the experiment online at https://quantum-computing.ibm.com/experiments/6d1a7104-879b-4588-8820-c4dafbe9ba5b
ok
test_set_service_job (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test setting service with a job. ... Not all post-processing has finished. Consider calling save() again after all post-processing is done to save any newly generated data.
You can view the experiment online at https://quantum-computing.ibm.com/experiments/5d152d44-7308-4c13-9fc1-047ebd102123
ok
test_update_analysis_result (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test updating an analysis result. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/f2555fe5-8c26-49ef-93eb-4f092bfa999a
You can view the experiment online at https://quantum-computing.ibm.com/experiments/f2555fe5-8c26-49ef-93eb-4f092bfa999a
ok
test_update_experiment_data (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test updating an experiment. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/1b8a6dbe-6239-430f-90be-20f4d67183b8
Not all post-processing has finished. Consider calling save() again after all post-processing is done to save any newly generated data.
You can view the experiment online at https://quantum-computing.ibm.com/experiments/1b8a6dbe-6239-430f-90be-20f4d67183b8
Job 617ac5b2ee2a905c0ee1aa43 failed:
Traceback (most recent call last):
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/db_experiment_data.py", line 344, in _add_jobs_data
    job_result, timeout = combined_timeout(job.result, timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/utils.py", line 123, in combined_timeout
    ret = func(timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/qiskit_ibm/job/ibm_circuit_job.py", line 251, in result
    raise IBMJobInvalidStateError('Unable to retrieve result for job {}. '
qiskit_ibm.job.exceptions.IBMJobInvalidStateError: 'Unable to retrieve result for job 617ac5b2ee2a905c0ee1aa43. Job was cancelled.'

Job 617ac5b3060289aebc0ed93d failed:
Traceback (most recent call last):
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/db_experiment_data.py", line 344, in _add_jobs_data
    job_result, timeout = combined_timeout(job.result, timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/.venv/lib/python3.8/site-packages/qiskit_experiments/database_service/utils.py", line 123, in combined_timeout
    ret = func(timeout)
  File "/home/osboxes/ibmq/qiskit-ibm/qiskit_ibm/job/ibm_circuit_job.py", line 251, in result
    raise IBMJobInvalidStateError('Unable to retrieve result for job {}. '
qiskit_ibm.job.exceptions.IBMJobInvalidStateError: 'Unable to retrieve result for job 617ac5b3060289aebc0ed93d. Job was cancelled.'

ok
test_update_figure (test.ibm.experiment.test_experiment_data_integration.TestExperimentDataIntegration)
Test updating a figure. ... You can view the experiment online at https://quantum-computing.ibm.com/experiments/b4547a8f-4576-42aa-8759-fb3101484179
ok

----------------------------------------------------------------------
Ran 18 tests in 93.050s

OK
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 40, in _python_exit
    t.join()
  File "/usr/lib/python3.8/threading.py", line 1011, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

I'm not sure if it's relevant but experiment 5d152d44-7308-4c13-9fc1-047ebd102123 was not cleaned up. It looks like that was created by test_set_service_job and might just be issue Qiskit/qiskit-ibm-provider#67.

Steps to reproduce the problem

Just run the tests.

What is the expected behavior?

Tests should pass and complete without hanging.

Suggested solutions

🤷

[mriedem]

I'm not sure if it's relevant but experiment 5d152d44-7308-4c13-9fc1-047ebd102123 was not cleaned up.

FWIW that experiment has job 617ac5adf08425032bba21fd which no longer exists.

[kt474]

Transferring this issue since the experiment tests no longer run in the provider repo