Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test failed in CI: test_update_uninitialized #7014

Open
smklein opened this issue Nov 7, 2024 · 2 comments
Open

test failed in CI: test_update_uninitialized #7014

smklein opened this issue Nov 7, 2024 · 2 comments
Labels
Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken.

Comments

@smklein
Copy link
Collaborator

smklein commented Nov 7, 2024

This test failed on a CI run on #7008:

https://github.com/oxidecomputer/omicron/pull/7008/checks?check_run_id=32685500144

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JC4DDASBJ2WPJ69PBA6XCCHH/Th0R0cFXl61qEHp7FlCTnmtfHMKy8l4gcZBCNLcFIL00NUAS/01JC4DEEWSF4SF13KZM4EFJP2M

From https://buildomat.eng.oxide.computer/wg/0/details/01JC4DDASBJ2WPJ69PBA6XCCHH/Th0R0cFXl61qEHp7FlCTnmtfHMKy8l4gcZBCNLcFIL00NUAS/01JC4DEEWSF4SF13KZM4EFJP2M#S6392in particular

Excerpt from the log showing the failure:

6392	2024-11-07T23:40:17.516Z	        FAIL [  14.347s] omicron-nexus::test_all integration_tests::updates::test_update_uninitialized
6393	2024-11-07T23:40:17.516Z	
6394	2024-11-07T23:40:17.516Z	--- STDOUT:              omicron-nexus::test_all integration_tests::updates::test_update_uninitialized ---
6395	2024-11-07T23:40:17.516Z	
6396	2024-11-07T23:40:17.516Z	running 1 test
6397	2024-11-07T23:40:17.516Z	test integration_tests::updates::test_update_uninitialized ... FAILED
6398	2024-11-07T23:40:17.516Z	
6399	2024-11-07T23:40:17.517Z	failures:
6400	2024-11-07T23:40:17.517Z	
6401	2024-11-07T23:40:17.517Z	failures:
6402	2024-11-07T23:40:17.517Z	    integration_tests::updates::test_update_uninitialized
6403	2024-11-07T23:40:17.517Z	
6404	2024-11-07T23:40:17.517Z	test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 381 filtered out; finished in 14.31s
6405	2024-11-07T23:40:17.517Z	
6406	2024-11-07T23:40:17.517Z	
6407	2024-11-07T23:40:17.517Z	--- STDERR:              omicron-nexus::test_all integration_tests::updates::test_update_uninitialized ---
6408	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.0.log
6409	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.0.log"
6410	2024-11-07T23:40:17.517Z	hint: Generated a random key:
6411	2024-11-07T23:40:17.517Z	hint:
6412	2024-11-07T23:40:17.517Z	hint:   ed25519:317be6e9847327d498582c265f44f46133df4e13879b429fcf660c2e84fc5da1
6413	2024-11-07T23:40:17.517Z	hint:
6414	2024-11-07T23:40:17.517Z	hint: To modify this repository, you will need this key. Use the -k/--key
6415	2024-11-07T23:40:17.517Z	hint: command line flag or the TUFACEOUS_KEY environment variable:
6416	2024-11-07T23:40:17.517Z	hint:
6417	2024-11-07T23:40:17.517Z	hint:   export TUFACEOUS_KEY=ed25519:317be6e9847327d498582c265f44f46133df4e13879b429fcf660c2e84fc5da1
6418	2024-11-07T23:40:17.517Z	hint:
6419	2024-11-07T23:40:17.517Z	hint: To prevent this default behavior, use --no-generate-key.
6420	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.1.log
6421	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.1.log"
6422	2024-11-07T23:40:17.517Z	DB URL: postgresql://root@[::1]:45691/omicron?sslmode=disable
6423	2024-11-07T23:40:17.517Z	DB address: [::1]:45691
6424	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.3.log
6425	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.3.log"
6426	2024-11-07T23:40:17.517Z	log file: /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.4.log
6427	2024-11-07T23:40:17.517Z	note: configured to log to "/var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.4.log"
6428	2024-11-07T23:40:17.517Z	WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6429	2024-11-07T23:40:17.517Z	WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpfnKGpK"
6430	2024-11-07T23:40:17.517Z		If you would like to access the database for debugging, run the following:
6431	2024-11-07T23:40:17.517Z	
6432	2024-11-07T23:40:17.517Z		# Run the database
6433	2024-11-07T23:40:17.517Z		cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpfnKGpK/data"
6434	2024-11-07T23:40:17.517Z		# Access the database. Note the port may change if you run multiple databases.
6435	2024-11-07T23:40:17.517Z		cockroach sql --host=localhost:32221 --insecure
6436	2024-11-07T23:40:17.517Z	WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 196466) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-3963749b3ee9bb2b-test_update_uninitialized.195832.2-clickhouse-QgILTI)
6437	2024-11-07T23:40:17.518Z	WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6438	2024-11-07T23:40:17.518Z	WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpYfWX7i
6439	2024-11-07T23:40:17.518Z	WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6440	2024-11-07T23:40:17.518Z	WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmp3JYePq
6441	2024-11-07T23:40:17.518Z	WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6442	2024-11-07T23:40:17.518Z	WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpFGtk5l
6443	2024-11-07T23:40:17.518Z	WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
6444	2024-11-07T23:40:17.518Z	WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpJ3M5SK
6445	2024-11-07T23:40:17.518Z	Error: repository fetch should have failed with 500 error
6446	2024-11-07T23:40:17.518Z	
6447	2024-11-07T23:40:17.518Z	Caused by:
6448	2024-11-07T23:40:17.518Z	    0: making request to server
6449	2024-11-07T23:40:17.518Z	    1: client error (SendRequest)
6450	2024-11-07T23:40:17.518Z	    2: connection closed before message completed
6451	2024-11-07T23:40:17.518Z	
6452	2024-11-07T23:40:17.518Z	Stack backtrace:
6453	2024-11-07T23:40:17.518Z	   0: <E as anyhow::context::ext::StdError>::ext_context
6454	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.92/src/backtrace.rs:27:14
6455	2024-11-07T23:40:17.518Z	   1: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::context
6456	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.92/src/context.rs:54:31
6457	2024-11-07T23:40:17.518Z	   2: {async_fn#0}
6458	2024-11-07T23:40:17.518Z	             at ./test-utils/src/http_testing.rs:323:28
6459	2024-11-07T23:40:17.518Z	   3: {async_fn#0}
6460	2024-11-07T23:40:17.518Z	             at ./test-utils/src/http_testing.rs:611:40
6461	2024-11-07T23:40:17.518Z	   4: {async_block#0}
6462	2024-11-07T23:40:17.518Z	             at ./tests/integration_tests/updates.rs:84:10
6463	2024-11-07T23:40:17.518Z	   5: poll<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>
6464	2024-11-07T23:40:17.518Z	             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
6465	2024-11-07T23:40:17.518Z	   6: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
6466	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
6467	2024-11-07T23:40:17.518Z	   7: with_budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
6468	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
6469	2024-11-07T23:40:17.518Z	   8: budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
6470	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
6471	2024-11-07T23:40:17.518Z	   9: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
6472	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
6473	2024-11-07T23:40:17.518Z	  10: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
6474	2024-11-07T23:40:17.518Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
6475	2024-11-07T23:40:17.519Z	  11: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
6476	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
6477	2024-11-07T23:40:17.519Z	  12: tokio::runtime::context::runtime::enter_runtime
6478	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
6479	2024-11-07T23:40:17.519Z	  13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
6480	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
6481	2024-11-07T23:40:17.519Z	  14: tokio::runtime::runtime::Runtime::block_on_inner
6482	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:363:45
6483	2024-11-07T23:40:17.519Z	  15: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
6484	2024-11-07T23:40:17.519Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:335:13
6485	2024-11-07T23:40:17.519Z	  16: test_update_uninitialized
6486	2024-11-07T23:40:17.519Z	             at ./tests/integration_tests/updates.rs:91:5
6487	2024-11-07T23:40:17.519Z	  17: test_all::integration_tests::updates::test_update_uninitialized::{{closure}}
6488	2024-11-07T23:40:17.519Z	             at ./tests/integration_tests/updates.rs:35:41
@smklein smklein added the Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken. label Nov 7, 2024
@smklein
Copy link
Collaborator Author

smklein commented Nov 7, 2024

My corresponding PR was adding new endpoints that are currently no-ops, I'm skeptical that would be related.

@smklein
Copy link
Collaborator Author

smklein commented Nov 8, 2024

Saw this again on another unrelated PR: https://buildomat.eng.oxide.computer/wg/0/details/01JC4MS802H4R9PQW8WK89JJWB/5BxKRwaDx52nKc1T2F4zG99RZVFQxNqj3Y8UH6fT3XzG7dSU/01JC4MT9CWRFYS1NH1VBQMRVC8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Test Flake Tests that work. Wait, no. Actually yes. Hang on. Something is broken.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@smklein