Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

task failure due to reboot request time out should print off all available logs up to that point #109

Open
anonymouse64 opened this issue Oct 5, 2020 · 0 comments

Comments

@anonymouse64
Copy link

Currently when debugging tests where the system is rebooted and it fails to come back after a reboot, we receive no information about the text execution like we would if the execute section had failed, etc. See for example this failure:


sergio@cachiomachine:~/workspace/snapcore/snapd$ spread external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:04:32 Project content is packed for delivery (4.66MB).
2020-10-02 16:04:32 If killed, discard servers with: spread -reuse-pid=233681 -discard
2020-10-02 16:04:32 Allocating external:ubuntu-core-20-64...
2020-10-02 16:04:32 Waiting for external:ubuntu-core-20-64 to make SSH available at localhost:8022...
2020-10-02 16:04:32 Allocated external:ubuntu-core-20-64.
2020-10-02 16:04:32 Connecting to external:ubuntu-core-20-64...
2020-10-02 16:04:32 Connected to external:ubuntu-core-20-64 at localhost:8022.
2020-10-02 16:04:32 Sending project content to external:ubuntu-core-20-64...
2020-10-02 16:04:33 Preparing external:ubuntu-core-20-64 (external:ubuntu-core-20-64)...
2020-10-02 16:04:33 Preparing external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64)...
2020-10-02 16:07:20 Preparing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64)...
2020-10-02 16:07:22 Executing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) (1/1)...
2020-10-02 16:08:56 Rebooting on external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) as requested...
2020-10-02 16:13:12 Rebooting on external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) as requested...
2020-10-02 16:18:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Current output:
-----
mkdir: cannot create directory ‘/home/gopath’: File exists
sudo: unable to resolve host ubuntu: Temporary failure in name resolution
-----
.
2020-10-02 16:23:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:28:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:33:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:38:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:43:11 Error executing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : kill-timeout reached after external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) reboot request
2020-10-02 16:43:11 Error debugging external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : ssh: unexpected packet in response to channel open: <nil>
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64 (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64 (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Discarding external:ubuntu-core-20-64...
2020-10-02 16:43:11 Successful tasks: 0
2020-10-02 16:43:11 Aborted tasks: 0
2020-10-02 16:43:11 Failed tasks: 1
    - external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:43:11 Failed task restore: 1
    - external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:43:11 Failed suite restore: 1
    - external:ubuntu-core-20-64:tests/core/
2020-10-02 16:43:11 Failed project restore: 1
    - external:ubuntu-core-20-64:project
error: unsuccessful run

The only information we get is the "is taking too long Current Output" message which is not enough information, and additionally the debug section is not executed.

@anonymouse64 anonymouse64 changed the title task failure due to reboot request should print off all available logs up to that point task failure due to reboot request time out should print off all available logs up to that point Oct 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant