fix(logcollection): save schema info output

[juliayakovlev]

This commit brings the few fixes:

1. save schema info on the runner locally and collect it in SCT folder
2. use silence decorator for every run of schema info command. It's to prevent stop info collection on failure
3. collect schema info from one live node

Testing

• 1 network interface
• 2 network interfaces
• gce
• upgrade-scylla-k8s-gke-test

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[juliayakovlev]

@grzywin @vponomaryov
I ran the upgrade-scylla-k8s-gke-test with this commit.

Test passed https://argus.scylladb.com/tests/scylla-cluster-tests/ae081b90-89ec-47ec-a515-b2acd04daeb1. The schema info was saved fast. The schema files are collected under sct-events folder:

< t:2024-12-11 18:51:07,943 f:tester.py       l:2979 c:UpgradeTest          p:INFO  > Save nodes user schema in the files
< t:2024-12-11 18:51:07,949 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'desc schema' output in the file. Node sct-cluster-us-east1-rack-1-0
 < t:2024-12-11 18:51:08,590 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'select JSON * from system_schema.tables' output in the file. Node sct-cluster-us-east1-rack-1-0
< t:2024-12-11 18:51:09,243 f:tester.py       l:3001 c:UpgradeTest          p:INFO  > Post test validators are starting..

Which test(s) is worth to run to validate this fix?

[fruch]

LGTM

now it's handled better

[vponomaryov]

@grzywin @vponomaryov I ran the upgrade-scylla-k8s-gke-test with this commit.

Test passed https://argus.scylladb.com/tests/scylla-cluster-tests/ae081b90-89ec-47ec-a515-b2acd04daeb1. The schema info was saved fast. The schema files are collected under sct-events folder:

< t:2024-12-11 18:51:07,943 f:tester.py       l:2979 c:UpgradeTest          p:INFO  > Save nodes user schema in the files
< t:2024-12-11 18:51:07,949 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'desc schema' output in the file. Node sct-cluster-us-east1-rack-1-0
 < t:2024-12-11 18:51:08,590 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'select JSON * from system_schema.tables' output in the file. Node sct-cluster-us-east1-rack-1-0
< t:2024-12-11 18:51:09,243 f:tester.py       l:3001 c:UpgradeTest          p:INFO  > Post test validators are starting..

Why in the events log archive?
Why not in the root of the db logs archive? I would expect to see it there because it is exactly DB info.

Which test(s) is worth to run to validate this fix?

It worth to see what happens when test ends with failure or error.
I see that the info is gathered before teardown, will it be triggered in case of test timeout? Critical error?
When test stage is aborted?

[juliayakovlev]

@grzywin @vponomaryov I ran the upgrade-scylla-k8s-gke-test with this commit.
Test passed https://argus.scylladb.com/tests/scylla-cluster-tests/ae081b90-89ec-47ec-a515-b2acd04daeb1. The schema info was saved fast. The schema files are collected under sct-events folder:

< t:2024-12-11 18:51:07,943 f:tester.py       l:2979 c:UpgradeTest          p:INFO  > Save nodes user schema in the files
< t:2024-12-11 18:51:07,949 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'desc schema' output in the file. Node sct-cluster-us-east1-rack-1-0
 < t:2024-12-11 18:51:08,590 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'select JSON * from system_schema.tables' output in the file. Node sct-cluster-us-east1-rack-1-0
< t:2024-12-11 18:51:09,243 f:tester.py       l:3001 c:UpgradeTest          p:INFO  > Post test validators are starting..

Why in the events log archive? Why not in the root of the db logs archive? I would expect to see it there because it is exactly DB info.

per request #9513 (comment).

Which test(s) is worth to run to validate this fix?

It worth to see what happens when test ends with failure or error. I see that the info is gathered before teardown, will it be triggered in case of test timeout? Critical error? When test stage is aborted?

• in case a test timeout/critical error the teardown will be called from sdcm.tester.teardown_on_exception and schema info will be collected
• in case aborting (sending interrupt signal or kill job run) the teardown is not called, schema info won't be collected.

I meant in my question which k8s/operator test is worth to run

[vponomaryov]

@grzywin @vponomaryov I ran the upgrade-scylla-k8s-gke-test with this commit.
Test passed https://argus.scylladb.com/tests/scylla-cluster-tests/ae081b90-89ec-47ec-a515-b2acd04daeb1. The schema info was saved fast. The schema files are collected under sct-events folder:

< t:2024-12-11 18:51:07,943 f:tester.py       l:2979 c:UpgradeTest          p:INFO  > Save nodes user schema in the files
< t:2024-12-11 18:51:07,949 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'desc schema' output in the file. Node sct-cluster-us-east1-rack-1-0
 < t:2024-12-11 18:51:08,590 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'select JSON * from system_schema.tables' output in the file. Node sct-cluster-us-east1-rack-1-0
< t:2024-12-11 18:51:09,243 f:tester.py       l:3001 c:UpgradeTest          p:INFO  > Post test validators are starting..

Why in the events log archive? Why not in the root of the db logs archive? I would expect to see it there because it is exactly DB info.

per request #9513 (comment).

The request doesn't match the implemented approach.
There was no request to store schema info in the sct events log archive.
And stating once again, the schema info must be in the root of the db log archive, it matches the request you referenced.

I meant in my question which k8s/operator test is worth to run

One successful and one failed should be enough.

[fruch]

@grzywin @vponomaryov I ran the upgrade-scylla-k8s-gke-test with this commit.
Test passed https://argus.scylladb.com/tests/scylla-cluster-tests/ae081b90-89ec-47ec-a515-b2acd04daeb1. The schema info was saved fast. The schema files are collected under sct-events folder:

< t:2024-12-11 18:51:07,943 f:tester.py       l:2979 c:UpgradeTest          p:INFO  > Save nodes user schema in the files
< t:2024-12-11 18:51:07,949 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'desc schema' output in the file. Node sct-cluster-us-east1-rack-1-0
 < t:2024-12-11 18:51:08,590 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'select JSON * from system_schema.tables' output in the file. Node sct-cluster-us-east1-rack-1-0
< t:2024-12-11 18:51:09,243 f:tester.py       l:3001 c:UpgradeTest          p:INFO  > Post test validators are starting..

Why in the events log archive? Why not in the root of the db logs archive? I would expect to see it there because it is exactly DB info.

per request #9513 (comment).

The request doesn't match the implemented approach. There was no request to store schema info in the sct events log archive. And stating once again, the schema info must be in the root of the db log archive, it matches the request you referenced.

I asked to be in the test side, i.e. no on every node.
if it's on the extra SCT file (which for some reason called events, it has more than that)
or in the db log file, I don't think it's make a bit difference, since no one expect it to be anywhere now.

so from my POV it's a nitpick, and can be a followup if one insists.
meanwhile this is failing/distributing multiple jobs

I meant in my question which k8s/operator test is worth to run

One successful and one failed should be enough.

[juliayakovlev]

@grzywin @vponomaryov I ran the upgrade-scylla-k8s-gke-test with this commit.
Test passed https://argus.scylladb.com/tests/scylla-cluster-tests/ae081b90-89ec-47ec-a515-b2acd04daeb1. The schema info was saved fast. The schema files are collected under sct-events folder:

< t:2024-12-11 18:51:07,943 f:tester.py       l:2979 c:UpgradeTest          p:INFO  > Save nodes user schema in the files
< t:2024-12-11 18:51:07,949 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'desc schema' output in the file. Node sct-cluster-us-east1-rack-1-0
 < t:2024-12-11 18:51:08,590 f:tester.py       l:2965 c:UpgradeTest          p:INFO  > Save command 'select JSON * from system_schema.tables' output in the file. Node sct-cluster-us-east1-rack-1-0
< t:2024-12-11 18:51:09,243 f:tester.py       l:3001 c:UpgradeTest          p:INFO  > Post test validators are starting..

Why in the events log archive? Why not in the root of the db logs archive? I would expect to see it there because it is exactly DB info.

per request #9513 (comment).

The request doesn't match the implemented approach. There was no request to store schema info in the sct events log archive. And stating once again, the schema info must be in the root of the db log archive, it matches the request you referenced.

I meant in my question which k8s/operator test is worth to run

One successful and one failed should be enough.

I'll move the logs to "db-cluster"