Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance the YAML result log with detail about failed or skipped test #2155

Open
sysarch-repo opened this issue Sep 19, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request

Comments

@sysarch-repo
Copy link

sysarch-repo commented Sep 19, 2024

Is your feature request related to a problem? Please describe.
The YAML result log can aid compliance with regulatory requirements, ensuring traceability and accountability in case audits or reviews are necessary. Example of the information stored per test:

items:
- name: increase_decrease_capacity
  status: failed
  type: essential
  points: 0
- name: node_drain
  status: skipped
  type: essential
  points: 0
- name: privileged_containers
  status: passed
  type: essential
  points: 100

In case of a failed, skipped or n/a test, there is no details about the root cause or failed resources stored in the YAML file. The stdout/stderr must be looked up to obtain this information which is a problem in machine-to-machine communications.

Describe the solution you'd like
Add information in the YAML file for failed, skipped and n/a tests. Examples:

Failed test:

details: "Failed to increase replicas from 1 to 3"
exception: "ERROR -- cnf-testsuite: Pod scaling has timed-out" 
details: "Users with cluster-admin RBAC permissions found"
failed_resources: ["ServiceAccount sonobuoy-serviceaccount"]

skipped test:
details: "node_drain chaos test requires the cluster to have at least two schedulable nodes"

n/a test:
details: "Pods are not using SELinux"

Remediation info needs not be captured in the YAML file.

Describe alternatives you've considered
Redirect the stdout/stderr output of the CNF testsuite to a file and parse the obtained data. This is ineffective as the stdout/stderr information does not seem to follow a common schema,

**How will this be tested? aka Acceptance Criteria (optional) **

(optional: unnecessary for things like some version bumps etc.)

Once this issue is addressed how will the fix be verified?

Additional context
Add any other context or screenshots about the feature request here.

@sysarch-repo sysarch-repo added the enhancement New feature or request label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants