Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve new CLI testing ensuring complete coverage of arguments cases #652

Merged
merged 11 commits into from
Nov 17, 2023

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented Nov 7, 2023

Fixes #562.

1. Changes:

  1. Adds unit test to verify the behavior and decisions of the new CLI based on user's input arguments.
  2. Ensures that tests have covered all possible states of the platform, cluster, event logs fields.

2. New CLI User Input:

ascli --platform <platform> --cluster <cluster> --eventlogs <eventlogs> [remainings args..]

3. Possible States:

  • platform:undefined or actual value.
  • cluster: undefined, cluster name, cluster property file or auto tuner file.
  • event logs: undefined or actual value.

4. Cases Covered:

We have a total of 16 cases (2 X 4 X 2). However, among these, there are only 8 unique test cases. Each of the 8 test case has an argument (platform or event logs) for which the result (pass or fail) remains same irrespective of the argument's value. Thus, each of the 8 test cases has two sub-cases: (a) argument is provided, and (b)argument is not provided.

For example, consider the user input where cluster argument is not provided but event logs are provided. It should pass whether the platform is provided or not.

List of Test Cases:

Platform Cluster Event Logs Result
Does Not Matter Undefined Undefined Fail
Does Not Matter Undefined Defined Pass
Defined Cluster Name Does Not Matter Pass (except onPrem)
Undefined Cluster Name Does Not Matter Fail
Does Not Matter Cluster Props Undefined Pass (except onPrem)
Does Not Matter Cluster Props Defined Pass
Does Not Matter Auto Tuner Undefined Fail
Does Not Matter Auto Tuner Defined Pass

5. Code Changes:

user_tools/tests/spark_rapids_tools_ut/test_tool_argprocessor.py:

  • In each unit test, we use the @register_triplet_test decorator with the parameters <platform, cluster, eventlog>. This decorator is responsible for adding the triplet value to a global dictionary.

6. Correctness Check:

The final unit test, test_arg_cases_coverage, checks if the dictionary contains all 16 triplets, thus ensuring that these unit tests cover all 16 cases.

Signed-off-by: Partho Sarthi <[email protected]>
Signed-off-by: Partho Sarthi <[email protected]>
@parthosa parthosa added the user_tools Scope the wrapper module running CSP, QualX, and reports (python) label Nov 7, 2023
@parthosa parthosa self-assigned this Nov 7, 2023
# Conflicts:
#	user_tools/tests/spark_rapids_tools_ut/test_tool_argprocessor.py
Signed-off-by: Partho Sarthi <[email protected]>
nartal1
nartal1 previously approved these changes Nov 14, 2023
Copy link
Collaborator

@nartal1 nartal1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @parthosa !

Signed-off-by: Partho Sarthi <[email protected]>
Copy link
Collaborator

@cindyyuanjiang cindyyuanjiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa!

Copy link
Collaborator

@nartal1 nartal1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa !

@parthosa parthosa merged commit 6d5c432 into NVIDIA:dev Nov 17, 2023
8 checks passed
@parthosa parthosa deleted the spark-rapids-tools-562 branch November 17, 2023 00:05
@parthosa parthosa added the new-cli scope out future new-cli work for the next few months label Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-cli scope out future new-cli work for the next few months user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add more argprocessor tests to cover all scenarios
3 participants