add prowler v4 parser

[kagahd]

add prowler v4 parser

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Configured Codepaths Analyzer	✅	0 findings
Secrets Analyzer	✅	0 findings
Sensitive Files Analyzer	✅	0 findings
AppSec Analyzer	✅	0 findings
Authn/Authz Analyzer	✅	0 findings

Note

🟢 Risk threshold not exceeded.

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

This pull request contains several changes related to the integration of the AWS Prowler V4 security assessment tool in the DefectDojo application. The key changes include:

1. Documentation Update: The addition of documentation for the AWS Prowler V4 parser, including information about the expected file format, JSON structure, and a sample scan data.
2. Configuration Update: The modification of the dojo/settings/.settings.dist.py.sha256sum file, which indicates changes to the configuration file. This requires a review of the actual configuration file changes to ensure no security-sensitive settings have been altered.
3. Supported Scanners Update: The addition of the AWS Prowler V4 scanner to the list of supported scanners in the dojo/settings/settings.dist.py file, along with the configuration of the deduplication algorithm for this scanner.
4. Parser Implementation: The addition of the dojo/tools/aws_prowler_v4/parser.py file, which contains the implementation of the parser for the AWS Prowler V4 security assessment tool. The parser focuses on processing the JSON output, deduplicating findings, and creating detailed security findings with relevant information.
5. Unit Tests: The addition of several unit test files (unittests/scans/aws_prowler_v4/no_vuln.json, unittests/scans/aws_prowler_v4/one_vuln.json, unittests/scans/aws_prowler_v4/many_vuln.json, and unittests/tools/test_aws_prowler_v4_parser.py) to ensure the correct functioning of the AWS Prowler V4 parser.

From an application security perspective, the key points to highlight are:

1. The focus on standardized data formats (JSON OCSF) and the inclusion of security-relevant information in the findings, such as severity, status, and remediation recommendations.
2. The implementation of deduplication logic to ensure that the same finding is not reported multiple times, improving the efficiency of the security review process.
3. The mapping of findings to the CWE (1032 for Security Configuration Weaknesses), which provides a standardized way to categorize the security issues.
4. The emphasis on the principle of least privilege and the identification of overly permissive IAM roles and policies, which are crucial for securing AWS environments.
5. The addition of comprehensive unit tests to ensure the reliability and accuracy of the AWS Prowler V4 parser, which is an important component of the application's security assessment capabilities.

Overall, these changes appear to be a positive contribution to the security and functionality of the DefectDojo application, as they enhance the integration and processing of security findings from the AWS Prowler V4 tool.

Files Changed:

1. docs/content/en/integrations/parsers/file/aws_prowler_v4.md: Updated documentation for the AWS Prowler V4 parser.
2. dojo/settings/.settings.dist.py.sha256sum: Updated the SHA-256 hash value, indicating changes to the dojo/settings/.settings.dist.py configuration file.
3. dojo/settings/settings.dist.py: Added the AWS Prowler V4 scanner to the list of supported scanners and configured the deduplication algorithm.
4. dojo/tools/aws_prowler_v4/parser.py: Implemented the parser for the AWS Prowler V4 security assessment tool.
5. unittests/scans/aws_prowler_v4/no_vuln.json: Added a test file with no vulnerabilities.
6. unittests/scans/aws_prowler_v4/one_vuln.json: Added a test file with a single vulnerability.
7. unittests/scans/aws_prowler_v4/many_vuln.json: Added a test file with multiple vulnerabilities.
8. unittests/tools/test_aws_prowler_v4_parser.py: Added unit tests for the AWS Prowler V4 parser.

Powered by DryRun Security

[manuel-sommer]

small typo

[Maffooch]

@kagahd check out this note
https://github.com/DefectDojo/django-DefectDojo/blob/67a75710d6991a3ee3d4ed8dca7d0ef5c9cef46a/dojo/settings/settings.dist.py#L1C1-L11C106

[kagahd]

It seems that only 1 unit test in alpine and also in debian failed:
FAIL: test_delete_preview (unittests.test_rest_framework.TestsTest.test_delete_preview)
However, I don't know how my code contribution could have any effect to make this unit test fail.
Any hints on this?

[mtesauro]

@kagahd We've had issues with those tests failing somewhat randomly. Give is a day or so to look into what's up with those tests before spending more time on this PR to try to make them pass.

Thanks for the contribution and your patience as we sort out those misbehaving tests.

[kagahd]

It seems it's not a good idea to have multiple parsers for the same scanner even though the same scanner may have different report formats (e.g. csv, json-asff, json-ocsf, etc.) wich would require different parsers.
It seems that DefectDojo is not designed to allow multiple parsers, called "tools", for the same scanner because deduplication and closing of findings that are missing in a report are bound to the "tool", hence the parser.

For example, if you import a prowler v3 report and then import a semantically identical prowler v4 report, DefectDojo would not detect any duplicates because two different parsers, "tools", were used to import both reports.
The same applies for closing findings that are missing in a report: if you import a prowler v3 report and then import an empty prowler v4 report, DefectDojo would not close any of the prowler v3 findings.

So I think the only viable solution to this problem is to implement only one parser, one "tool", for one scanner. That means that we would have only one parser, one "tool", for Prowler which must be agnostic of the different report formats that Prowler may produce or may have produced in former Prowler versions.

[kagahd]

Dear @mtesauro, @Maffooch, @manuel-sommer & all the DefectDojo team,

I have a question regarding the implementation for only one Prowler parser for several report formats.
If there is only one Prowler scanner for all report formats, how do you define the different DEDUPLICATION_ALGORITHM_PER_PARSER? For the Prowler parsers v3 and v4 it is DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL which results in unique_id_from_tool but for the Prowler parser versions before that, called AWS Prowler Scan in DefectDojo, it is DEDUPE_ALGO_HASH_CODE which results in hash_code.

How about renaming the parser AWS Prowler Scan to something like AWS Prowler legacy? The current Prowler v3 and v4 parser could be merged into only one parser, able to parse different formats of reports, i.e. json and json-ocsf, as both use the same DEDUPLICATION_ALGORITHM_PER_PARSER. We could or should call the parser for example Prowler v3 plus, without the prefix AWS because Prowler is able to create more than just AWS reports. The suffix v3 plus indicates that it can be used from Prowler version 3.

What do you think of the idea or what would you suggest as the best solution?

[manuel-sommer]

Hi @kagahd,
you have multiple options:

• Take a look at RustyHog https://github.com/DefectDojo/django-DefectDojo/blob/master/dojo/tools/rusty_hog/parser.py --> But here the problem is that you have only one deduplication parser available. Here the right parser is automatically detected. Try it in the GUI, it is pretty nice.
• Take a look at Checkmarx Scan https://github.com/DefectDojo/django-DefectDojo/blob/master/dojo/tools/checkmarx/parser.py --> Here you have one parser with 2 deduplications available.

Thus, maybe you can:

• Merge Prowler, Prowler3 and Prowler4 together to "Prowler" and use the functionalities of both RustyHog and Checkmarx Scan to resolve all problems.
• For this, you would have to separate the different prowler versions into multiple files (classes), like I did in multiple PRs, e.g. https://github.com/DefectDojo/django-DefectDojo/pull/9922/files (I separated here the json and csv part).

If you do it this way, you might have to add a migrationsfile if you rename the parser name, e.g. look at #9355 or #9522.

Does this help you?

[kagahd]

Does this help you?

Indeed, that helps, thanks a lot for the pointers, @manuel-sommer!

[kagahd]

I close this PR in favor of PR #10372