extend AWS prowler v3 parser (#10372)

* add prowler v4 parser * remove line * fix typo * add settings.dist.py although it's written that one should not touch it but use env vars * add modified .settings.dist.py.sha256sum * extend prowler v3 parser to parse also prowler v4 reports in oscf-json format * update aws_prowler_v3.md * revert settings * add modified .settings.dist.py.sha256sum * revert docker-compose.yml * make ruff happy * separate prowler v3 and v4 parsers * renaming * add prowler v4 parser * remove line * fix typo * add settings.dist.py although it's written that one should not touch it but use env vars * add modified .settings.dist.py.sha256sum * extend prowler v3 parser to parse also prowler v4 reports in oscf-json format * update aws_prowler_v3.md * revert settings * add modified .settings.dist.py.sha256sum * make ruff happy * separate prowler v3 and v4 parsers * renaming * Update helm lock file Signed-off-by: DefectDojo <[email protected]> * make ruff happy --------- Signed-off-by: DefectDojo <[email protected]> Co-authored-by: DefectDojo <[email protected]>
DefectDojo · Jul 3, 2024 · 8b9f9a4 · 8b9f9a4
1 parent 3b14123
commit 8b9f9a4
Show file tree

Hide file tree

Showing 14 changed files with 665 additions and 136 deletions.
diff --git a/docs/content/en/integrations/parsers/file/aws_prowler_v3.md b/docs/content/en/integrations/parsers/file/aws_prowler_v3.md
diff --git a/docs/content/en/integrations/parsers/file/aws_prowler_v3plus.md b/docs/content/en/integrations/parsers/file/aws_prowler_v3plus.md
@@ -0,0 +1,163 @@
+---
+title: "AWS Prowler V3"
+toc_hide: true
+---
+
+### File Types
+DefectDojo parser accepts a native `json` file produced by prowler v3 with file extension `.json` or a `ocsf-json` file produced by prowler v4 with file extension `.ocsf.json`. 
+Please note: earlier versions of AWS Prowler create output data in a different format. See our other [prowler parser documentation](https://documentation.defectdojo.com/integrations/parsers/file/aws_prowler/) if you are using an earlier version of AWS Prowler. 
+
+JSON reports can be created from the [AWS Prowler v3 CLI](https://docs.prowler.com/projects/prowler-open-source/en/v3/tutorials/reporting/#json) using the following command: `prowler <provider> -M json`
+
+JSON-OCSF reports can be created from the [AWS Prowler v4 CLI](https://docs.prowler.cloud/en/latest/tutorials/reporting/#json) using the following command: `prowler <provider> -M json-ocsf`
+
+
+### Acceptable Prowler v3 JSON format
+Parser expects an array of assessments. All properties are strings and are required by the parser.
+
+~~~
+
+[
+        {
+            "AssessmentStartTime": "example_timestamp",
+            "FindingUniqueId": "example_uniqueIdFromTool",
+            "Provider": "example_provider",
+            "CheckID": "acm_certificates_expiration_check",
+            "CheckTitle": "Check if ACM Certificates are about to expire in specific days or less",
+            "CheckType": [
+                "Example ASFF-Compliant Finding Type"
+            ],
+            "ServiceName": "example_awsServiceName",
+            "SubServiceName": "",
+            "Status": "FAIL",
+            "StatusExtended": "Example status description",
+            "Severity": "example_severity",
+            "ResourceType": "AwsCertificateManagerCertificate",
+            "ResourceDetails": "",
+            "Description": "Example general test description.",
+            "Risk": "Example test impact description.",
+            "RelatedUrl": "https://docs.aws.amazon.com/config/latest/developerguide/acm-certificate-expiration-check.html",
+            "Remediation": {
+                "Code": {
+                    "NativeIaC": "",
+                    "Terraform": "",
+                    "CLI": "",
+                    "Other": ""
+                },
+                "Recommendation": {
+                    "Text": "Example recommendation.",
+                    "Url": "https://docs.aws.amazon.com/config/latest/developerguide/example_related_documentation.html"
+                }
+            },
+            "Compliance": {
+                    "GDPR": [
+                        "article_32"
+                    ],
+                ...
+            },
+            "Categories": [],
+            "DependsOn": [],
+            "RelatedTo": [],
+            "Notes": "",
+            "Profile": null,
+            "AccountId": "example_accountId",
+            "OrganizationsInfo": null,
+            "Region": "example_region",
+            "ResourceId": "example.resource.id.com",
+            "ResourceArn": "arn:aws:acm:us-east-1:999999999999:certificate/ffffffff-0000-0000-0000-000000000000",
+            "ResourceTags": {}
+        }
+    ...
+]
+
+~~~
+
+### Acceptable Prowler v4 JSON-OCSF format
+The parser expects an array of assessments. All properties are strings and are required by the parser.
+
+~~~
+[{
+    "metadata": {
+        "event_code": "iam_role_administratoraccess_policy_permissive_trust_relationship",
+        "product": {
+            "name": "Prowler",
+            "vendor_name": "Prowler",
+            "version": "4.2.1"
+        },
+        "version": "1.2.0"
+    },
+    "severity_id": 4,
+    "severity": "High",
+    "status": "Suppressed",
+    "status_code": "FAIL",
+    "status_detail": "IAM Role myAdministratorExecutionRole has AdministratorAccess policy attached that has too permissive trust relationship.",
+    "status_id": 3,
+    "unmapped": {
+        "check_type": "",
+        "related_url": "https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_administrator",
+        "categories": "trustboundaries",
+        "depends_on": "",
+        "related_to": "",
+        "notes": "CAF Security Epic: IAM",
+        "compliance": {}
+    },
+    "activity_name": "Create",
+    "activity_id": 1,
+    "finding_info": {
+        "created_time": "2024-06-03T14:15:19.382075",
+        "desc": "Ensure IAM Roles with attached AdministratorAccess policy have a well defined trust relationship",
+        "product_uid": "prowler",
+        "title": "Ensure IAM Roles with attached AdministratorAccess policy have a well defined trust relationship",
+        "uid": "prowler-aws-iam_role_administratoraccess_policy_permissive_trust_relationship-123456789012-us-east-1-myAdministratorExecutionRole"
+    },
+    "resources": [
+        {
+            "cloud_partition": "aws",
+            "region": "us-east-1",
+            "data": {
+                "details": ""
+            },
+            "group": {
+                "name": "iam"
+            },
+            "labels": [],
+            "name": "myAdministratorExecutionRole",
+            "type": "AwsIamRole",
+            "uid": "arn:aws:iam::123456789012:role/myAdministratorExecutionRole"
+        }
+    ],
+    "category_name": "Findings",
+    "category_uid": 2,
+    "class_name": "DetectionFinding",
+    "class_uid": 2004,
+    "cloud": {
+        "account": {
+            "name": "",
+            "type": "AWS_Account",
+            "type_id": 10,
+            "uid": "123456789012",
+            "labels": []
+        },
+        "org": {
+            "name": "",
+            "uid": ""
+        },
+        "provider": "aws",
+        "region": "us-east-1"
+    },
+    "event_time": "2024-06-03T14:15:19.382075",
+    "remediation": {
+        "desc": "Apply the principle of least privilege. Instead of AdministratorAccess, assign only the permissions necessary for specific roles and tasks. Create custom IAM policies with minimal permissions based on the principle of least privilege. If a role really needs AdministratorAccess, the trust relationship must be well defined to restrict it usage only to the Principal, Action, Audience and Subject intended for it.",
+        "references": [
+            "https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege"
+        ]
+    },
+    "risk_details": "The AWS-managed AdministratorAccess policy grants all actions for all AWS services and for all resources in the account and as such exposes the customer to a significant data leakage threat. It is therefore particularly important that the trust relationship is well defined to restrict it usage only to the Principal, Action, Audience and Subject intended for it.",
+    "type_uid": 200401,
+    "type_name": "Create"
+}]
+
+~~~
+
+### Sample Scan Data
+Unit tests of AWS Prowler v3 JSON and Prowler v4 JSON-OCSF can be found at https://github.com/DefectDojo/django-DefectDojo/tree/master/unittests/scans/aws_prowler_v3.
diff --git a/dojo/tools/aws_prowler_v3/__init__.py → dojo/tools/aws_prowler_v3plus/__init__.py b/dojo/tools/aws_prowler_v3/__init__.py → dojo/tools/aws_prowler_v3plus/__init__.py
diff --git a/dojo/tools/aws_prowler_v3plus/parser.py b/dojo/tools/aws_prowler_v3plus/parser.py
@@ -0,0 +1,24 @@
+from dojo.tools.aws_prowler_v3plus.prowler_v3 import AWSProwlerV3Parser
+from dojo.tools.aws_prowler_v3plus.prowler_v4 import AWSProwlerV4Parser
+
+
+class AWSProwlerV3plusParser:
+    SCAN_TYPE = ["AWS Prowler V3"]
+
+    def get_scan_types(self):
+        return AWSProwlerV3plusParser.SCAN_TYPE
+
+    def get_label_for_scan_types(self, scan_type):
+        return AWSProwlerV3plusParser.SCAN_TYPE[0]
+
+    def get_description_for_scan_types(self, scan_type):
+        return "Exports from AWS Prowler v3 in JSON format or from Prowler v4 in OCSF-JSON format."
+
+    def get_findings(self, file, test):
+        if file.name.lower().endswith('.ocsf.json'):
+            return AWSProwlerV4Parser().process_ocsf_json(file, test)
+        elif file.name.lower().endswith('.json'):
+            return AWSProwlerV3Parser().process_json(file, test)
+        else:
+            msg = 'Unknown file format'
+            raise ValueError(msg)
diff --git a/dojo/tools/aws_prowler_v3/parser.py → dojo/tools/aws_prowler_v3plus/prowler_v3.py b/dojo/tools/aws_prowler_v3/parser.py → dojo/tools/aws_prowler_v3plus/prowler_v3.py
@@ -1,4 +1,3 @@
-
 import hashlib
 import json
 import textwrap
@@ -8,24 +7,6 @@
 
 
 class AWSProwlerV3Parser:
-    SCAN_TYPE = ["AWS Prowler V3"]
-
-    def get_scan_types(self):
-        return AWSProwlerV3Parser.SCAN_TYPE
-
-    def get_label_for_scan_types(self, scan_type):
-        return AWSProwlerV3Parser.SCAN_TYPE[0]
-
-    def get_description_for_scan_types(self, scan_type):
-        return "Export of AWS Prowler JSON V3 format."
-
-    def get_findings(self, file, test):
-        if file.name.lower().endswith('.json'):
-            return self.process_json(file, test)
-        else:
-            msg = 'Unknown file format'
-            raise ValueError(msg)
-
     def process_json(self, file, test):
         dupes = {}
 
@@ -96,9 +77,3 @@ def process_json(self, file, test):
                 dupes[dupe_key] = find
 
         return list(dupes.values())
-
-    def formatview(self, depth):
-        if depth > 1:
-            return "* "
-        else:
-            return ""
diff --git a/dojo/tools/aws_prowler_v3plus/prowler_v4.py b/dojo/tools/aws_prowler_v3plus/prowler_v4.py
@@ -0,0 +1,83 @@
+import hashlib
+import json
+import textwrap
+from datetime import date
+
+from dojo.models import Finding
+
+
+class AWSProwlerV4Parser:
+    def process_ocsf_json(self, file, test):
+        dupes = {}
+
+        data = json.load(file)
+        # mapping of json fields between Prowler v3 and v4:
+        # https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/reporting/#json
+        for deserialized in data:
+
+            status = deserialized.get('status_code')
+            if status.upper() != 'FAIL':
+                continue
+
+            account_id = deserialized.get('cloud', {}).get('account', {}).get("uid", '')
+            region = deserialized.get('resources', [{}])[0].get('region', '')
+            provider = deserialized.get('cloud', {}).get('provider', '')
+            compliance = ''
+            compliance_field = deserialized.get('unmapped', {}).get("compliance", {})
+            if compliance_field:
+                compliance = ' | '.join([f"{key}:{','.join(value)}" for key, value in compliance_field.items()])
+            result_extended = deserialized.get('status_detail')
+            general_description = deserialized.get('finding_info', {}).get('desc', '')
+            asff_compliance_type = deserialized.get('unmapped', {}).get('check_type', '')
+            severity = deserialized.get('severity', 'Info').capitalize()
+            aws_service_name = deserialized.get('resources', [{}])[0].get('group', {}).get('name', '')
+            impact = deserialized.get('risk_details')
+            mitigation = deserialized.get('remediation', {}).get("desc", '')
+            documentation = deserialized.get('remediation', {}).get("references", '')
+            documentation = str(documentation) + "\n" + str(deserialized.get('unmapped', {}).get('related_url', ''))
+            security_domain = deserialized.get('resources', [{}])[0].get('type', '')
+            timestamp = deserialized.get("event_time")
+            resource_arn = deserialized.get('resources', [{}])[0].get('uid', '')
+            resource_id = deserialized.get('resources', [{}])[0].get('name', '')
+            unique_id_from_tool = deserialized.get('finding_info', {}).get('uid', '')
+            if not resource_arn or resource_arn == "":
+                component_name = str(provider) + "-" + str(account_id) + "-" + str(region) + "-" + str(resource_id)
+            else:
+                component_name = resource_arn
+
+            description = "**Issue:** " + str(result_extended) + \
+                          "\n**Description:** " + str(general_description) + \
+                          "\n**AWS Account:** " + str(account_id) + \
+                          "\n**Region:** " + str(region) + \
+                          "\n**AWS Service:** " + str(aws_service_name) + \
+                          "\n**Security Domain:** " + str(security_domain) + \
+                          "\n**Compliance:** " + str(compliance) + \
+                          "\n**ASFF Compliance Type:** " + str(asff_compliance_type)
+
+            # improving key to get duplicates
+            dupe_key = hashlib.sha256(unique_id_from_tool.encode('utf-8')).hexdigest()
+            if dupe_key in dupes:
+                find = dupes[dupe_key]
+                if description is not None:
+                    find.description += description + "\n\n"
+                find.nb_occurences += 1
+            else:
+                find = Finding(
+                    title=textwrap.shorten(result_extended, 150),
+                    cwe=1032,  # Security Configuration Weaknesses, would like to fine tune
+                    test=test,
+                    description=description,
+                    component_name=component_name,
+                    unique_id_from_tool=unique_id_from_tool,
+                    severity=severity,
+                    references=documentation,
+                    date=date.fromisoformat(timestamp[:10]),
+                    static_finding=True,
+                    dynamic_finding=False,
+                    nb_occurences=1,
+                    mitigation=mitigation,
+                    impact=impact,
+                )
+                dupes[dupe_key] = find
+
+        return list(dupes.values())
diff --git a/...tests/scans/aws_prowler_v3/many_vuln.json → ...s/scans/aws_prowler_v3plus/many_vuln.json b/...tests/scans/aws_prowler_v3/many_vuln.json → ...s/scans/aws_prowler_v3plus/many_vuln.json