base trufflehog hashes on file_path and payload

[brieucR]

Description

Hashes based on description are not suitable on a long-term basis. Any change from Trufflehog or DefectDojo contributors to add/delete/modify fields in the description will lead to duplicates.
Currently, the best way to deduplicate findings is to consider file_path and payload fields.
That ensures a new finding is created each time a secret is introduced in a file.

Checklist

This checklist is for your information.

• Make sure to rebase your PR against the very latest dev.
• Features/Changes should be submitted against the dev.
• Give a meaningful name to your PR, as it may end up being used in the release notes.
• Your code is flake8 compliant.
• Your code is python 3.11 compliant.
• Add the proper label to categorize your PR.

Extra information

Please clear everything below when submitting your pull request, it's here purely for your information.

Moderators: Labels currently accepted for PRs:

• Import Scans (for new scanners/importers)
• enhancement
• performance
• feature
• bugfix
• maintenance (a.k.a chores)
• dependencies
• New Migration (when the PR introduces a DB migration)
• settings_changes (when the PR introduces changes or new settings in settings.dist.py)

Contributors: Git Tips

Rebase on dev branch

If the dev branch has changed since you started working on it, please rebase your work after the current dev.

On your working branch mybranch:

git rebase dev mybranch

In case of conflict:

 git mergetool
 git rebase --continue

When everything's fine on your local branch, force push to your myOrigin remote:

git push myOrigin --force-with-lease

To cancel everything:

git rebase --abort

Squashing commits

git rebase -i origin/dev

• Replace pick by fixup on the commits you want squashed out
• Replace pick by reword on the first commit if you want to change the commit message
• Save the file and quit your editor

Force push to your myOrigin remote:

git push myOrigin --force-with-lease

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Server-Side Request Forgery Analyzer	✅	0 findings
Configured Codepaths Analyzer	✅	0 findings
IDOR Analyzer	✅	0 findings
Sensitive Files Analyzer	✅	0 findings
SQL Injection Analyzer	✅	0 findings
Authn/Authz Analyzer	✅	0 findings
Secrets Analyzer	✅	0 findings

Note

🟢 Risk threshold not exceeded.

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

The code changes in this pull request focus on improving the integration between the DefectDojo application and the Trufflehog security scanner. The key changes include:

1. Updating the configuration in the settings.dist.py file to map the "file_path" and "payload" attributes from the Trufflehog scan results to the corresponding fields in the DefectDojo finding model. This ensures that the relevant information from the Trufflehog scan, such as the file path and the sensitive data payload, are properly imported and stored in the DefectDojo application.

2. Modifying the parser.py file to include the raw, unredacted version of the sensitive information detected by Trufflehog in the payload field of the Finding object. This change raises some security concerns, as the sensitive data could potentially be exposed if not properly secured and access-controlled.

To address the security concerns introduced by the changes to the parser.py file, the application security team should consider implementing the following recommendations:

• Secure storage and access control for the payload field in the Finding object.
• Redaction or obfuscation of the sensitive information before storing or displaying it.
• Robust logging and monitoring mechanisms to track access and usage of the sensitive information.
• Clear policies and procedures for the secure handling of sensitive information detected by Trufflehog.

By addressing these security considerations, the application security team can ensure that the changes to the DefectDojo application do not introduce additional security risks while still providing the benefits of improved integration with the Trufflehog security scanner.

Files Changed:

1. dojo/settings/settings.dist.py:

   • The configuration for the "Trufflehog Scan" parser has been updated to map the "file_path" and "payload" attributes from the Trufflehog scan results to the corresponding fields in the DefectDojo finding model.
   • This change ensures that the relevant information from the Trufflehog scan is properly imported and stored in the DefectDojo application, which is important for security analysts to investigate and remediate the identified security issues.

2. dojo/tools/trufflehog/parser.py:

   • The get_findings_v3 method has been modified to include the raw, unredacted version of the sensitive information detected by Trufflehog in the payload field of the Finding object.
   • This change raises security concerns, as the sensitive data could potentially be exposed if not properly secured and access-controlled. Recommendations have been provided to address these concerns.

Powered by DryRun Security

[p-l-]

That's a great idea. Thanks for proposing that change!

[Maffooch]

I am not seeing rawV2 used in any of the unit test files. Is this a new field? Would there be any concern for folks using older versions of truffle hog without rawV2 in their scan reports? In those cases, the only field that would be used for deduplication would be the file path

[brieucR]

@Maffooch I had a further look on that, another way to do this would be using url.

This field can be completed using Trufflehog v2 and Trufflehog v3.
Furthermore it's unique as it contains path, commit_hash and line_number.

We can also add the component_name to make sure we add no regression from before.

[brieucR]

With this update we make sure that any further change from Trufflehog or on the default description won't impact issue duplications

[brieucR]

As url was left empty until now, projects might have issues when upgrading DefectDojo.
@Maffooch is there a migration mechanism so that I can write a script to fill url field for existing findings?

[brieucR]

I've created a new PR here:
#10118

[mtesauro]

Closing since #10118 was opened

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[mtesauro]

Hmm, must have gotten distracted. Closing for real this time.