Metrics performance improvements

[dogboat]

Description

This patch updates some of the metrics views (those handled by the dojo.metrics.views.metrics() view function -- the dashboard, findings, endpoints, and product type views) to use database aggregation instead of Python data wrangling to speed up page loads. Much of the support functionality was moved to a new module, dojo.metrics.utils.

Locally, page loads generally decreased from around mid-20s seconds to under a couple.

Aggregates are the same, though some display is changed -- the metrics charts include a bit more data, including the current month now, e.g. for Findings product type metrics on the same dataset:

Old (data up through 4/30):

New (data includes June statistics):

Data tables have been changed to include Info severity information (discrepancies in total counts between current/closed metrics and this patch can be attributed to this).

To support faster load times, The 'Detail Breakdown' table has been changed to only list the first 50 Finding details along with a count of total (if more than 50); pagination across the entire dataset can be added in a future patch.

Blake gave fantastic direction on this and deserves credit.

Test results

The following metrics pages were tested:

https://localhost/metrics?date=5&view=dashboard
https://localhost/metrics/product/type?type=Finding
https://localhost/metrics/product/type?type=Endpoint
https://localhost/metrics/product/type/$id (varied over $date)

They load faster and appear to be comparable, with the above noted changes, to the existing versions.

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Server-Side Request Forgery Analyzer	✅	0 findings
Configured Codepaths Analyzer	✅	0 findings
IDOR Analyzer	✅	0 findings
Sensitive Files Analyzer	✅	0 findings
SQL Injection Analyzer	✅	0 findings
Authn/Authz Analyzer	❕	2 findings
Secrets Analyzer	✅	0 findings

Note

🟢 Risk threshold not exceeded.

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

The code changes in this pull request are primarily focused on improving the data representation and visualization of the security metrics in the Defect Dojo application. The changes include updates to the Django templates, utility functions, and database queries used to fetch and process the metrics data.

The key changes include:

1. Improvements to the data accuracy and consistency in the dashboard, such as using the "epoch" property for the x-axis of charts and standardizing the severity labels.
2. Refactoring of the finding_queries and endpoint_queries functions into separate utility functions, which improves the overall code organization and maintainability.
3. Optimization of the database queries used to fetch the metrics data, reducing the number of queries and simplifying the data structures.
4. Enhancements to the user interface, including improved navigation, filtering, and data visualization features.

From an application security perspective, the changes do not introduce any obvious security vulnerabilities. The code focuses on improving the data presentation and processing, without any significant changes to the underlying security-sensitive functionality. However, it's important to review the entire codebase and ensure that proper security practices are followed throughout the application, such as input validation, secure coding, and adherence to best practices.

Files Changed:

1. dojo/templates/dojo/dashboard-metrics.html: The changes improve the data representation and visualization of the security metrics, ensuring accurate and consistent data presentation.
2. dojo/metrics/utils.py: The code defines the finding_queries and endpoint_queries functions, which handle the logic for generating metrics and analytics for findings and endpoints, respectively.
3. dojo/metrics/views.py: The changes refactor the finding_querys and endpoint_querys functions into separate utility functions, improving the overall code organization and maintainability.
4. dojo/templates/dojo/metrics.html: The changes enhance the user interface of the metrics dashboard, including improved navigation, filtering, and data visualization features.
5. unittests/test_metrics_queries.py: The changes optimize the database queries used to fetch the metrics data, reducing the number of queries and simplifying the data structures.

Powered by DryRun Security

[mtesauro]

Approved

[dogboat]

[sc-6399]