Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⭐ implement StoreResults for resources data #3096

Merged
merged 6 commits into from
Jan 25, 2024
Merged

Conversation

arlimus
Copy link
Member

@arlimus arlimus commented Jan 23, 2024

This extends the proto to store resources data directly. This is the next step in the data recordings journey that makes this data available to upstream targets. We are starting our journey in v10, so this is still behind a feature-flag for now.

  1. The recording interface moves to LLX, which also defines the runtime. This allows anyone to request the recording layer from the runtime and standardizes setters/getters.
  2. StoreResultsReq is extended to support resources recording data via the new resources field. To avoid confusion, the data field has been renamed to query_data, because it captures MQL query data only. The StoreQueryData method was adjusted analogously.
  3. The implementation of recordings stays with runtime for now, analogous to runtime itself having a proto/implementation split
  4. The feature-flag is StoreResourcesData. Only when it is active will scan results be sent upstream. We continue to not send shell and run data by default. I think we may want to consider sending these but may want to think about a CLI flag to explicitly send data to avoid unwanted side-effects. scan semantically already sends data up, so users expect it to behave this way.
  5. The new proto message ResourceRecording captures all recording data for each resource and was adopted from the previous recording implementation (where it was non-proto).

Follow-ups:

As this is only the start, there are 2 vital things before we can spread this more broadly. These have to be done in the v10 timeframe before the feature becomes default:

  1. Implement this analogously for cnspec
  2. Dedupe resources and query data. We currently duplicate query and recording data, which is annoying.

This extends the proto to store resources data directly. This is the next step in the data recordings journey that makes this data available to upstream targets. We are starting our journey in v10, so this is still behind a feature-flag for now.

Signed-off-by: Dominik Richter <[email protected]>
Signed-off-by: Dominik Richter <[email protected]>
Signed-off-by: Dominik Richter <[email protected]>
Copy link
Contributor

github-actions bot commented Jan 23, 2024

Test Results

742 tests  ±0   742 ✅ ±0   12s ⏱️ ±0s
 83 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 25a4108. ± Comparison against base commit b120ab4.

This pull request removes 3 and adds 3 tests. Note that renamed tests count towards both.
go.mondoo.com/cnquery/v10/llx ‑ TestSuccess/2024-01-21_23:21:04.710718129_+0000_UTC_m=+0.000878042
go.mondoo.com/cnquery/v10/llx ‑ TestTruthy/2024-01-21_23:21:04.710718129_+0000_UTC_m=+0.000878042
go.mondoo.com/cnquery/v10/llx ‑ TestTruthy/2024-01-21_23:21:04.710718129_+0000_UTC_m=+0.000878042#01
go.mondoo.com/cnquery/v10/llx ‑ TestSuccess/2024-01-23_09:09:21.809737613_+0000_UTC_m=+0.009860502
go.mondoo.com/cnquery/v10/llx ‑ TestTruthy/2024-01-23_09:09:21.809737613_+0000_UTC_m=+0.009860502
go.mondoo.com/cnquery/v10/llx ‑ TestTruthy/2024-01-23_09:09:21.809737613_+0000_UTC_m=+0.009860502#01

♻️ This comment has been updated with latest results.

Signed-off-by: Dominik Richter <[email protected]>
Signed-off-by: Dominik Richter <[email protected]>
Copy link
Member

@chris-rock chris-rock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice improvement @arlimus Just added one comment.

@@ -355,7 +355,8 @@ message UpdateAssetJobsReq {
// Store results for a given asset
message StoreResultsReq {
string asset_mrn = 1;
map<string, cnquery.llx.Result> data = 3;
map<string, cnquery.llx.Result> query_data = 3;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While there is no technical reason, I'd like to do avoid the rename for now. Can we keep the old name for the field and just extend the comments?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, happy to adjust it

Copy link
Member

@chris-rock chris-rock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @arlimus

@chris-rock chris-rock merged commit 83023b4 into main Jan 25, 2024
12 checks passed
@chris-rock chris-rock deleted the dom/data-collect branch January 25, 2024 07:16
@github-actions github-actions bot locked and limited conversation to collaborators Jan 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants