-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
⭐ implement StoreResults for resources data #3096
Conversation
This extends the proto to store resources data directly. This is the next step in the data recordings journey that makes this data available to upstream targets. We are starting our journey in v10, so this is still behind a feature-flag for now. Signed-off-by: Dominik Richter <[email protected]>
Signed-off-by: Dominik Richter <[email protected]>
Signed-off-by: Dominik Richter <[email protected]>
Test Results742 tests ±0 742 ✅ ±0 12s ⏱️ ±0s Results for commit 25a4108. ± Comparison against base commit b120ab4. This pull request removes 3 and adds 3 tests. Note that renamed tests count towards both.
♻️ This comment has been updated with latest results. |
Signed-off-by: Dominik Richter <[email protected]>
Signed-off-by: Dominik Richter <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice improvement @arlimus Just added one comment.
explorer/cnquery_explorer.proto
Outdated
@@ -355,7 +355,8 @@ message UpdateAssetJobsReq { | |||
// Store results for a given asset | |||
message StoreResultsReq { | |||
string asset_mrn = 1; | |||
map<string, cnquery.llx.Result> data = 3; | |||
map<string, cnquery.llx.Result> query_data = 3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While there is no technical reason, I'd like to do avoid the rename for now. Can we keep the old name for the field and just extend the comments?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, happy to adjust it
based on https://github.com/mondoohq/cnquery/pull/3096\#discussion_r1465430644 Signed-off-by: Dominik Richter <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @arlimus
This extends the proto to store resources data directly. This is the next step in the data recordings journey that makes this data available to upstream targets. We are starting our journey in v10, so this is still behind a feature-flag for now.
StoreResultsReq
is extended to support resources recording data via the newresources
field. To avoid confusion, thedata
field has been renamed toquery_data
, because it captures MQL query data only. TheStoreQueryData
method was adjusted analogously.StoreResourcesData
. Only when it is active will scan results be sent upstream. We continue to not sendshell
andrun
data by default. I think we may want to consider sending these but may want to think about a CLI flag to explicitly send data to avoid unwanted side-effects.scan
semantically already sends data up, so users expect it to behave this way.ResourceRecording
captures all recording data for each resource and was adopted from the previous recording implementation (where it was non-proto).Follow-ups:
As this is only the start, there are 2 vital things before we can spread this more broadly. These have to be done in the v10 timeframe before the feature becomes default: