🐛 fix github discovery for v9 #1655

vjeffrey · 2023-09-07T00:20:29Z

czunker

Thanks for fixing this @vjeffrey

providers/github/resources/discovery.go

providers/github/resources/github_org.go

czunker · 2023-09-07T04:30:56Z

I gave it a try:

cnquery run github org mondoohq --discover repos -c "github.repository.license.spdxId" --verbose                           ✔ │ 06:20:34 
DBG using provider github with connector github
DBG no need to update provider last-refresh=58.448175428s provider=github
DBG running provider plugin path=/home/christian/.config/mondoo/providers/github/github
→ loaded configuration from /etc/opt/mondoo/mondoo.yml using source default
DBG resolved 27 assets
DBG starting query execution qrid=Xiag1Adi3l0=
DBG finished query execution qrid=Xiag1Adi3l0=
DBG rmN/PU9BjTuMDKP3oZQ+j/kU/Cfvkp43OHrhOb+XrZl8zyhYpzNgNYLj9DnlWxhtCnFZPFGJU0gBGky8CvE4lw== finished
DBG graph has received all datapoints
rpc error: code = Unknown desc = no user provided
github.repository.license.spdxId: no data available
DBG starting query execution qrid=Xiag1Adi3l0=
DBG rmN/PU9BjTuMDKP3oZQ+j/kU/Cfvkp43OHrhOb+XrZl8zyhYpzNgNYLj9DnlWxhtCnFZPFGJU0gBGky8CvE4lw== finished
DBG graph has received all datapoints
DBG finished query execution qrid=Xiag1Adi3l0=
rpc error: code = Unknown desc = no user provided
....

To me, it looks like the discovery of the repos is working. But then some data is missing to actually create the repo resource.

vjeffrey · 2023-09-07T06:25:14Z

if i try to run the command you tried in v8, i also error --- i think it needs to be:

go run apps/cnquery/cnquery.go run github org mondoohq  -c "github.organization.repositories{license.spdxId }" --verbose

czunker · 2023-09-07T10:24:22Z

if i try to run the command you tried in v8, i also error --- i think it needs to be:
go run apps/cnquery/cnquery.go run github org mondoohq  -c "github.organization.repositories{license.spdxId }" --verbose

I did a bit more testing. My first command was wrong, but what I tried to achieve, works in v8 in case you use the correct parameters:

cnquery run github org mondoohq --discover repository -c "github.repository.license.spdxId"                                                                       
→ loaded configuration from /etc/opt/mondoo/mondoo.yml using source default
→ discover related assets for 1 asset(s)
→ resolved assets resolved-assets=26
Query encountered errors:
cannot cast resource to resource type: <nil>
github.repository.license.spdxId: no data available
...
github.repository.license.spdxId: "Apache-2.0"
github.repository.license.spdxId: "MPL-2.0"
github.repository.license.spdxId: "MPL-2.0"

...

In my first try, I used the wrong discovery option: --discover repo instead of --discover repository

But using the correct discovery in v9, does not show the same result as in v8.

providers/github/provider/provider.go

czunker · 2023-09-07T10:28:52Z

Also the error message in my first try was misleading. I opened a PR to fix this: #1657

vjeffrey · 2023-09-12T21:15:20Z

alright, this is fixed up now,
cnquery shell github org mondoohq --discover all
cnquery shell github org mondoohq --discover repos,users

vjeffrey · 2023-09-12T21:16:01Z

hmm, actually i have the platform ids, they look a little off

czunker · 2023-09-13T05:09:26Z

alright, this is fixed up now, cnquery shell github org mondoohq --discover all cnquery shell github org mondoohq --discover repos,users

The discovery options are different for v8:

cnquery run github org mondoohq --discover repository -c "asset{ ids }" 
→ loaded configuration from /etc/opt/mondoo/mondoo.yml using source default
→ discover related assets for 1 asset(s)
→ resolved assets resolved-assets=26
asset: {
  ids: [
    0: "//platformid.api.mondoo.app/runtime/github/owner/mondoohq/repository/installer"
  ]
}
...

v9:

cnquery run github org mondoohq --discover repos -c "asset{ ids }" 
→ loaded configuration from /etc/opt/mondoo/mondoo.yml using source default
asset: {
  ids: [
    0: "//platformid.api.mondoo.app/runtime/github/organization/Mondoo Inc"
  ]
}
asset: {
  ids: [
    0: "//platformid.api.mondoo.app/runtime/github/owner/Mondoo Inc/repository/installer"
  ]
}

And as already mentioned the platformIDs: "Mondoo Inc" vs. "mondoohq"

czunker · 2023-09-13T10:00:11Z

providers/github/resources/discovery.go

+			})
+		}
+	}
+	if stringx.Contains(targets, connection.DiscoveryUsers) {


Nice! That's actually a new feature. We don't have this in v8.

czunker · 2023-09-13T11:08:58Z

providers/github/resources/github_repo.go

 		// TODO: figure out if the client has the permission to fetch the protection rules
-		return nil, nil
+		return nil, err


@vjeffrey I'd like your opinion on this one.

When I keep it as is, this panics:

cnquery run github repo czunker/go-vex -c "github.repository{ name branches[0]{*} commits[0]{*} }" ! using builtin provider for github → loaded configuration from /etc/opt/mondoo/mondoo.yml using source default ! using builtin provider for github panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1180360] goroutine 14 [running]: go.mondoo.com/cnquery/providers/github/resources.(*mqlGithubBranchprotection).MqlID(0x2f0000c000cd04ea?) /home/christian/workspace/mondoo/github.com/cnquery/providers/github/resources/github.lr.go:3937 go.mondoo.com/cnquery/llx.resource2result({0x22861a0, 0x0}, {0xc000fea348, 0x18}) /home/christian/workspace/mondoo/github.com/cnquery/llx/data_conversions.go:313 +0x52

When I only create a resource with an ID, it prevents the panic, but I get "cannot convert" error.

In v8, this simply displayed as "null":

cnquery run github repo czunker/go-vex -c "github.repository{ name branches[0]{ name protectionRules {*} } commits[0]{ sha commit {*}} }" ... github.repository: { ... name: "go-vex" branches[0]: { protectionRules: null name: "christian/extend_csaf_advisories" } }

This change works, but is quite different from v8. Also, this might apply to other resources.

i'll check that out, i think i ran into that before on aws

this looks like it's an issue not limited to this resource -- returning nil in these types of situations isn't behaving the way we expect it to - so we can probably move this pr forward and fix that in a followup

I rebased this on main, as #1726 was merged.

I changed the field accordingly, and now it looks good:

nquery run github repo czunker/go-vex -c "github.repository{ name branches[0]{ name protectionRules {*} }}" --verbose DBG using provider github with connector github DBG no need to update provider last-refresh=29.146104804s provider=github DBG running provider plugin path=/home/christian/.config/mondoo/providers/github/github → loaded configuration from /etc/opt/mondoo/mondoo.yml using source default DBG resolved 1 assets ... github.repository: { name: "go-vex" branches[0]: { protectionRules: null name: "christian/extend_csaf_advisories" } }

awesome! thank you @arlimus for that fix!

Signed-off-by: Christian Zunker <[email protected]>

czunker · 2023-09-14T13:22:13Z

I gave some other fields a try where we also return nil, nil, but those seem fine, e.g.:

cnquery run github repo czunker/backend -c "github.repository.openMergeRequests{*}"
...
github.repository.openMergeRequests: []

@vjeffrey I think this PR is done for now, or do you see any open topics?

czunker

LGTM. @vjeffrey should have another look as I also did part of the commits.

vjeffrey · 2023-09-14T14:20:27Z

just tested again, this is working great, lets get it in!

vjeffrey force-pushed the vj/github-fixes-v9 branch 2 times, most recently from 393c670 to 018fc64 Compare September 7, 2023 02:24

czunker reviewed Sep 7, 2023

View reviewed changes

providers/github/resources/discovery.go Outdated Show resolved Hide resolved

providers/github/resources/github_org.go Outdated Show resolved Hide resolved

vjeffrey force-pushed the vj/github-fixes-v9 branch from 018fc64 to 17c7b32 Compare September 7, 2023 06:22

vjeffrey force-pushed the vj/github-fixes-v9 branch from 17c7b32 to 06cf4ca Compare September 7, 2023 06:26

czunker reviewed Sep 7, 2023

View reviewed changes

providers/github/provider/provider.go Outdated Show resolved Hide resolved

vjeffrey force-pushed the vj/github-fixes-v9 branch from 06cf4ca to 2714913 Compare September 12, 2023 21:09

vjeffrey force-pushed the vj/github-fixes-v9 branch from 7b18f8f to bc61623 Compare September 13, 2023 05:46

czunker reviewed Sep 13, 2023

View reviewed changes

vjeffrey and others added 8 commits September 14, 2023 09:13

🐛 fix github discovery for v9; fix a few resource issues

5919cc0

fixes

9c1ad8a

more fixes

4a4ad34

support repository and repos for discovery

0bca981

fix platform ids for orgs and users

041b219

discovery and platformID fixes

4e38fbf

Signed-off-by: Christian Zunker <[email protected]>

🧹 return nil when no branch protection found

db6bc0e

Fix null protectionrules

0d12aa8

Signed-off-by: Christian Zunker <[email protected]>

czunker force-pushed the vj/github-fixes-v9 branch from 517017a to 0d12aa8 Compare September 14, 2023 13:11

czunker approved these changes Sep 14, 2023

View reviewed changes

vjeffrey merged commit 92834a4 into main Sep 14, 2023
10 checks passed

vjeffrey deleted the vj/github-fixes-v9 branch September 14, 2023 14:20

github-actions bot locked and limited conversation to collaborators Sep 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 fix github discovery for v9 #1655

🐛 fix github discovery for v9 #1655

vjeffrey commented Sep 7, 2023 •

edited

Loading

czunker left a comment

czunker commented Sep 7, 2023

vjeffrey commented Sep 7, 2023

czunker commented Sep 7, 2023

czunker commented Sep 7, 2023

vjeffrey commented Sep 12, 2023

vjeffrey commented Sep 12, 2023

czunker commented Sep 13, 2023

czunker Sep 13, 2023

czunker Sep 13, 2023

vjeffrey Sep 13, 2023

vjeffrey Sep 13, 2023

czunker Sep 14, 2023

vjeffrey Sep 14, 2023

czunker commented Sep 14, 2023

czunker left a comment

vjeffrey commented Sep 14, 2023

🐛 fix github discovery for v9 #1655

🐛 fix github discovery for v9 #1655

Conversation

vjeffrey commented Sep 7, 2023 • edited Loading

czunker left a comment

Choose a reason for hiding this comment

czunker commented Sep 7, 2023

vjeffrey commented Sep 7, 2023

czunker commented Sep 7, 2023

czunker commented Sep 7, 2023

vjeffrey commented Sep 12, 2023

vjeffrey commented Sep 12, 2023

czunker commented Sep 13, 2023

czunker Sep 13, 2023

Choose a reason for hiding this comment

czunker Sep 13, 2023

Choose a reason for hiding this comment

vjeffrey Sep 13, 2023

Choose a reason for hiding this comment

vjeffrey Sep 13, 2023

Choose a reason for hiding this comment

czunker Sep 14, 2023

Choose a reason for hiding this comment

vjeffrey Sep 14, 2023

Choose a reason for hiding this comment

czunker commented Sep 14, 2023

czunker left a comment

Choose a reason for hiding this comment

vjeffrey commented Sep 14, 2023

vjeffrey commented Sep 7, 2023 •

edited

Loading