OCPBUGS-19690: Enable host network to access host sysctls #497

yuumasato · 2024-03-15T18:59:30Z

With HostNetwork: true the sysctl net.core.bpf_jit_harden becomes visible to the scanner container.
Below is a pod that has access to the sysctls:

apiVersion: v1
kind: Pod
metadata:
  name: list-sysctls
spec:
  hostNetwork: true
  volumes:
    - name: host
      hostPath:
        path: /
        type: Directory
  containers:
  - name: list
    command:
      - cat
      - /host/proc/sys/net/ipv6/conf/all/accept_ra
      - /host/proc/sys/net/core/bpf_jit_harden
    image: registry.access.redhat.com/ubi8/ubi-minimal
    securityContext:
      runAsUser: 0
      privileged: true
    volumeMounts:
    - name: host
      mountPath: /host

$ oc create -f list-syctls-proc.yaml
$ oc logs list-sysctls

DNSPolicy: ClusterFirstWithHostNet allows the CO to upload to resultserver, otherwise we get the following error:
{"level":"info","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Trying to upload to resultserver","url":"https://upstream-rhcos4-high-worker-rs:8443/"} {"level":"error","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Failed to upload results to server","error":"Post \"https://upstream-rhcos4-high-worker-rs:8443/\": dial tcp: lookup upstream-rhcos4-high-worker-rs on 10.0.0.2:53: no such host","stacktrace":"github.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:316\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:18\ngithub.com/cenkalti/backoff/v4.doRetryNotify[...]\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:88\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:61\ngithub.com/cenkalti/backoff/v4.RetryNotify\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:49\ngithub.com/cenkalti/backoff/v4.Retry\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:38\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:299\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.handleCompleteSCAPResults.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:390"}
Use the content from Re-enable runtime check on network related sysctls content#11722, to check whether the scanner container can access the sysctls correctly.
oc compliance bind -S default-auto-apply -N test profile/upstream-rhcos4-moderate

EDIT: I have re-tested and DNSPolicy: ClusterFirstWithHostNet indeed solves the no such host error when trying to upload to resultserver.

openshift-ci-robot · 2024-03-15T18:59:36Z

@yuumasato: This pull request references Jira Issue OCPBUGS-19690, which is invalid:

expected the bug to target either version "4.16." or "openshift-4.16.", but it targets "4.15.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

With HostNetwork: true the sysctl net.core.bpf_jit_harden becomes visible to the scanner container.
Below is a pod that has access to the sysctls:
apiVersion: v1
kind: Pod
metadata:
 name: list-sysctls
spec:
 hostNetwork: true
 volumes:
   - name: host
     hostPath:
       path: /
       type: Directory
 containers:
 - name: list
   command:
     - cat
     - /host/proc/sys/net/ipv6/conf/all/accept_ra
     - /host/proc/sys/net/core/bpf_jit_harden
   image: registry.access.redhat.com/ubi8/ubi-minimal
   securityContext:
     runAsUser: 0
     privileged: true
   volumeMounts:
   - name: host
     mountPath: /host
$ oc create -f list-syctls-proc.yaml
$ oc logs list-sysctls

But with HostNetwork: true, the CO fails to upload to resultserver.
{"level":"info","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Trying to upload to resultserver","url":"https://upstream-rhcos4-high-worker-rs:8443/"} {"level":"error","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Failed to upload results to server","error":"Post \"https://upstream-rhcos4-high-worker-rs:8443/\": dial tcp: lookup upstream-rhcos4-high-worker-rs on 10.0.0.2:53: no such host","stacktrace":"github.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:316\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:18\ngithub.com/cenkalti/backoff/v4.doRetryNotify[...]\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:88\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:61\ngithub.com/cenkalti/backoff/v4.RetryNotify\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:49\ngithub.com/cenkalti/backoff/v4.Retry\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:38\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:299\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.handleCompleteSCAPResults.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:390"}

DNSPolicy: ClusterFirstWithHostNet is my unsuccessful attempt to fix that.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Vincent056 · 2024-03-15T22:47:51Z

nice finding!

BhargaviGudi · 2024-03-19T05:55:29Z

/hold for test

BhargaviGudi · 2024-04-17T10:58:29Z

Verification passed with 4.16.0-0.nightly-2024-04-16-195622 + compliance-operator with PR #497 code + PR #11722 code

Install CO

$ oc get pb
NAME              CONTENTIMAGE                                 CONTENTFILE         STATUS
ocp4              ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     VALID
rhcos4            ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   VALID
upstream-ocp4     ghcr.io/complianceascode/k8scontent:11722    ssg-ocp4-ds.xml     VALID
upstream-rhcos4   ghcr.io/complianceascode/k8scontent:11722    ssg-rhcos4-ds.xml   VALID

Create custom wrscan
create auto-rem-ss to scan wrscan mcp rule only

$ oc get ss auto-rem-ss -oyaml
apiVersion: compliance.openshift.io/v1alpha1
autoApplyRemediations: true
autoUpdateRemediations: true
kind: ScanSetting
maxRetryOnTimeout: 3
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"compliance.openshift.io/v1alpha1","autoApplyRemediations":true,"autoUpdateRemediations":true,"kind":"ScanSetting","maxRetryOnTimeout":3,"metadata":{"annotations":{},"creationTimestamp":"2023-09-25T02:05:43Z","generation":1,"name":"auto-rem-ss","namespace":"openshift-compliance","resourceVersion":"43973","uid":"29426481-7cd1-48f0-a3cf-934c96f651eb"},"rawResultStorage":{"pvAccessModes":["ReadWriteOnce"],"rotation":5,"size":"2Gi"},"roles":["wrscan"],"scanTolerations":[{"operator":"Exists"}],"schedule":"0 1 * * *","showNotApplicable":false,"strictNodeScan":false,"timeout":"30m"}
  creationTimestamp: "2024-04-17T10:14:11Z"
  generation: 1
  name: auto-rem-ss
  namespace: openshift-compliance
  resourceVersion: "108142"
  uid: b3a50385-baad-43cf-8ac3-2fb3f1c502a6
rawResultStorage:
  pvAccessModes:
  - ReadWriteOnce
  rotation: 5
  size: 2Gi
roles:
- wrscan
scanTolerations:
- operator: Exists
schedule: 0 1 * * *
showNotApplicable: false
strictNodeScan: false
suspend: false
timeout: 30m

Create ssb

$ oc compliance bind -N rhcos4-high-test -S auto-rem-ss profile/upstream-rhcos4-high
Creating ScanSettingBinding rhcos4-high-test
$ oc get scan
NAME                 PHASE   RESULT
upstream-rhcos4-high-wrscan   DONE    NON-COMPLIANT

All the rules with auto-remediations are applied after 3 rounds are rescan.

$ oc compliance rerun-now scansettingbinding rhcos4-high-test
Rerunning scans from 'rhcos4-high-test': upstream-rhcos4-high-wrscan
Re-running scan 'openshift-compliance/upstream-rhcos4-high-wrscan'
$ oc get ccr -l compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-status=FAIL  
No resources found in openshift-compliance namespace.

BhargaviGudi · 2024-04-17T10:58:35Z

/unhold

BhargaviGudi · 2024-04-17T10:58:45Z

/label qe-approved

BhargaviGudi · 2024-04-17T10:59:02Z

/lgtm

yuumasato · 2024-04-19T16:15:31Z

@BhargaviGudi Thank you for testing this.

I re-tested again and cannot reproduce the error I had mentioned in PR description.
I was probably doing something wrong before.

yuumasato · 2024-04-19T16:19:59Z

Below are some of the runtime objects collected, they match the static configuration now.

<unix-sys:sysctl_item id="100008715" status="exists">
  <unix-sys:name>net.core.bpf_jit_harden</unix-sys:name>
  <unix-sys:value>2</unix-sys:value>
</unix-sys:sysctl_item>
<ind-sys:textfilecontent_item id="100008714" status="exists">
  <ind-sys:filepath>/etc/sysctl.d/75-sysctl_net_core_bpf_jit_harden.conf</ind-sys:filepath>
  <ind-sys:path>/etc/sysctl.d</ind-sys:path>
  <ind-sys:filename>75-sysctl_net_core_bpf_jit_harden.conf</ind-sys:filename>
  <ind-sys:pattern>^[\s]*net.core.bpf_jit_harden[\s]*=[\s]*(.*)[\s]*$</ind-sys:pattern>
  <ind-sys:instance datatype="int">1</ind-sys:instance>
  <ind-sys:line>^[\s]*net.core.bpf_jit_harden[\s]*=[\s]*(.*)[\s]*$</ind-sys:line>
  <ind-sys:text>net.core.bpf_jit_harden=2</ind-sys:text>
  <ind-sys:subexpression>2</ind-sys:subexpression>
</ind-sys:textfilecontent_item>

<unix-sys:sysctl_item id="100008621" status="exists">
  <unix-sys:name>net.ipv6.conf.default.accept_ra</unix-sys:name>
  <unix-sys:value>0</unix-sys:value>
</unix-sys:sysctl_item>
<ind-sys:textfilecontent_item id="100008620" status="exists">
  <ind-sys:filepath>/etc/sysctl.d/75-sysctl_net_ipv6_conf_default_accept_ra.conf</ind-sys:filepath>
  <ind-sys:path>/etc/sysctl.d</ind-sys:path>
  <ind-sys:filename>75-sysctl_net_ipv6_conf_default_accept_ra.conf</ind-sys:filename>
  <ind-sys:pattern>^[\s]*net.ipv6.conf.default.accept_ra[\s]*=[\s]*(.*)[\s]*$</ind-sys:pattern>
  <ind-sys:instance datatype="int">1</ind-sys:instance>
  <ind-sys:line>^[\s]*net.ipv6.conf.default.accept_ra[\s]*=[\s]*(.*)[\s]*$</ind-sys:line>
  <ind-sys:text>net.ipv6.conf.default.accept_ra=0</ind-sys:text>
  <ind-sys:subexpression>0</ind-sys:subexpression>
</ind-sys:textfilecontent_item>

Vincent056

/lgtm

openshift-ci · 2024-06-20T15:56:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BhargaviGudi, Vincent056, yuumasato

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [BhargaviGudi,Vincent056]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yuumasato · 2024-06-20T16:16:55Z

/jira refresh

openshift-ci-robot · 2024-06-20T16:16:59Z

@yuumasato: This pull request references Jira Issue OCPBUGS-19690, which is invalid:

expected the bug to target either version "4.17." or "openshift-4.17.", but it targets "4.16.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

yuumasato · 2024-06-20T16:34:32Z

/jira refresh

openshift-ci-robot · 2024-06-20T16:34:39Z

@yuumasato: This pull request references Jira Issue OCPBUGS-19690, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.17.0) matches configured target version for branch (4.17.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @xiaojiey

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rhmdnd · 2024-06-25T16:46:57Z

The ROSA failure here looks like a provisioning/setup issue before the test even runs. Attempting to recheck since I'm not convinced the failure is due to this patch.

rhmdnd · 2024-06-25T16:47:06Z

/test e2e-rosa

'hostNetwork: true' grants access to the host's sysctl configurations. 'dnsPolicy: ClusterFirstWithHostnet' is required to access services.

yuumasato · 2024-06-26T08:11:04Z

Rebased to latest master, lets see how testing goes.

github-actions · 2024-06-26T09:47:01Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:497

Vincent056 · 2024-06-28T15:16:54Z

/lgtm

openshift-ci-robot · 2024-06-28T15:21:44Z

@yuumasato: Jira Issue OCPBUGS-19690: All pull requests linked via external trackers have merged:

ComplianceAsCode/compliance-operator#497

Jira Issue OCPBUGS-19690 has been moved to the MODIFIED state.

In response to this:

With HostNetwork: true the sysctl net.core.bpf_jit_harden becomes visible to the scanner container.
Below is a pod that has access to the sysctls:
apiVersion: v1
kind: Pod
metadata:
 name: list-sysctls
spec:
 hostNetwork: true
 volumes:
   - name: host
     hostPath:
       path: /
       type: Directory
 containers:
 - name: list
   command:
     - cat
     - /host/proc/sys/net/ipv6/conf/all/accept_ra
     - /host/proc/sys/net/core/bpf_jit_harden
   image: registry.access.redhat.com/ubi8/ubi-minimal
   securityContext:
     runAsUser: 0
     privileged: true
   volumeMounts:
   - name: host
     mountPath: /host
$ oc create -f list-syctls-proc.yaml
$ oc logs list-sysctls

DNSPolicy: ClusterFirstWithHostNet allows the CO to upload to resultserver, otherwise we get the following error:
{"level":"info","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Trying to upload to resultserver","url":"https://upstream-rhcos4-high-worker-rs:8443/"} {"level":"error","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Failed to upload results to server","error":"Post \"https://upstream-rhcos4-high-worker-rs:8443/\": dial tcp: lookup upstream-rhcos4-high-worker-rs on 10.0.0.2:53: no such host","stacktrace":"github.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:316\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:18\ngithub.com/cenkalti/backoff/v4.doRetryNotify[...]\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:88\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:61\ngithub.com/cenkalti/backoff/v4.RetryNotify\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:49\ngithub.com/cenkalti/backoff/v4.Retry\n\tgithub.com/cenkalti/backoff/[email protected]/retry.go:38\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:299\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.handleCompleteSCAPResults.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:390"}

Use the content from Re-enable runtime check on network related sysctls content#11722, to check whether the scanner container can access the sysctls correctly.
oc compliance bind -S default-auto-apply -N test profile/upstream-rhcos4-moderate

EDIT: I have re-tested and DNSPolicy: ClusterFirstWithHostNet indeed solves the no such host error when trying to upload to resultserver.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot added the jira/valid-reference label Mar 15, 2024

openshift-ci bot requested review from BhargaviGudi and mrogers950 March 15, 2024 18:59

openshift-ci-robot added the jira/invalid-bug label Mar 15, 2024

yuumasato mentioned this pull request Mar 15, 2024

Re-enable runtime check on network related sysctls ComplianceAsCode/content#11722

Merged

openshift-ci bot added the do-not-merge/hold label Mar 19, 2024

openshift-ci bot removed the do-not-merge/hold label Apr 17, 2024

openshift-ci bot added the qe-approved label Apr 17, 2024

openshift-ci bot assigned BhargaviGudi Apr 17, 2024

openshift-ci bot added the lgtm label Apr 17, 2024

BhargaviGudi approved these changes Apr 18, 2024

View reviewed changes

openshift-ci bot added the approved label Apr 18, 2024

yuumasato requested review from Vincent056 and rhmdnd and removed request for mrogers950 April 19, 2024 16:20

Vincent056 approved these changes Jun 20, 2024

View reviewed changes

openshift-ci bot assigned Vincent056 Jun 20, 2024

openshift-ci-robot added the jira/severity-important label Jun 20, 2024

openshift-ci-robot added jira/valid-bug and removed jira/invalid-bug labels Jun 20, 2024

openshift-ci bot requested a review from xiaojiey June 20, 2024 16:34

Use the host's network namespace for scans

f05e870

'hostNetwork: true' grants access to the host's sysctl configurations. 'dnsPolicy: ClusterFirstWithHostnet' is required to access services.

yuumasato force-pushed the enable_host_network_for_sysctls branch from a10228d to f05e870 Compare June 26, 2024 08:10

openshift-ci bot removed the lgtm label Jun 26, 2024

openshift-ci bot added the lgtm label Jun 28, 2024

openshift-merge-bot bot merged commit f3e5a91 into ComplianceAsCode:master Jun 28, 2024
14 checks passed

yuumasato deleted the enable_host_network_for_sysctls branch June 28, 2024 15:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-19690: Enable host network to access host sysctls #497

OCPBUGS-19690: Enable host network to access host sysctls #497

yuumasato commented Mar 15, 2024 •

edited

Loading

openshift-ci-robot commented Mar 15, 2024

Vincent056 commented Mar 15, 2024

BhargaviGudi commented Mar 19, 2024

BhargaviGudi commented Apr 17, 2024 •

edited by xiaojiey

Loading

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

yuumasato commented Apr 19, 2024

yuumasato commented Apr 19, 2024

Vincent056 left a comment

openshift-ci bot commented Jun 20, 2024

yuumasato commented Jun 20, 2024

openshift-ci-robot commented Jun 20, 2024

yuumasato commented Jun 20, 2024

openshift-ci-robot commented Jun 20, 2024

rhmdnd commented Jun 25, 2024

rhmdnd commented Jun 25, 2024

yuumasato commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

Vincent056 commented Jun 28, 2024

openshift-ci-robot commented Jun 28, 2024

OCPBUGS-19690: Enable host network to access host sysctls #497

OCPBUGS-19690: Enable host network to access host sysctls #497

Conversation

yuumasato commented Mar 15, 2024 • edited Loading

openshift-ci-robot commented Mar 15, 2024

Vincent056 commented Mar 15, 2024

BhargaviGudi commented Mar 19, 2024

BhargaviGudi commented Apr 17, 2024 • edited by xiaojiey Loading

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

yuumasato commented Apr 19, 2024

yuumasato commented Apr 19, 2024

Vincent056 left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jun 20, 2024

yuumasato commented Jun 20, 2024

openshift-ci-robot commented Jun 20, 2024

yuumasato commented Jun 20, 2024

openshift-ci-robot commented Jun 20, 2024

rhmdnd commented Jun 25, 2024

rhmdnd commented Jun 25, 2024

yuumasato commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

Vincent056 commented Jun 28, 2024

openshift-ci-robot commented Jun 28, 2024

yuumasato commented Mar 15, 2024 •

edited

Loading

BhargaviGudi commented Apr 17, 2024 •

edited by xiaojiey

Loading