Support running as non-root on OpenShift/CRI-O and log collection

[jvoravong]

Is your feature request related to a problem? Please describe.

Description:

There is a log collection permissions issue when deploying on OpenShift/CRI-O and trying to run the agent as a non-root user. Last tested on OpnShift version v4.13 with both otel or fluentd as the logging engine. This problem is not present when Docker and containerd are the runtime.

• We use a custom security context constraint on OpenShift with RunAsUser set to RunAsAny. Documentation on SCC indicates that this configuration should be feasible.
• The root issue arises from the chart’s inability to consistently set the log data file permissions so the agent container can read the log data.
• The security context constraint used likely has to be updated. The daemonset for the agent may also need to be updated.

Error:

Error: cannot start pipelines: storage client: open /var/addon/splunk/otel_pos/receiver_filelog_: permission denied
2023/08/15 15:31:28 main.go:94: application run finished with error: cannot start pipelines: storage client: open /var/addon/splunk/otel_pos/receiver_filelog_: permission denied

Describe the solution you'd like

Make the necessary updates to the involved security context constraints or agent daemonset so that OpenShift/CRI-O users can collect logs while also running the agent in non-root user mode.

Describe alternatives you've considered

No response

Additional context

How To Reproduce:

  distribution: openshift
  logsEngine: otel
  agent:
    securityContext:
      runAsUser: 20000
      runAsGroup: 20000

  distribution: openshift
  logsEngine: fluentd
  fluentd:
    securityContext:
      runAsUser: 20000
      runAsGroup: 20000

Links:

• https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/templates/securityContextConstraints.yaml
• https://docs.openshift.com/container-platform/4.13/authentication/managing-security-context-constraints.html
• https://developers.redhat.com/blog/2020/10/26/adapting-docker-and-kubernetes-containers-to-run-on-red-hat-openshift-container-platform
• DOC Update: Log collection and running the container in non-root user mode #887

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. If this issue is still relevant, please leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

Moving to Splunk ideas. Use https://ideas.splunk.com/ideas/SFXIMMID-I-470 to follow this development moving forward.