node-problem-detector not able to detect kernel log events for a Kind cluster

[pravarag]

I've been trying to run node-problem-detector on a local kind cluster with 3 nodes (1 master, 2 worker). And after installing it as DaemonSet, firstly I'm seeing there are three pods running across three nodes including master. And also, when I pass any Kernel message as test, I don't see any events getting generated either in npd pod nor in the node's description.

[wangzhen127]

You may need to tune your daemonset yaml

• Update the node selector to ignore master, or remove the node-problem-detector label on master.

  node-problem-detector/deployment/node-problem-detector.yaml

  Line 11 in 13b65d0

  app: node-problem-detector

• For watching kernel messages, it depends on how your NPD is configured in the daemonset.

  node-problem-detector/deployment/node-problem-detector.yaml

  Line 31 in 13b65d0

  - --config.system-log-monitor=/config/kernel-monitor.json,/config/docker-monitor.json

[BenTheElder]

Note: kind clusters are sharing the host kernel with sketchy isolation.

What's the use case for NPD-on-kind?

[cmontemuino]

Note: kind clusters are sharing the host kernel with sketchy isolation.

What's the use case for NPD-on-kind?

It's local testing and CI in my case.

[BenTheElder]

For testing NPD a fake should be used or a remote VM, we shouldn't introduce issues into the CI host's kernel and if we don't then we won't see any?

for local development, you could use a VM or local-up-cluster.sh or kubeadm init

kind is generally attempting to create a container that appears like a node, but it's on a shared kernel, in a container, which kubelet doesn't clearly support.

in general kind works best for testing API interactions and node to node interactions but not kernel / host / resource limits for now unfortunately

[cmontemuino]

Just in case it helps other people, the following configuration works pretty well with my KinD installation:

--config.system-log-monitor=/config/kernel-monitor.json,/config/systemd-monitor.json \
--config.custom-plugin-monitor=/config/iptables-mode-monitor.json,/config/network-problem-monitor.json,/config/kernel-monitor-counter.json,/config/systemd-monitor-counter.json

That helped me to quickly understand what's going on behind the scenes, and then deploy node-problem-detector in our clusters.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[proto-h]

There is a bug in handling time for kmsg around this check https://github.com/kubernetes/node-problem-detector/blob/v0.8.20/pkg/systemlogmonitor/logwatchers/kmsg/log_watcher_linux.go#L111 - this check simply does not work correctly probably due a bug in the euank/go-kmsg-parser.