Handlers not getting called for some events

[hapatrick]

We have noticed that sometimes some of our handlers are not getting called when we expect them to. E.g. we have a handler on the status field of an object, the status field is updated but the handler never gets called. If we restart the operator, the handler is called as expected after the restart.

This is a rare event; most of the time things work as expected.

Has anyone else noticed this? It is certainly possible this is a problem with our code rather than kopf, but if so I don't see what we are doing wrong. Any suggestions on how to remedy this or what to look at to understand what's happening are appreciated.

[psontag]

We had similar problems. But after configuring server and client timeouts for the watch streams as mentioned in #698 it was fixed. Something like this

@kopf.on.startup()
def configure(settings: kopf.OperatorSettings, **_):
    settings.watching.server_timeout = 120
    settings.watching.client_timeout = 150

[hapatrick]

Thanks. It sounds like this is exactly what we need. We will try it out.

[alanbchristie]

I have a some cluster-wide kopf operators based on...

• kubernetes 19.15.0
• kopf 1.35.4

...that, after a as-yet-to-be-determined number of days, stop responding to any new events. Just like @hapatrick, if bounce the operator Pod the missing events are picked up and acted upon. I will try the timeouts suggested and report in a few weeks if the operator stability has improved.

[hapatrick]

What version of volcano is this with? Version 1.6.0 includes a fix that is supposed to address the original issue (volcano-sh/volcano#2169). Unfortunately we haven't had time to test this yet. Of course it's possible you're having some other problem.

[alanbchristie]

Sorry, this is not an issue with volcano - I have my own operator based on kopf and it's basically "falling asleep" after prolonged periods just as you described here. It feels like an issue with kopf rather than my operator - especially as it "falls asleep" without any warning or error. If it's fixed with some settings one could argue the basic setting are either wrong or there's a silently handled error that shouldn't be silent.

[hapatrick]

Sorry, I got confused about which github page I was on -- thought this was the issue I opened in volcano! Anyway, our problem ended up being with volcano, not kopf at all. I probably should have closed this.