mybinder.org is down and returning ERROR 500: Internal Server Error.

[rgaiacs]

$ wget https://mybinder.org/
--2023-10-31 13:04:58--  https://mybinder.org/
Resolving mybinder.org (mybinder.org)... 35.239.125.45
Connecting to mybinder.org (mybinder.org)|35.239.125.45|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-10-31 13:05:11 ERROR 500: Internal Server Error.

[manics]

Just taken a quick look at the prod-redirector logs, there are several
tornado.curl_httpclient.CurlError: HTTP 599: Could not resolve host: ovh2.mybinder.org
errors 🙁

[manics]

And several
gesis health check failed, attempt 2 of 5: HTTP 599: Resolving timed out after 5001 milliseconds
errors too. OVH2 and Gesis are accessible, so maybe there's a DNS or networking problem in the GKE cluster.

[manics]

The kube-dns dnsmasq container is crashing

[manics]

@yuvipanda is it possible the updated cryptnono is killing something it shouldn't?

[yuvipanda]

@manics I've access to the cluster now, looking

[yuvipanda]

@manics yep that's what's going on. I temporarily deleted the daemonset, should bring it back up.

[yuvipanda]

@manics I had looked at how this interacted with minesweeper but not ban.py!

[yuvipanda]

@manics #2796 turns it off everywhere.

[yuvipanda]

You can see the cause here: https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.pod_name%3D%22prod-cryptnono-r9p8c%22;cursorTimestamp=2023-10-31T13:06:27.123120038Z;duration=PT1H?referrer=search&project=binderhub-288415. I don't want to disclose that on the public github repo though.

[yuvipanda]

I also thought we no longer had anything running on GKE, so I only checked OVH2 after deployment :( I guess the redirector is actually still on GKE!

[rgaiacs]

@yuvipanda #2796 didn't resolved the issue. mybinder.org is still down. Thanks for the help!

[yuvipanda]

@rgaiacs hmm, the deploy seems to have brought the execwhacker back. I've manually killed it again and mybinder.org is back.

I'll take a closer look soon.

[yuvipanda]

This was fixed.