Unable to connect to Grafana IP after reboot

[wheeler01]

After rebooting my Pi after making some changes I have been unable to access Grafana after executing main.yml post reboot. I am able to access PiHole. When running main.yml No errors are thrown, everything is either ok or skipped. main.yml and anything ansible was edited. I am unsure what is going on since we aren't getting any errors and everything seems to be running well. Any assistance would be appreciated.

[martinbrose]

Hi @wheeler01 ,

a few questions in order to help you:

1. You are stating you made some changes. What changes exactly are you referring to? What do you want to achieve?

2. Has everything been running ok before you made the changes?

3. Could you please share your current main.yml?

4. Could you please run docker ps and share the output? This will establish if your grafana docker is running and on which port it is listening.

Thanks

[wheeler01]

1. The changes I referred to aren't relevant to internet pi. The changes are relevant to homebridge which is also installed on the pi I have a default installation of internet pi except changing configuration variables to match my configuration.

2. After first installing internet pi everything ran fine. I had to reboot the pi due to some software updates. Pi hole is still runnin, homebridge is still running, internet pi appears to be running but the web interface is not available.

3. My main.yml is attached.
   main_yml.txt

4. Docker ps is attached. docker ps output.txt

Thank you.

[martinbrose]

Hi @wheeler01,

had a very quick look at your docker ps output and it looks like your Grafana docker container has a problem. Possibly constantly restarting...

Could you do a docker logs container_id, changing the container_id with the container id of your Grafana docker container?

And also, which Raspberry Pi version are you using? You might need to specify a previous version. I believe they sometimes miss or drop a specific architecture. So that could also be the problem.

Thanks

[wheeler01]

When running docker run grafana/grafana it attempts to start but errors out throwing this:
logger=grafana.update.checker t=2024-05-13T15:01:20.556342358Z level=error msg="Update check failed" error="failed to get stable version from grafana.com: Get "https://grafana.com/api/grafana/versions/stable\": dial tcp: lookup grafana.com: i/o timeout" duration=10.00194632s

When running docker logs it returns a wall of this error:
Error: ✗ Get "https://grafana.com/api/plugins/flant-statusmap-panel/versions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[martinbrose]

Looks like a weird grafana upgrade problem.

I would recommend you:

1. Stop the internet-monitoring with docker-compose down
2. Remove the grafana container
3. Prune the old grafana container
4. Start the internet-monitoring with docker-compose up, which should download grafana again and set it up from scratch without any upgrade.

I hope that helps...

[wheeler01]

I am not extremely familiar with docker. I do not know how to do steps #2 and #3. Can you offer any advice or assistance?

[martinbrose]

Sure, no problem:

2. Remove the container with docker rm container_id, replacing the container_id with the one from the Grafana container

3. Prune the container and volumes with docker system prune --volumes

[wheeler01]

Turns out the problem was actually Pi Hole. Pie hole was blocking grafana. As soon as I whitelisted in Pi hole everything started working.

[martinbrose]

Hi @wheeler01,

thanks for sharing the solution.
Tbh, this didn't even cross my mind. ;)

Glad it's solved! Enjoy!

[martinbrose]

Maybe one last bit: would you mind sharing the Grafana host that was blocked? It just seems so strange...

[quadespresso]

@martinbrose according to this update it looks like it couldn't resolve grafana.com, ie, dial tcp: lookup grafana.com: i/o timeout.

As the classic haiku goes:

It's not DNS
There's no way it's DNS
It was DNS

🙂

[Terrabytee]

Turns out the problem was actually Pi Hole. Pie hole was blocking grafana. As soon as I whitelisted in Pi hole everything started working.

@wheeler01 What domains did you add to your Pi-Hole whitelist? I seem to be having this same error and I cannot figure out how to fix it.

[aeburnett]

I would also like to know. I am having the same issue.

[Terrabytee]

I would also like to know. I am having the same issue.

@aeburnett Just located this through Google, and it resolved (no pun intended) my issue.
#349 (comment)

[SecOpsRJennings]

I am having very similar problems. I am running an rpi4b as a network device using rpiOS to host the internet-pi app. The network router provides a static IP address of 192.168.10.3 to the internet-pi and also provides the same address to the network as the DNS IP address.

My Grafana dashboard doesn't start, because of the "https://grafana.com/api/plugins/flant-statusmap-panel/versions" issue. This leads me to believe that the problem is DNS access for Grafana.

I see a solution in issue #8, setting the DNS of the device to 127.0.0.1 but I don't know how to do that because my /etc/resolv.conf always declares the nameserver to be 192.168.10.3, from the router.

I found a Stackoverflow solution that says to prepend the ports declared in the docker-compose.yml file with the IP address, but this takes me into the ansible and docker scripts and I have no experience there.

Can someone help me out here?

Solved: I have found how to stop NetworkManager from updating the /etc/resolv.conf file. I then rebuilt pi-hole.
It seems to work now, even after reboots.