Segmentation fault (core dumped) running nagios

[gurubobnz]

Hi,

Our nagios installation using docker-nagios has been stable for some time. A week ago it started looping on startup, and I found that nagios itself was segfaulting during startup. It seemed to resolve itself, and now it has happened again.

I have dug into the /etc/service/nagios/run script, and when I run the run script manually I get this:

root@df50e7d4e2b6:/etc/service/nagios# ./run 
checking permissions for nagios & nagiosgraph

Nagios Core 4.4.7
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2022-04-14
License: GPL

Website: https://www.nagios.org
Nagios 4.4.7 starting... (PID=1336)
Local time is Tue Aug 23 23:18:36 UTC 2022
wproc: Successfully registered manager as @wproc with query handler
wproc: Registry request: name=Core Worker 1338;pid=1338
wproc: Registry request: name=Core Worker 1339;pid=1339
wproc: Registry request: name=Core Worker 1340;pid=1340
wproc: Registry request: name=Core Worker 1341;pid=1341
wproc: Registry request: name=Core Worker 1342;pid=1342
wproc: Registry request: name=Core Worker 1343;pid=1343
wproc: Registry request: name=Core Worker 1345;pid=1345
wproc: Registry request: name=Core Worker 1344;pid=1344
wproc: Registry request: name=Core Worker 1348;pid=1348
wproc: Registry request: name=Core Worker 1349;pid=1349
wproc: Registry request: name=Core Worker 1347;pid=1347
wproc: Registry request: name=Core Worker 1346;pid=1346
Segmentation fault (core dumped)

I'm not sure how to debug this further. Any help appreciated.

[gurubobnz]

Well after leaving this running in that restarting state it finally decided it wanted to start up and seems to be stable.

nagios_1                  | 2022-08-24T02:51:55.952709129Z checking permissions for nagios & nagiosgraph
nagios_1                  | 2022-08-24T02:51:55.959741170Z 
nagios_1                  | 2022-08-24T02:51:55.959772812Z Nagios Core 4.4.7
nagios_1                  | 2022-08-24T02:51:55.959782296Z Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
nagios_1                  | 2022-08-24T02:51:55.959790605Z Copyright (c) 1999-2009 Ethan Galstad
nagios_1                  | 2022-08-24T02:51:55.959800581Z Last Modified: 2022-04-14
nagios_1                  | 2022-08-24T02:51:55.959813398Z License: GPL
nagios_1                  | 2022-08-24T02:51:55.959825656Z 
nagios_1                  | 2022-08-24T02:51:55.959839352Z Website: https://www.nagios.org
nagios_1                  | 2022-08-24T02:51:55.959852142Z Nagios 4.4.7 starting... (PID=6652)
nagios_1                  | 2022-08-24T02:51:55.959865094Z Local time is Wed Aug 24 02:51:55 UTC 2022
nagios_1                  | 2022-08-24T02:51:55.959878356Z nagios: Nagios 4.4.7 starting... (PID=6652)
nagios_1                  | 2022-08-24T02:51:55.959892893Z nagios: Local time is Wed Aug 24 02:51:55 UTC 2022
nagios_1                  | 2022-08-24T02:51:55.959905266Z nagios: LOG VERSION: 2.0
nagios_1                  | 2022-08-24T02:51:55.959922814Z wproc: Successfully registered manager as @wproc with query handler
nagios_1                  | 2022-08-24T02:51:55.959937946Z nagios: qh: Socket '/opt/nagios/var/rw/nagios.qh' successfully initialized
nagios_1                  | 2022-08-24T02:51:55.959946931Z nagios: qh: core query handler registered
nagios_1                  | 2022-08-24T02:51:55.959954561Z nagios: qh: echo service query handler registered
nagios_1                  | 2022-08-24T02:51:55.959962076Z nagios: qh: help for the query handler registered
nagios_1                  | 2022-08-24T02:51:55.959969581Z nagios: wproc: Successfully registered manager as @wproc with query handler
nagios_1                  | 2022-08-24T02:51:55.961933463Z wproc: Registry request: name=Core Worker 6654;pid=6654
nagios_1                  | 2022-08-24T02:51:55.961985763Z nagios: wproc: Registry request: name=Core Worker 6654;pid=6654
nagios_1                  | 2022-08-24T02:51:55.961999325Z wproc: Registry request: name=Core Worker 6655;pid=6655
nagios_1                  | 2022-08-24T02:51:55.962073726Z nagios: wproc: Registry request: name=Core Worker 6655;pid=6655
nagios_1                  | 2022-08-24T02:51:55.962083649Z wproc: Registry request: name=Core Worker 6658;pid=6658
nagios_1                  | 2022-08-24T02:51:55.962089622Z nagios: wproc: Registry request: name=Core Worker 6658;pid=6658
nagios_1                  | 2022-08-24T02:51:55.962116897Z wproc: Registry request: name=Core Worker 6656;pid=6656
nagios_1                  | 2022-08-24T02:51:55.962230062Z nagios: wproc: Registry request: name=Core Worker 6656;pid=6656
nagios_1                  | 2022-08-24T02:51:55.962242774Z wproc: Registry request: name=Core Worker 6659;pid=6659
nagios_1                  | 2022-08-24T02:51:55.962326859Z wproc: Registry request: name=Core Worker 6660;pid=6660
nagios_1                  | 2022-08-24T02:51:55.962383886Z nagios: wproc: Registry request: name=Core Worker 6659;pid=6659
nagios_1                  | 2022-08-24T02:51:55.962396071Z nagios: wproc: Registry request: name=Core Worker 6660;pid=6660
nagios_1                  | 2022-08-24T02:51:55.962616302Z wproc: Registry request: name=Core Worker 6657;pid=6657
nagios_1                  | 2022-08-24T02:51:55.962627903Z nagios: wproc: Registry request: name=Core Worker 6657;pid=6657
nagios_1                  | 2022-08-24T02:51:55.963381643Z wproc: Registry request: name=Core Worker 6665;pid=6665
nagios_1                  | 2022-08-24T02:51:55.963393942Z nagios: wproc: Registry request: name=Core Worker 6665;pid=6665
nagios_1                  | 2022-08-24T02:51:55.963451446Z wproc: Registry request: name=Core Worker 6664;pid=6664
nagios_1                  | 2022-08-24T02:51:55.963462196Z nagios: wproc: Registry request: name=Core Worker 6664;pid=6664
nagios_1                  | 2022-08-24T02:51:55.963527884Z wproc: Registry request: name=Core Worker 6661;pid=6661
nagios_1                  | 2022-08-24T02:51:55.963539900Z nagios: wproc: Registry request: name=Core Worker 6661;pid=6661
nagios_1                  | 2022-08-24T02:51:55.963599881Z wproc: Registry request: name=Core Worker 6663;pid=6663
nagios_1                  | 2022-08-24T02:51:55.963613470Z nagios: wproc: Registry request: name=Core Worker 6663;pid=6663
nagios_1                  | 2022-08-24T02:51:55.963810311Z wproc: Registry request: name=Core Worker 6662;pid=6662
nagios_1                  | 2022-08-24T02:51:55.963821860Z nagios: wproc: Registry request: name=Core Worker 6662;pid=6662
nagios_1                  | 2022-08-24T02:51:58.437932469Z Successfully launched command file worker with pid 6666
nagios_1                  | 2022-08-24T02:51:58.437991041Z nagios: Successfully launched command file worker with pid 6666

[himslm01]

Could the problem be this issue?

https://community.spiceworks.com/topic/2455449-nagios-core-4-4-6-works-correct-nagios-core-4-4-7-give-segmentation-fault

The solution of setting this parameter has worked for me. Hopefully the issue will be fixed upstream and we can remove this setting again.

check_for_updates=0

[valpo]

Could the problem be this issue?

https://community.spiceworks.com/topic/2455449-nagios-core-4-4-6-works-correct-nagios-core-4-4-7-give-segmentation-fault

The solution of setting this parameter has worked for me. Hopefully the issue will be fixed upstream and we can remove this setting again.

check_for_updates=0

I just ran into the same problem, your solution works for me, thank you!

Mathias

[samlnx03]

it works for me also. Thx

[luis15pt]

Also worked, just to clarify anyone getting here, you need to edit the nagios.cfg inside the etc directory

[gurubobnz]

I can't quite tell what the effect of check_for_updates is other than perhaps notifying the administrator. Here's the bit from the config:

# CHECK FOR UPDATES
# This option determines whether Nagios will automatically check to
# see if new updates (releases) are available.  It is recommend that you
# enable this option to ensure that you stay on top of the latest critical
# patches to Nagios.  Nagios is critical to you - make sure you keep it in
# good shape.  Nagios will check once a day for new updates. Data collected
# by Nagios Enterprises from the update check is processed in accordance 
# with our privacy policy - see https://api.nagios.org for details.

check_for_updates=1

Given we're all grown ups perhaps it make sense for this to be check_for_updates=0 in the docker container (either literally in the config, or as a config option somehow) to at the very least work around this segfault until the underlying cause has been resolved.

For now I have just edited this value in the container and restarted it, but it will revert next time I pull a fresh file. Actually now that I think about it, having a copy of the config on the outside of the container and mounting it over the top would achieve the same - at the expense of ignoring any future changes to the config in the container itself.

[himslm01]

having a copy of the config on the outside of the container and mounting it over the top would achieve the same - at the expense of ignoring any future changes to the config in the container itself.

Yes - that is what I do.

I run this image in a kubernetes cluster with all of the configuration external to the container/pod in kubernetes ConfigMaps which are mounted over the files in the image. If I were running this image as a container in Docker I'd be doing the same, by mounting directories as volumes over the folders in the image.

As far as I can see, other than stopping the executable from seg-faulting, the only other difference is the warning on the home screen.

[Kolossi]

check_for_updates=0 worked for me too.

One of the things I use nagios for is to look for updated versions of all the docker images I use - including nagios! - so I can cope until @JasonRivers does a 4.4.8 for us:

4.4.8 – 2022-10-04
FIXES

• Minor interface updates
• Fixed crash when checking for updates using SSL

(source: https://www.nagios.org/projects/nagios-core/history/4x/)