critical/cli: Cannot parse available features. Path '/etc/icinga2/features-available' does not exist.

[AqujZ1K9]

Hi Guys,
am I the only one with this issue when creating the container from the provided docker-compose

Pinging mysql:3306 critical/cli: Cannot parse available features. Path '/etc/icinga2/features-available' does not exist. run-parts: /opt/setup/50-icinga2 exited with return code 1

haven't changed anything at the compose-file.
Running docker on PhotonOs(x64).
Anything I'm not seeing here?
Thanks!

[jjethwa]

Hi @AqujZ1K9

Hmm, that's strange. What version of docker are you using? The error refers to this script: https://github.com/jjethwa/icinga2/blob/master/content/opt/setup/50-icinga2

/etc/icinga2/features-available should exist as it's part of the standard icinga2 install, but we might need to add a line to create it if it doesn't exist.

[AqujZ1K9]

@jjethwa thanks for reaching out!
Docker version 19.03.10, build 9424aea

whats stange that the binding seems to be ok. Anything I could check ?

[jjethwa]

No problem 😃 Can you double check the /etc/icinga2 volume and see if it has the features-available directory under it? Sorry, this will require digging under your docker var directory.

[AqujZ1K9]

root@icinga2:/etc/icinga2/features-available# ls api.conf command.conf debuglog.conf gelf.conf icingadb.conf influxdb.conf mainlog.conf opentsdb.conf statusdata.conf checker.conf compatlog.conf elasticsearch.conf graphite.conf ido-mysql.conf livestatus.conf notification.conf perfdata.conf syslog.conf root@icinga2:/etc/icinga2/features-available#

within the container it seems to be all up and running.

once I'm starting the mysql container, the ping succeeds and the icinga container exits with the Path '/etc/icinga2/features-available' does not exist.

[jjethwa]

Could it be that it's somehow trying to run the script on the mysql container? I've never seen that happen before though.

[AqujZ1K9]

the problem seems to be the line : icinga2 feature enable ido-mysql livestatus compatlog command

the whole icinga2 cli seems to be broken for me.

e.g.: icinga2 feature list

root@icinga2:/var/log/icinga2/crash# icinga2 feature list Could not create directory '/var/log/icinga2/crash': Error 13, Permission denied critical/Application: Error: Function call 'glob' for file '/etc/icinga2/features-available/*.conf' failed with error code 13, 'Permission denied'

any Idea where to go for that ?

[jjethwa]

Thanks for the additional details, this looks like it's actually erroring out in the CLI API calls, I originally thought it was the docker scripts. Can you check on the existence of /var/log/icinga2 and the permissions and ownership? Can you also check the permissions and ownership for /etc/icinga2 and etc/icinga2/features-available/*.conf as well?

[AqujZ1K9]

/var/log/icinga2 exists:
drwxr-xr-x 3 nagios adm 4096 Dec 16 22:41 icinga2

/etc/icinga2
drwxr-xr-x 8 nagios root 4096 Dec 16 22:41 icinga2

/etc/icinga2/features-available/
drwxr-x--- 2 nagios root 4096 Dec 16 22:41 .

/etc/icinga2/features-available/*.conf
-rw-r--r-- 1 nagios root 183 Dec 16 22:41 api.conf -rw-r--r-- 1 nagios root 112 Dec 16 22:41 checker.conf -rw-r--r-- 1 nagios root 140 Dec 16 22:41 command.conf -rw-r--r-- 1 nagios root 159 Dec 16 22:41 compatlog.conf -rw-r--r-- 1 nagios root 235 Dec 16 22:41 debuglog.conf -rw-r--r-- 1 nagios root 193 Dec 16 22:41 elasticsearch.conf -rw-r--r-- 1 nagios root 206 Dec 16 22:41 gelf.conf -rw-r--r-- 1 nagios root 190 Dec 16 22:41 graphite.conf -rw-r--r-- 1 nagios root 91 Dec 16 22:41 icingadb.conf -rw------- 1 nagios root 241 Dec 16 22:41 ido-mysql.conf -rw-r--r-- 1 nagios root 560 Dec 16 22:41 influxdb.conf -rw-r--r-- 1 nagios root 127 Dec 16 22:41 livestatus.conf -rw-r--r-- 1 nagios root 159 Dec 16 22:41 mainlog.conf -rw-r--r-- 1 nagios root 129 Dec 16 22:41 notification.conf -rw-r--r-- 1 nagios root 564 Dec 16 22:41 opentsdb.conf -rw-r--r-- 1 nagios root 146 Dec 16 22:41 perfdata.conf -rw-r--r-- 1 nagios root 225 Dec 16 22:41 statusdata.conf -rw-r--r-- 1 nagios root 127 Dec 16 22:41 syslog.conf

• are the permissions correct ? What should they be ?

additon, when logging into the container as "nagios" I get:

bash: /etc/bash.bashrc: Permission denied bash: /var/lib/nagios/.bashrc: Permission denied

thanks a lot!

[jjethwa]

Thanks, those permissions look correct and you won't be able to start an interactive shell with the nagios user as it's meant for daemon use only. Is there anything in the API log under /var/log/icinga2 that might be related? Sorry about all of the back and forth, I've just never seen this before, so I'm trying to collect as much information on it.

[AqujZ1K9]

no logs under /var/log/icinga2
Icinga2 CLi can't even write Logs:
icinga2 feature list
Could not create directory '/var/log/icinga2/crash': Error 13, Permission denied

Thanks for helping so far!

adding: just pull up a new vm running PhotonOS4.0 -Beta. just to be sure if anything on the host is corrupt. Same behavior here.

[jjethwa]

I'll try pulling down Photon later today and see if I can reproduce the issue. In the meantime, try:

mkdir -p /var/log/icinga2/crash
chown -R nagios:adm /var/log/icinga2

And running the CLI command again. Let's see if we can get something in the logs this time 😞

[AqujZ1K9]

doesn't seem to do anything /help...

[jjethwa]

So you're getting the error still when running the CLI command?

[AqujZ1K9]

yes sir

[jjethwa]

Eek, thanks for your patience @AqujZ1K9 I'll get that VM going a bit later and test it out.

[jjethwa]

Hi @AqujZ1K9

Need just a little more time, I ran an update on my laptop last night that is causing issues

[AqujZ1K9]

no worries. I've changed to an Debian Buster Host now and don't have these Issues anymore. But his can still be interesting for other tho.
Thanks for helping out!

[jjethwa]

Thanks for the update, @AqujZ1K9

Once I get my laptop updated properly, I'll try to spend time and see if I can figure it out.

[gianmarco-mameli]

Hi, sorry for attaching my problem to this old issue, but in fact I've the same problem, same error as @AqujZ1K9.
I've builded my image to updating Icinga2 at version 2.13.1. When starting the error is

critical/cli: Cannot parse available features. Path '/etc/icinga2/features-available' does not exist.

the folder is mapped to a volume on a glusterfs, operating system Debian Bullseye 11.0 arm64
Already checked/performed all solutions of this thread

Thans

[jjethwa]

Hi @gianmarco-mameli

No problem! I had forgotten about this issue, sorry @AqujZ1K9 😞

Is etc/icinga2/features-available available in the volume? What are the permissions? The CLI utility will make modifications such as symlinking to files inside.

[gianmarco-mameli]

the folder is available, and the permissions are correct. Deploying an old image builded by me it works without problems.
I'm expert of Icinga2 because I'm administrator at my work of an infrastructure with 2 master and 22 satellites. I've tried any solution I know but it doesn't work. The folder is in the correct place and with correct permission but strangely the command for enable features fail.
I need the docker image for my raspberry pis cluster.
Maybe the problem is the glusterfs, but the other user uses normal fs

[gianmarco-mameli]

update: removing volume /etc/icinga2 works, but i've noticed the version of icinga is very old.
I'm trying rebuild the image with raspbian repo for armhf packages

[jjethwa]

Hi @gianmarco-mameli

I would not think it's gluster as that would be transparent to the app. Is there anything else interesting in the logs that might help? I'm wondering if you can start the container in interactive mode so you can poke around inside? You might have to override the run script

[jjethwa]

Just saw the new comment, so some progress. Keep me in the loop, I haven't used the rasbian repo 😃

[gianmarco-mameli]

Hi, after 1 year, I decided to rebuild this image for my Raspberry Pi Docker Swarm cluster, but errors still present, with or without mounted volumes.
For testing I forked your project and made some adjustments to docker file, like adding packages and changing base image
I've tried with debian:bookworm-slim and ubuntu:kinetic, these are devel images but the only version that have all icinga2 packages updated form armhf platform

examples:
critical/cli: Cannot parse available features. Path '/etc/icinga2/features-available' does not exist.
Could not create directory '/var/log/icinga2/crash': Error 13, Permission denied
critical/Application: Error: Function call 'glob' for file '/etc/icinga2/features-available/*.conf' failed with error code 13, 'Permission denied'

[jjethwa]

Hi @gianmarco-mameli

Sorry, I still don't have a Pi setup to test myself, so I'll need to rely on your testing and reports. Are these errors occuring at container startup or during the image building process?

Can you provide the docker run command you are using?

[gianmarco-mameli]

Hi @jjethwa no problem, I try to explain as must as possible to you.
The errors came out when I try to run the entrypoint "/opt/run" or in a simple check running "icinga2 daemon -C" or "icinga2 feature list".
I cannot explain why, but it seems that icinga is not capable to read any of the folder like cache or config, or log. I already tried modifying the permission in every method i know and found on web.
The deploy is made by a compose file in a stack that you can find below, ( I removed some non important info and replaced with [ ] )
If you think the volumes in GlusterFS filesystem maybe are the problem (me too I supposed the same), I can replicate without any mounted volumes.
The platform is Armv7, docker running on Raspbian Bullseye on 3x Rapsberry pi 4 4gb

Let me know if you need more information

version: "3.8"
services:
  icinga2:
    image: <localregistry>:5000/icinga2:latest
    hostname: icinga2.docker.local
    volumes:
      - icinga2_etc_icinga2:/etc/icinga2
      - icinga2_etc_icingaweb2:/etc/icingaweb2
      - icinga2_var_lib_icinga2:/var/lib/icinga2
      - icinga2_var_lib_php_sessions:/var/lib/php/sessions
      - icinga2_var_log_apache2:/var/log/apache2
      - icinga2_var_log_icinga2:/var/log/icinga2
      - icinga2_var_log_icingaweb2:/var/log/icingaweb2
      - icinga2_var_log_supervisor:/var/log/supervisor
      - icinga2_usr_lib_nagios_extra_plugins:/usr/lib/nagios/extra_plugins
      - icinga2_var_lib_centreon_centplugins:/var/lib/centreon/centplugins
      - icinga2_var_cache_icinga2:/var/cache/icinga2
      - "/etc/msmtprc:/etc/msmtprc:ro"
      - "/etc/aliases:/etc/aliases:ro"
      - "/etc/locale.gen:/etc/locale.gen:ro"
    ports:
      - "5665:5665"
    environment:
      - TZ=Europe/Rome
      - ICINGA2_FEATURE_GRAPHITE=true
      - ICINGA2_FEATURE_GRAPHITE_HOST=graphite
      - ICINGA2_FEATURE_GRAPHITE_PORT=2003
      - ICINGA2_FEATURE_GRAPHITE_URL=http://graphite
      - ICINGA2_FEATURE_GRAPHITE_SEND_THRESHOLDS=true
      - ICINGA2_FEATURE_GRAPHITE_SEND_METADATA=false
      - ICINGAWEB2_ADMIN_USER=admin
      - ICINGAWEB2_ADMIN_PASS=***
      - ICINGAWEB2_FEATURE_DIRECTOR=true
      # - ICINGA2_FEATURE_DIRECTOR_KICKSTART=false
      - ICINGA2_FEATURE_DIRECTOR_USER=icinga2-director
      - ICINGA2_FEATURE_DIRECTOR_PASS=***
      # - ICINGA2_USER_FULLNAME=Icinga2 Docker Monitoring Instance
      - DEFAULT_MYSQL_HOST=mysql
      - MYSQL_ROOT_PASSWORD=***
      - DEFAULT_MYSQL_PASS=***
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        max_attempts: 3
      labels:
        [ ]
    networks:
      - traefik_network

volumes:
  icinga2_etc_icinga2:
    driver: glusterfs
    name: gfs/icinga2/etc/icinga2
  icinga2_etc_icingaweb2:
    driver: glusterfs
    name: gfs/icinga2/etc/icingaweb2
  icinga2_var_lib_icinga2:
    driver: glusterfs
    name: gfs/icinga2/var/lib/icinga2
  icinga2_var_lib_php_sessions:
    driver: glusterfs
    name: gfs/icinga2/var/lib/php/sessions
  icinga2_var_log_apache2:
    driver: glusterfs
    name: gfs/icinga2/var/log/apache2
  icinga2_var_log_icinga2:
    driver: glusterfs
    name: gfs/icinga2/var/log/icinga2
  icinga2_var_log_icingaweb2:
    driver: glusterfs
    name: gfs/icinga2/var/log/icingaweb2
  icinga2_var_log_supervisor:
    driver: glusterfs
    name: gfs/icinga2/var/log/supervisor
  icinga2_usr_lib_nagios_extra_plugins:
    driver: glusterfs
    name: gfs/icinga2/usr/lib/nagios/extra_plugins
  icinga2_var_lib_centreon_centplugins:
    driver: glusterfs
    name: gfs/icinga2/var/lib/centreon/centplugins
  icinga2_var_cache_icinga2:
    driver: glusterfs
    name: gfs/icinga2/var/cache/icinga2

networks:
  traefik_network:
    external: true
    name: traefik_network

[jjethwa]

Hi @gianmarco-mameli

Thanks, that was helpful!

Can you try running it with the run command?

docker run -p 80:80 -h icinga2 -t jordan/icinga2:latest

I don't think the FS should make a difference as it should be transparent...hopefully 🤞

https://github.com/jjethwa/icinga2/blob/master/content/opt/setup/50-icinga2#L33 should handle the ownership issues when starting up using volumes.

[gianmarco-mameli]

your latest image does not support arm v7

The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested

[jjethwa]

Oops, sorry! There is the old version of the arm build, but can you try with the image you created? Just trying to see if it comes up without any issues with no volumes, etc

[gianmarco-mameli]

Hi, the container stops for mysql problem

Generating locales (this might take a while)... Clearing symlinks in /etc/ssl/certs... Generation complete. done. Updating certificates in /etc/ssl/certs... rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL 127 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. =>MySQL datadir is empty...initializing Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 2022-09-23 20:28:13 0 [ERROR] mariadbd: Can't create/write to file '/var/lib/mysql/aria_log_control' (Errcode: 13 "Permission denied") 2022-09-23 20:28:13 0 [ERROR] mariadbd: Got error 'Can't create file' when trying to use aria control file '/var/lib/mysql/aria_log_control' 2022-09-23 20:28:13 0 [ERROR] Plugin 'Aria' init function returned error. 2022-09-23 20:28:13 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed. 2022-09-23 20:28:13 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required) 2022-09-23 20:28:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF 2022-09-23 20:28:13 0 [ERROR] Failed to initialize plugins. 2022-09-23 20:28:13 0 [ERROR] Aborting 2022-09-23 20:28:13 0 [ERROR] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool.incomplete' for writing: Permission denied

[jjethwa]

Thanks! I'm going to have to look up these errors as I've never seen them before. It's from this part of the setup process: https://github.com/jjethwa/icinga2/blob/master/content/opt/setup/20-mysql

Nothing too crazy, just initializing an empty database.

[gianmarco-mameli]

Ok, thanks. I'm preparing the packages of icinga2 and Icingaweb2 updated for bullseye, all compiled in docker by source. The idea is to try install all in the image via deb packages, to use Bullseye as a base image. This is because unfortunately the icingaweb2 packages are old in the Icinga bullseye package repo for raspbian.
I keep you updated

[jjethwa]

Sounds good, @gianmarco-mameli

All the updates to get bullseye working for amd64 were on May 19th https://github.com/jjethwa/icinga2/commits/master

[jjethwa]

Hi @gianmarco-mameli

Just checking in with you. Do you need any help? 😄

[adamparker]

This is likely to be some volume permission issue within docker.

I had the same issue, and went through to troubleshoot it. The folder exists but for some reason or other the container is unable to see it properly. It makes no sense which is why I think its docker related.

I removed the /data/icinga folder and rebuilt the image and it started working again. You could try doing a 'rm -rf /etc/icinga2' and then manually copy the /etc/icinga2.dist' folder again to see if that fixes it. Then you can copy over your previous config.