Installation script ACL adjustments don't persist log rotation

[portertech]

Problem

The installation script currently adjusts log file ACLs making it possible for an unprivileged system user (otelcol-sumo) to ingest log events. This works in most cases, however, the ACLs do not persist log rotation (a log file is replaced by an empty file without the ACL).

Possible Solutions

Most Linux distributions use logrotate to handle the rotation of log files. Users can configure logrotate to use a postrotate script to reapply the ACL. For example:

The value of $1 is the log file path (set by logrotate).

    postrotate
        /usr/bin/setfacl -m u:otelcol-sumo:r,g:otelcol-sumo:r $1
    endscript

Unfortunately, we cannot create a logrotate config that applies this postrotate script to all log files for a few reasons. This could apply log rotation to the wrong files or the postrotate statement is overridden by more specific log file path matches and the ACL is not applied. For example:

/etc/logrotate.d/otel

/var/log/* {
    postrotate
        /usr/bin/setfacl -m u:otelcol-sumo:r,g:otelcol-sumo:r $1
    endscript
}

On my Rocky 9 (RHEL) host, this configuration would be overridden by the default rsyslog configuration:

/etc/logrotate.d/rsyslog

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
    endscript
}

To fix the log file ACLs after rotation on my host, I updated the rsyslog configuration:

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
        /usr/bin/setfacl -m u:otelcol-sumo:r,g:otelcol-sumo:r $1
    endscript
}

Not convinced we can script the "safe" installation of logrotate config. This is likely to be addressed via documentation and the trial user onboarding flow dialog. For example, we already call out file ACLs as a prerequisite:

[portertech]

I found that logrotate on RHEL 9 does correctly apply the parent directory's default file ACL. Setting the correct parent directory default file ACL results in the desired outcome:

setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x "/var/log/"

For example:

[root@opentelemetry var]# getfacl log
# file: log
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

[root@opentelemetry var]# getfacl log/messages
# file: log/messages
# owner: root
# group: root
user::rw-
group::r--
other::r--

[root@opentelemetry var]# cat /etc/logrotate.d/rsyslog 
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
    endscript
}
[root@opentelemetry var]# logrotate --force  /etc/logrotate.d/rsyslog 
[root@opentelemetry var]# getfacl log/messages
# file: log/messages
# owner: root
# group: root
user::rw-
group::---
other::---

[root@opentelemetry var]# setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x "/var/log/"
[root@opentelemetry var]# getfacl log
# file: log
# owner: root
# group: root
user::rwx
user:otelcol-sumo:r-x
group::r-x
group:otelcol-sumo:r-x
mask::r-x
other::r-x
default:user::rwx
default:user:otelcol-sumo:r-x
default:group::r-x
default:group:otelcol-sumo:r-x
default:mask::r-x
default:other::r-x

[root@opentelemetry var]# getfacl log/messages
# file: log/messages
# owner: root
# group: root
user::rw-
user:otelcol-sumo:r-x
group::---
group:otelcol-sumo:r-x
mask::r-x
other::---

[root@opentelemetry var]# setfacl -b log/messages
[root@opentelemetry var]# getfacl log/messages
# file: log/messages
# owner: root
# group: root
user::rw-
group::---
other::---

[root@opentelemetry var]# logrotate --force  /etc/logrotate.d/rsyslog 
[root@opentelemetry var]# getfacl log/messages
# file: log/messages
# owner: root
# group: root
user::rw-
user:otelcol-sumo:r-x           #effective:r--
group::r-x                      #effective:r--
group:otelcol-sumo:r-x          #effective:r--
mask::r--
other::r--

[portertech]

If this isn't the case on a particular distribution/version, we could create a "dummy" log file and logrotate configuration to adjust the ACL of specific files. The logrotate configuration would execute on the regular schedule. For example:

[root@opentelemetry log]# ls -l /var/log/otel-acls 
-rw-r--r--+ 1 root root 90 Apr 24 13:07 /var/log/otel-acls
[root@opentelemetry log]# cat /var/log/otel-acls 
# DO NOT DELETE - Used by logrotate to adjust log file ACLs for OpenTelemetry collection.
[root@opentelemetry log]# cat /etc/logrotate.d/otel-acls 
/var/log/otel-acls
{
    postrotate
        /usr/bin/setfacl -m u:otelcol-sumo:r,g:otelcol-sumo:r "/var/log/messages"
        echo "# DO NOT DELETE - Used by logrotate to adjust log file ACLs for OpenTelemetry collection." > /var/log/otel-acls
    endscript
}
[root@opentelemetry log]# logrotate --force  /etc/logrotate.d/otel-acls 
[root@opentelemetry log]# getfacl messages
# file: messages
# owner: root
# group: root
user::rw-
user:otelcol-sumo:r--
group::r-x
group:otelcol-sumo:r--
mask::r-x
other::r--