Postgresql with grafana-agent generates excessively large /var/log/syslog files

[Vultaire]

Steps to reproduce

Deploy postgresql with grafana-agent related through the cos-agent endpoint.

Expected behavior

Grafana-agent is able to collect relevant log files from prometheus without significant logging overhead.

Actual behavior

/var/log/syslog is excessively large, growing to 17 gigabytes in just over 48 hours, with an estimated 16 gigabytes being due to grafana-agent logging about the patroni log files. More specifically, roughly every 30 seconds I'm seeing 10,080 log entries, one for each of those 10,080 per-minute patroni log files.

Each line is a little over 300 characters. Repeated 10,080 times 24602 times a day (2,880 times), that comes out to 10,080 * 300 * 2,800 = 8,467,200,000 bytes, i.e. just short of 8 gigabytes per day. This means that for the just-over-2-day-old syslog file mentioned above, roughly 16 out of the 17 gigabytes mentioned is estimated to be due to these messages. And there's also the I/O impact of the same.

Versions

Operating system: Ubuntu 22.04.5 LTS

Juju CLI: 3.6.0-genericlinux-amd64

Juju agent: 3.4.6

Charm revision: 468 (14/stable)

LXD: N/A

Log output

Example log entry for the record:

Dec  8 00:00:22 dev-landscape-server-3 grafana-agent.grafana-agent[2939957]: ts=2024-12-08T00:00:22.530765687Z caller=filetargetmanager.go:181 level=info component=logs logs_config=log_file_scraper msg="received file watcher event" name=/snap/grafana-agent/51/shared-logs/patroni/patroni.log.10080 op=CREATE

Additional context

I think this should be reviewed, and perhaps should be discussed with
the COS team if necessary, to see how this charm can accomplish its
goals while also not generating excessively large syslogs due to the
interaction of grafana-agent with this charm.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6174.

This message was autogenerated

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6175.

This message was autogenerated

[Vultaire]

It seems this was a duplicate of #524, which itself refers to canonical/grafana-agent-operator#153. Closing this particular bug.