Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TZ Errors when server is set to local time zone #2

Open
oester opened this issue Sep 16, 2021 · 1 comment
Open

TZ Errors when server is set to local time zone #2

oester opened this issue Sep 16, 2021 · 1 comment

Comments

@oester
Copy link

oester commented Sep 16, 2021

Data appears to be flowing to fluent-bit, but it's posting errors when sending to loki... any ideas?

Sep 14 06:53:24 ubuntu-server td-agent-bit[13650]: [2021/09/14 06:53:24] [error] [output:loki:loki.0] 192.168.15.71:3100, HTTP status=400 Not retrying.
Sep 14 06:53:24 ubuntu-server td-agent-bit[13650]: entry for stream '{city="null", country="null", job="fluentbit"}' has timestamp too new: 2021-09-14 12:53:19 +0000 UTC

"timestamp too new" - ?? The server is running with TZ "America/Central" and was showing time "6:53" when I starting pushing syslog data from my UDM-Pro to fluent-bit.

This server just has fluent-bit and loki running on it, the Grafana server is remote. If I switch the server to UTC, the errors disappear

@proffalken
Copy link
Contributor

Hey, thanks for logging this and apologies for the delay in getting around to looking at it.

Try commenting out this section of the fluent-bit config and restarting the service.

Those lines run a script that converts everything to UTC, so if the data reaching Loki is in UTC it will be 6hrs ahead of the time on the server, which is possibly what's leading to this issue.

Let me know how you get on?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants