Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclear description of logic that happens on an error in the "parser filter" #442

Open
jwreford99 opened this issue Jan 25, 2023 · 0 comments

Comments

@jwreford99
Copy link

jwreford99 commented Jan 25, 2023

Describe the bug

emit_invalid_record_to_error in the parser filter is a setting which allows users to decide on the outcome that happens when a record is unable to parsed for one of three reasons.
If set to true it says that the record will be given the @ERROR label, and "If you want to ignore these errors, set false".

However, I was seeing different behaviour, and looking at the relevant piece of source code https://github.com/fluent/fluentd/blob/c48ca04a593f2f4628c545d74c90f6caba513bee/lib/fluent/plugin/filter_parser.rb#L72-L95
it became clear that the documentation isn't quite clear.

Notably, the source code clearly has an extra condition that if reserve_data is true then it will not discard these errors (as I would of expected), but instead it allows them to pass but with an unparsed record. I fully understand why this behaviour exists, but I think the documentation around this could be included.

Link to the problematic documentation

https://docs.fluentd.org/filter/parser#emit_invalid_record_to_error
https://docs.fluentd.org/filter/parser#reserve_data

Expected explanation

I would expect the documentation for emit_invalid_record_to_error to clearly say that if reserve_data is set to true, then the record will not be discarded but will instead be passed through without any parsing happening.

Additional context

An example configuration which was not clear from the documentation is:

<filter gearset.*>
  @type parser
  <parse>
    @type json
    json_parser json
  </parse>
  key_name log
  reserve_data true
  emit_invalid_record_to_error false
</filter>

I am happy to open a PR myself, but given this is my first open-source contribution I figured I would put an issue in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant