ecs: on-by-default plus docs #12830

yaauie · 2021-04-19T17:18:15Z

PREVIEW: https://logstash_12830.docs-preview.app.elstc.co/guide/en/logstash/master/ecs-ls.html

Release notes

Sets ECS to on-by-default (v1, for now). This setting will again change to v8 when ECS v8 goes into public Alpha in advance of the Stack 8.0.0 release.

What does this PR do?

This change makes ECS Compatibility modes on-by-default (as previously promised in #11623) and adds documentation about how to configure ECS Compatibility Modes at the plugin instance, pipeline, and global levels.

These docs are from the perspective of someone running (the unreleased, pre-alpha) Logstash 8, and a backport will need to be entirely re-worded from the perspective of Logstash 7.x to lead people toward the upgrade.

If a user wants to "lock in" pre-ECS behaviour in 7.x, they can do at different levels so by:

declaring ecs_compatibility => disabled on each individual plugin; OR
declaring pipeline.ecs_compatibility: disabled in each pipeline's definition (config/pipelines.yml or Central Management)
declaring pipeline.ecs_compatibility: disabled in config/logstash.yml to provide a global default for pipelines.

Why is it important/What is the impact to the user?

The transition to ECS being on-by-default with Logstash 8.0 is a potentially sharp edge, as the plugins that implement ECS Compatibility modes will operate differently than in their legacy versions when run in Logstash 8, unless specific effort is made to opt-out of the breaking change.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

yaauie · 2021-04-19T17:20:33Z

config/logstash.yml

@@ -64,7 +64,23 @@
 # if there are multiple workers.
 # "false" will disable any extra processing necessary for preserving ordering.
 #
-pipeline.ordered: auto
+# pipeline.ordered: auto


the default value is auto. commenting this out makes it match the rest of the config file where we don't declare default values in the config file we ship.

yaauie · 2021-04-19T17:22:51Z

docs/static/settings/ecs-compatibility.asciidoc

+==== Opting Out
+
+In Logstash 8, these plugins are run in ECS mode by default, but you can opt-out and use the legacy behaviour in one of several ways.
+This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.


Suggested change

This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.

This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to continue running them without modification in 8.0 to produce the same shape of data that they produced before your upgrade.

yaauie · 2021-04-19T17:28:53Z

docs/static/settings/ecs-compatibility.asciidoc

+  path.config: "/etc/path/to/ecs-pipeline.config"
+  pipeline.ecs_compatibility: v1
+
+NOTE: Until the General Availability of Logstash 8.0.0, any value for `pipeline.ecs_compatibility` other than `disabled` -- including the default value on this pre-release branch -- may have undesireable consequences when performing upgrades.


This caveat will go away before Logstash 8.0 reaches GA, which is why I call out the value on the pre-release branch. I want people hitting this doc to have a "wait a minute" moment.

For the 7.x backport, the text will be more like:

NOTE: Until the General Availability of Logstash 8.0.0 and the final minor release of Logstash 7.x, any value for pipeline.ecs_compatibility other than disabled may have undesireable consequences when performing upgrades.
As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause you to automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces.

And 7.final will simply not need this caveat at all.

karenzone

Suggestions for your consideration.I have tested these suggestions locally, and hope that I moved them over correctly. Github isn't an ideal tool for collaborating on docs, and I have no way to test until these suggestions are incorporated or rejected. This review is about structure and standards so there might be a bit more fine tuning (esp. behavior between versions) to do. Fantastic job on the docs.

docs/static/ecs-compatibility.asciidoc

yaauie · 2021-04-30T00:03:09Z

jenkins test this again please

karenzone

LGTM 🎉
Nice explanation. Thanks for adding this.

yaauie · 2021-06-17T22:20:55Z

Ongoing test failures are unrelated; I've filed https://github.com/elastic/logstash/pull/13002/files to fix.

Since this is a "breaking" change, I will wait until Monday when other folks are around to get both merged.

Co-authored-by: Karen Metts <[email protected]>

We know that ECS version 8 will release along-side Logstash 8.0, but its scope is still coming into focus. In this changeset, we change the default value of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a significantly closer approximation to what will eventually ship in Logstash 8.0.0.

Co-authored-by: Karen Metts <[email protected]>

* noop: avoid declaring default value in config file (cherry picked from commit f8fe4b6) * docs: ecs compatibility from 7.x perspective Co-authored-by: Karen Metts <[email protected]> (cherry picked from commit 69702ca) * ecs docs: pr feedback a11y link text Co-authored-by: Karen Metts <[email protected]> Co-authored-by: Karen Metts <[email protected]>

* master: (41 commits) Test: resolve integration failure due ECS mode (elastic#13044) Feat: event factory support (elastic#13017) Doc: Add geoip database API to node stats (elastic#13019) Add geoip database metrics to /node/stats API (elastic#13004) ecs: on-by-default plus docs (elastic#12830) ispec: fix cross-spec leak from fatal error integration specs (elastic#13002) Fix UBI source URL (elastic#13008) update fpm to allow pkg creation on jdk11+jruby 9.2 (elastic#13005) Add unit test to grant that production aliases correspond to a published RubyGem (elastic#12993) Fix logstash.bat not setting exit code (elastic#12948) Use the OS separator to invoke gradlew from Rake script (elastic#13000) Allow per-pipeline config of ECS Compatibility mode via Central Management (elastic#12861) Update jinja2 dependency in docker build (elastic#12994) fix database manager with multiple pipelines (elastic#12862) Fix Reflections stack traces when process yml files in classpath and debug is enabled (elastic#12991) Fix/log4j routing to avoid create spurious file (elastic#12965) Deps: update JRuby to 9.2.19.0 (elastic#12989) Doc: Add tip for checking for existing field (elastic#12899) Added test to cover the installation of aliased plugins (elastic#12967) CI: Update logstash_release.json after 7.3.12 (elastic#12986) ...

yaauie added docs breaking change labels Apr 19, 2021

yaauie commented Apr 19, 2021

View reviewed changes

yaauie force-pushed the ecs-compatibility-docs branch 2 times, most recently from 53c61e3 to 939f93b Compare April 19, 2021 17:58

roaksoax added status:needs-review status:work-in-progress and removed status:needs-review status:work-in-progress labels Apr 19, 2021

roaksoax added status:work-in-progress and removed status:needs-review labels Apr 27, 2021

yaauie added status:needs-review and removed status:work-in-progress labels Apr 28, 2021

yaauie marked this pull request as ready for review April 28, 2021 12:29

karenzone reviewed Apr 28, 2021

View reviewed changes

yaauie force-pushed the ecs-compatibility-docs branch 3 times, most recently from 48eb209 to 070df2b Compare April 28, 2021 22:25

karenzone approved these changes Apr 30, 2021

View reviewed changes

roaksoax added status:approved and removed status:needs-review labels Apr 30, 2021

karenzone mentioned this pull request May 26, 2021

doc: add pipeline.ecs_compatibility docs #12421

Merged

5 tasks

yaauie force-pushed the ecs-compatibility-docs branch 3 times, most recently from 087bf44 to a956af6 Compare June 17, 2021 20:59

yaauie and others added 4 commits June 21, 2021 15:16

noop: avoid declaring default value in config file

f8fe4b6

docs: ecs compatibility from 7.x perspective

69702ca

Co-authored-by: Karen Metts <[email protected]>

docs: ecs from 8.x perspective

9e00a7e

Co-authored-by: Karen Metts <[email protected]>

yaauie force-pushed the ecs-compatibility-docs branch from a956af6 to 9e00a7e Compare June 21, 2021 15:17

yaauie merged commit 1a4be95 into elastic:master Jun 21, 2021

yaauie mentioned this pull request Jun 21, 2021

Backport PR #12830 to 7.x: ecs-compatibility docs #13016

Merged

5 tasks

yaauie added the v7.14.0 label Jun 21, 2021

kares mentioned this pull request Jun 22, 2021

Missing (ECS) schema for ES 8.0 logstash-plugins/logstash-output-elasticsearch#1022

Closed

roaksoax removed the status:approved label Oct 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ecs: on-by-default plus docs #12830

ecs: on-by-default plus docs #12830

yaauie commented Apr 19, 2021 •

edited by karenzone

Loading

yaauie Apr 19, 2021

yaauie Apr 19, 2021

yaauie Apr 19, 2021

karenzone left a comment

yaauie commented Apr 30, 2021

karenzone left a comment

yaauie commented Jun 17, 2021

	This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.
	This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to continue running them without modification in 8.0 to produce the same shape of data that they produced before your upgrade.

ecs: on-by-default plus docs #12830

ecs: on-by-default plus docs #12830

Conversation

yaauie commented Apr 19, 2021 • edited by karenzone Loading

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

yaauie Apr 19, 2021

Choose a reason for hiding this comment

yaauie Apr 19, 2021

Choose a reason for hiding this comment

yaauie Apr 19, 2021

Choose a reason for hiding this comment

karenzone left a comment

Choose a reason for hiding this comment

yaauie commented Apr 30, 2021

karenzone left a comment

Choose a reason for hiding this comment

yaauie commented Jun 17, 2021

yaauie commented Apr 19, 2021 •

edited by karenzone

Loading