[Meta] Upgrading from 7.x to 8.0

[roaksoax]

This issue tracks all the items that need to be handled for the 8.0 release.

• Support Breaking Changes in Stack Components:
  • Elasticsearch finalizing Types Removal (upstream: Types removal in 8.0 elasticsearch#41059)
    • Elasticsearch Output no longer sends type when target ES >= 8.0 Only add _type if ES version < 8 logstash-plugins/logstash-output-elasticsearch#892
    • Elasticsearch Output no longer sends type when target ES >= 7.0 and user don't customize it with document_type Remove type when connecting to ES 7 and user doesn't specify it logstash-plugins/logstash-output-elasticsearch#994
    • X-Pack Internal Monitoring uses type, but is possibly slated for removal in 8.0
    • Investigate _type usage in elasticsearch output Specifying types in bulk requests is deprecated logstash-plugins/logstash-output-elasticsearch#915
  • [kaise] Elasticsearch migration to composable templates and removal of _template API (upstream: Composable Templates elasticsearch#53101)
    • Implement Composable Templates to support Elasticsearch 8.0 logstash-plugins/logstash-output-elasticsearch#944
  • Switch ES plugins to Logstash specific APIs (Following System Indices [Meta] Logstash use system indices API  #12291)
  • Drop _xpack API endpoint usage (and settings) Switch license checking from _xpack to _license #11382
  • Agent Migration
  • GeoIP transition to using service.
• Logstash Breaking Changes
  • Internal Collection - Deprecation in favor of agent (or drop sending data directly to ES cluster) ([Stack Monitoring] Remove internal collectors #11169)
  • [kaise] Execution Engine - Remove Ruby Execution Engine (remove ruby execution #11236)
  • ECS - Pipelines in ECS-Compatibility mode as the default (meta: [META] A Path to ECS-by-default in Logstash Plugins #11623)
  • DOC - Document breaking changes
  • [kaise] Settings - Remove "field-reference-parse" Remove "field-reference-parser" setting #12355: INPROGRESS
  • [andsel] Modules - Remove modules infrastructure and existing modules (Azure/Netflow)
  • Elasticsearch Output under default distribution requires licensed Elasticsearch (check if ES is licensed before pushing out events to it logstash-plugins/logstash-output-elasticsearch#875)
  • Operating System support changes in 8.0.0 Drop support for older Operating Systems for 8.0.0 #12511
  • Auditing for terminology: master/slave, *list, and others Auditing for terminology: master/slave, *list, and others #12052
  • [andsel] Use LS_JAVA_HOME instead of JAVA_HOME [Meta ] Use LS_JAVA_HOME in favor of JAVA_HOME #12725 (See Remove support for JAVA_HOME elasticsearch#69149 for references in ES)
  • Review/evaluate other breaking changes ( breaking change )
    and open issues targeted for 8.0.0
• Other (e.g. Stack Features Logstash should Support)
  • Elasticsearch Introduction of Hidden Indices (upstream: Implement hidden indices elasticsearch#50452)
  • System Indices - Clean-up old implementation ([Meta] Logstash use system indices API  #12291)
  • Runtime Fields
  • Logstash Breaking Changes
    • avoid Ruby Event wrapper - just rely on the Java org.logstash.Event in Ruby land
  • ? stack update JRuby to 9.3 (would resolve loading LS might (rarely) fail due JRuby auto-load bug #12077)
    • Java collection patches revisit LS' Java (collection) integration patches #12800
    • actively (CI) testing against jruby-head
  • New 'major' release of Ruby filter (Breaking: restrict unintended user constant usage logstash-plugins/logstash-filter-ruby#56)
  • Document upgrade instructions

[yaauie]

@roaksoax I do not believe that the "Hidden Indices" feature of Elasticsearch should be classified with "Breaking Changes". It is a new feature, by which a user can now explicitly configure an index to not be available to wildcard queries. Logstash does not use this feature when creating indices (yet?) and any wildcard queries that are performed in the Elasticsearch Input and Filter plugins will continue to work as-is unless a user explicitly configures an index to not be returned.

[jsvd]

Adding a terminology audit task that will result in breaking changes #12052

[roaksoax]

This issue no longer seems relevant. I'm gonna go ahead and close it.