Migrating to 8.0.0 for beats user

[nchaulet]

Description

While trying to figure how logstash work with data stream with beats. I setup a logstash 8.0.0 and a metricbeat 8.0.0 and I found that all the beats data are send in log-generic-default (I think it's expected as beats do not send data_stream.* field)

It looks like it possible to add these field with the add_data_stream processor but I got the following error in metricbeat Exiting: error initializing processors: the processor action add_data_stream does not exist. Valid actions.

@ruflin @jsvd Are we aware of these limitation? and do we documented how to migrate to logstash 8.0.0 for beats user I did not find anything here #11743

[nchaulet]

Actually I think I misconfigured logstash I missed the index part in the elasticsearch output so there is no issue my bad.

[nchaulet]

Hum looks like we have this issue when we add the index parameter Invalid data stream configuration, following parameters are not supported index ...

[jsvd]

The Elasticsearch output for Logstash allows for either using the index setting or the data streams parameters.
The data stream parameters are meant to guide users to the pre-existing data streams that ship with ES.
The index parameters is free form and just fills in the index value of a bulk action. So in practice it can write indices and to datastreams as bulk requests don't distinguish them.
However, we can't support both in a single output such as

elasticsearch {
   use_data_stream => true
   data_stream_type => metrics
   index => "my-random-index"
}

As this would be conflicting, it needs two separate outputs:

elasticsearch {
   use_data_stream => true
   data_stream_type => metrics
}
elasticsearch {
   index => "my-random-index"
}

[nchaulet]

@jsvd If we do not want to write to both output we have to create a new pipeline right? I think it could be a good suggestion we suggest for user that want to add elastic agent to an existing logstash configuration

[jsvd]

if it's a new deployment and the user is starting from scratch we should present a simple pipeline:

input {
  agent { 
     # ...
  }
}
filter {
  # TODO
}
output {
  elasticsearch {
    # use_data_streams => true (default)
    cloud_id =>
    api_key =>
  }
}

If the user is migrating from Logstash 7 to 8 we should warn that the ES output defaults to data streams and data needs to be ECS compatible.
a) if this isn't the problem (default assumption) then we can give a similar configuration to the first scenario
b) if the user doesn't want agent-driven data streams (logs/metrics/synthetics/etc), they will need to disable datastreams and set index => (which they probably already do in their existing configuration).

I recall you had a list of different scenarios, maybe we can go back to those and check if there are still open questions and perhaps close here and open an issue in the elasticsearch output plugin repository if there's something directly related to this plugin.

[nchaulet]

@jsvd > a) if the user doesn't want agent-driven data streams (logs/metrics/synthetics/etc), they will need to disable datastreams and set index => (which they probably already do in their existing configuration).

What about if the user has existing input like beats (not datastream driven( and it want to keep it, and add agent datastream, if we suggest the user to add the agent in a different pipeline it should work right?

[jsvd]

It would be equivalent to the start anew scenario, we can give users a simple pipeline of agent to ES that defaults to DS