pipeline parameter for Elasticsearch 5+

[danizen]

Elasticsearch 5+ supports ingestion pipelines. They are similar to taggers. Probably useful to support.

[essiembre]

Couldn't they be set up independently of crawlers? You envision them as Committer configuration options?

[danizen]

Yup - just a pipeline configuration option that is passed to Elasticsearch to invoke that pipeline.
I don't need it, like, at all. Because taggers are more flexible than Elasticsearch pipelines. However, someone might.

I find the pipelines useful when fixing up data, e.g. do a query on all documents having a field "openi.error" and "openi.summary", and drop the "openi.error" column. Then, I can use an update_by_query that runs the pipeline.

Pipelines are more useful when someone doesn't have a norconex crawler, but since they can create javascript functions inside Elasticsearch, they may be committed to using it for some reason.

So, it is just a completeness of support thang that I didn't want to miss.