[Performance] Suspicious _msearch request might be slowing down the display of dashboards

[mastropi]

@mbourgue Possible performance issue discussed last week on the phone.

In dashboards, there seems to be an _msearch request that is not related to the dashboard request.

This is highlighted in the first screenshot below where the execution time of 2.5 sec and the POST payload for the usual INCOME dashboard are enclosed in red boxes. Note that the POST payload is very short as it corresponds to just one visualization of the visualizations present in the dashboard.

The correct _msearch request associated to the actual dashboard request is highlighted in the second screenshot where the execution time of 20 sec and the POST payload containing the dashboard request are enclosed in green boxes.

The question is: is the first _msearch request really necessary, and if not, is it slowing down the actual _msearch associated to the dashboard request?

1) Apparently unnecessary _msearch request

2) Actual dashboard _msearch request

[mastropi]

Note that this "phantom" request distorts the query statistics which are used by the dashboard acceleration process, making the query appear much more requested than other queries. (The "phantom" query tends to be the same at every new request and it is not clear when it changes. I have also observed the phantom request no longer be there, but I cannot tell what causes its disappearance).