[Feature Request] Concurrent executing query in InnerHitsPhase

[kkewwei]

Is your feature request related to a problem? Please describe

In our product environment, we utilize the nested field and inner_hits, the source contains excessive number of fields, and inner_hits query will match about 80 sub documents within a parent document. we find that the FetchPhase is costly, it will cost 3s+, but when we delete the inner_hits, it will only cost 700ms.

In FetchPhase, each document will execute the InnerHitsPhase serially (regards as sub query phase), if we need to
fetch values from multi documents in FetchPahse, then the overall fetch phase will be very slow.

Describe the solution you'd like

Concurrent executing inner_hits in FetchPhase.

In some case, if the source is too large, it will also cost much time to fetch document in FetchPhase, It also seem necessary to fetch doc concurrently in some scenarios.

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[sandeshkr419]

@jed326 / @sohami - Do you guys have some thoughts here?

[jed326]

Thanks for the feature request @kkewwei!

Concurrent fetch is something I had briefly discussed with @sohami before, but it's not something we saw any use cases for (until now). When we were looking at concurrent search the performance on aggregations is what we were focusing on, and in a lot of cases users will do aggregations with either size=0 or just the default size so in those so concurrent fetch was not really needed or useful.

I'd be happy to review any designs or PRs for this!

[kkewwei]

@jed326 / @sohami - Do you guys have some thoughts here?

Thanks for the feature request @kkewwei!

Concurrent fetch is something I had briefly discussed with @sohami before, but it's not something we saw any use cases for (until now). When we were looking at concurrent search the performance on aggregations is what we were focusing on, and in a lot of cases users will do aggregations with either size=0 or just the default size so in those so concurrent fetch was not really needed or useful.

I'd be happy to review any designs or PRs for this!

@jed326, If you haven't started on it, can you assign it to me? I'm pleased to have a try. Of course, I will make a draft first

[jed326]

can you assign it to me?

@kkewwei done!

[kkewwei]

It seems that serial execution of each document's innerhit can speed up due to querycache, while concurrent executing will result in performance degradation. I'm still testing.