[Bug] Pagination Error

[em1064]

Describe the bug

Pagination works fine as long as the pages queried have data.

As soon as the page has no data, then this error will occur.

error getting data frame. unable to merge fields due to different fields
(Trace ID: 763b0b20f3281df1fac45df608e94594)

Version Details:

• Grafana version : v10.3.0-63079
• Plugin version : 2.1.0

Additional context

Other related Nice to haves

• 5 page max is very small. The given argument for the limitation:

  • The limitation to the number of maximum pages per query is limited to 5. This is to avoid resource exhaustion and DDOSing of the target system.
  • I do not see how valid this is as you can achieve the same by duplicating the Query and changing the offset values manually. I got to 102 queries before I gave up. Then all is needed is to merge the queries into one when transforming data.
  • If there must be a limit, 100-500 seems fine. Especially when using List of Values Pagination type where the field value could take a Custom Variable in CSV.
    

• Would be great if Max pages supported variables for a dynamic value. Most APIs will have an endpoint to give a total count, which can be used to calculate numbers of pages to query.

[LudwigJan]

I am facing the same error:

Grafana version : 10.2.0-60982
Plugin version : 2.1.0

[yesoreyeram]

I am able to reproduce the behaviour when no results available in one of the pages. We can fix this in future iterations.

BTW, there is no plans to increase the max pages for the original reasons. ( you can somehow trigger 1000 pages. but we don't want to make it very obvious )

[justmedude]

Having a similar issue. I'm not a golang person (yet), but I'm thinking that this could be handled in:
https://github.com/yesoreyeram/grafana-plugins/blob/main/lib/go/transformations/merge.go

Maybe somewhere after line 26:

if len(outFrame.fields) == 0 {
  continue
}

[ceccillia]

Having same issue here. It would be nice to have Max Pages as dynamic variables support field or skip pages with empty json.
My reproduce scenario:
{ "startAt": 0, "maxResults": 1000, "total": 3355, "issues": [] }
If I start at >3355 issue number - I have empty output in issues. But I cant change maxResults every time, because I have dynamic URL