Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pagination in QueryWithContinuation (dual continuation) #69

Open
CXuesong opened this issue May 23, 2020 · 0 comments
Open

Support pagination in QueryWithContinuation (dual continuation) #69

CXuesong opened this issue May 23, 2020 · 0 comments

Comments

@CXuesong
Copy link
Owner

CXuesong commented May 23, 2020

The current implementation of QueryWithContinuation, or the related WikiPagePropertyProvider<T> parameter classes, are not design to support pagination inside page properties (&prop= values, such as &props=revision).

Though there is some basic logic in RefreshPagesAsync to merge prop list when there are some props need pagination, we have not introduced proper public API to let consumer adjust the pagination size of props itself. IWikiPagePropertyProvider<T>.GetMaxPaginationSize is used to restrict the max allowed title count (e.g. count of title= values), instead of props item count (e.g. x in rvlimit=x).

The justification for not breaking the wall between WikiPagePropertyList<T> (calls QueryWithContinuation) and WikiPagePropertyProvider before is the potential complexity to deal with dual continuation (#68 (comment)). But perhaps we may break the wall, and make it possible for consumer to plug IWikiPagePropertyProvider into WikiPagePropertyList instead of deriving from the latter one.

Before this feature has been implemented, for example, you may consider the following approach to get the complete page property list when the pages are taken from another wikilist/generator (#68 (comment))

  • Use CategoryMembersGenerator.ListItemAsync to get a sequence of WikiPageStubs,
  • Convert WikiPageStubs into WikiPages,
  • Refresh WikiPage sequence by batches (e.g., pywikibot batches pages by 30 items)
@CXuesong CXuesong added this to the Future milestone May 23, 2020
@CXuesong CXuesong changed the title Support pagination in QueryWithContinuation Support pagination in QueryWithContinuation (dual continuation) May 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant