datastore: re-use query object for continuation queries #1635
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1634
We've been calculating continuation query properties, like
limit, from the original query the user provided. We should be calculating those properties from the last query that ran.Example scenario: Datastore seems to return results in chunks of 300. So, with a dataset with 1,000 entities, imagine a query for just 600 of them:
How our code works behind the scenes:
NOT_FINISHEDmessageNOT_FINISHED, we build a new query that sets the new limit to 300:original_limit (600) - results_returned (300)
MORE_RESULTS_AFTER_LIMITmessageNow imagine a query for 601:
How our code works behind the scenes:
NOT_FINISHEDmessageNOT_FINISHED, we build a new query that sets the new limit to 301:original_limit (601) - results_returned (300)
NOT_FINISHEDmessageNOT_FINISHED, we build a new query that sets the new limit to 301:original_limit (601) - results_returned (300)
NOT_FINISHEDmessageNOT_FINISHED, we build a new query that sets the new limit to 301:original_limit (601) - results_returned (300)
NO_MORE_RESULTSmessageThe problem above is the formula,
original_limit-results_returned. It should have beenlast_query_limit-results_returned. So now, we clone the query up front, and modify it as we continue to make queries, so that these calculations are accurate.