Today, we're diving deep into a really useful feature of the AWS
boto3 Python SDK, the Paginator. If you've ever struggled with dealing with paged API results using
NextToken, then this post is for you. We'll use SNS topics as our example to show you how it works. We'll also see how it compares to using
NextToken. Let's get started.
What is a Paginator?
Some AWS operations return results that are incomplete and require subsequent requests in order to get the full response. The process of sending subsequent requests to continue where a previous request left off is called pagination.
Paginators are a feature of boto3 that acts as an abstraction over the process of iterating over an entire result set of a truncated API operation.
A Side-by-Side Comparison of NextToken and Paginator with SNS Topics
Let's go straight into the example. I've chosen SNS topics to demonstrate how Paginator and NextToken work. Of course, you can use this approach with any AWS service. As you can see below, I have created 103 SNS topics for this example.
First, let's dive into an example using
NextToken to list SNS topics in AWS. We're using Python's
boto3 library, and here's how it goes.
import boto3 client = boto3.client('sns') next_token = None all_topics =  while True: if next_token: response = client.list_topics(NextToken=next_token) else: response = client.list_topics() all_topics.extend(response['Topics']) next_token = response.get('NextToken', None) if next_token is None: break print(len(all_topics))
# Output 103
In this code, we create an SNS client and then set up a
while loop to keep making API calls until there are no more pages left to retrieve. We use a variable called
next_token to keep track of where we are. The
list_topics() method fetches the topics.
None, so the first API call fetches the first 100 topics. We store these topics in the
all_topics list using the
extend() method. Then, we check for the presence of
NextToken in the API response. If it's there, we use it in the next API call to get the remaining topics. The loop continues until
NextToken is no longer returned, meaning we've fetched all topics. At the end, we print the total number of topics gathered.
So, we start with the first 100 topics and then grab the remaining 3 in our case in the subsequent API call. All in all, it's a manual process but gets the job done. Next up, let's see how Paginator can make this even easier.
Paginators are created via the
get_paginator() method of a boto3 client. The
get_paginator() method accepts an operation name and returns a reusable
Paginator object. You then call the
paginate method of the Paginator, passing in any relevant operation parameters to apply to the underlying API operation. The
paginate method then returns an iterable
import boto3 client = boto3.client('sns') paginator = client.get_paginator('list_topics') all_topics =  for page in paginator.paginate(): all_topics.extend(page['Topics']) print(len(all_topics))
# Output 103
In this example, we're using the Paginator feature available in the
boto3 Python SDK for AWS. We start by creating an SNS client, just like before. This time, though, we use the
get_paginator method to get a Paginator object for the 'list_topics' operation. The Paginator object is responsible for handling the pagination logic for us.
We then initialize an empty list called
all_topics to store our results. The Paginator object's
paginate() method returns an iterator, so we loop through each page using a
for loop. Within the loop, we extend our
all_topics list with the topics found in the current page, which are stored in
page['Topics']. Finally, we print out the total number of topics collected. The Paginator takes care of dealing with
NextToken behind-the-scenes, making the code cleaner and more straightforward.
NextToken vs Paginator
In summary, both NextToken and Paginator serve the same purpose: they help you navigate through paginated API responses. If you like having full control and don't mind managing
NextToken yourself, then using NextToken can be a straightforward approach.
However, if you prefer a cleaner, more automated way to handle pagination, then Paginator is the way to go. It abstracts away the complexities of dealing with
NextToken and makes your code easier to read and maintain. Both methods have their merits, so the choice between them depends on your specific needs and coding style.