Today we're talking about how to use 'NextToken' in AWS API calls, specifically when using the Boto3 Python SDK. If you're dealing with long lists of items from AWS services, this is the guide for you.
I'm sure many of you are familiar with Boto3. For those who aren't, it's a Python SDK that allows us to interact with AWS services. Think of it as a tool that lets us automate all the stuff we can do manually in the AWS Console, but using Python.
So, why are we talking about this 'NextToken'? Well, when we're dealing with AWS services that have lots of items to list—say, files in an S3 bucket or topics in SNS—AWS usually breaks it down into smaller chunks. The 'NextToken' is essentially our bookmark. It helps us keep track of where we left off so we can loop through everything without missing out.
Before we jump in, this post assumes you have some knowledge of AWS and basic Python skills.
Example - List SNS Topics
Let's move on to the example. I've chosen SNS topics to demonstrate how 'NextToken' functions. Of course, you can use this approach with any AWS service.
In this example, I've created 103 SNS topics. When you call the API to list these topics, it defaults to showing only the first 100. Along with this list, the API response will include a key called 'NextToken', which is a string of data we'll use later.
So you might be wondering about the other three topics that we didn't see. To view them, you'll need to make another API call. This time, include the 'NextToken' from the previous response, and you'll see the remaining three topics.
# First request import boto3 client = boto3.client('sns') response = client.list_topics() print(len(response['Topics'])) print(response.get('NextToken', 'No more Topics'))
First, we import the Boto3 library and set up an SNS client. Then, we make an API call using
client.list_topics() to get a list of SNS topics. I've included a
Topics list, which will indicate how many topics we've retrieved. We also print out the value of 'NextToken' if it exists; otherwise, it'll print 'No more Topics'. This 'NextToken' is what we'll use to get the rest of the topics in our next API call.
# Output 100 AaEdHpfgQAoxULcs1t7ZBSsdf5G4erROSy5yp8Lmk1sdf=
When you run the script
next_token_ex.py, you see two lines in the output. The first line, which shows '100', tells us that we've successfully pulled 100 SNS topics from the API call.
The second line, that string of characters, is the 'NextToken'. This token is our key to accessing the remaining topics that didn't show up in the first API call. We'll use this 'NextToken' in our next API request to get the rest of the topics.
So, what we've learned is that the API gave us the first 100 topics and a 'NextToken' to continue where we left off. On to the next part to see how to use this token.
import boto3 client = boto3.client('sns') response = client.list_topics( NextToken="AaEdHpfgQAoxULcs1t7ZBSsdf5G4erROSy5yp8Lmk1sdf=" ) print(len(response['Topics'])) print(response.get('NextToken', 'No more Topics'))
Let's get into the second request. In this snippet, the code is pretty much the same as the first one, but with a small difference. We're now including the
NextToken in our
client.list_topics() API call. This token is the one we got from our first request.
3 No more Topics
When you run this updated script, the output shows '3' as the first line. That tells us that we've successfully retrieved the remaining three SNS topics that were not included in the first API call.
The second line in the output says 'No more Topics', indicating that we've reached the end of our list. There's no more 'NextToken', meaning we've successfully looped through all the topics.
This is Just too Hard to Manage
You might be thinking, "Why go through all this trouble with multiple API calls? I might as well just get this information from the AWS Console." And you're right; running these scripts multiple times can feel a bit cumbersome.
But don't worry, there's a more efficient way to handle this. In our next example, we'll use a
while loop to automate the process and make it much smoother.
Using a While Loop
Let's dive into this Python code and understand how it fetches all SNS topics using the
boto3 library. The output "103" indicates that all 103 topics have been successfully retrieved, but how did we get there? Here's how each part of the code contributes.
import boto3 client = boto3.client('sns') next_token = None all_topics =  while True: if next_token: response = client.list_topics(NextToken=next_token) else: response = client.list_topics() all_topics.extend(response['Topics']) next_token = response.get('NextToken', None) if next_token is None: break print(len(all_topics))
# Output 103
We have two new variables to begin with,
next_token, set to
None, and an empty list called
all_topics. We enter a
while True loop to keep fetching topics until there are no more. Inside the loop, we check if we have a
next_token. If we do, we use it to get the next set of topics. Otherwise, we get the first set of topics. We add these topics to our
all_topics list using
Then, we check for a new
NextToken and update
next_token accordingly. If there's no new
NextToken, we break out of the loop. Finally, we print the length of
all_topics to see how many topics we've collected. In this example, we end up with 103, which means we've successfully grabbed all the SNS topics.
A Better Approach - Paginator
Let's dive into a better way of doing things using a Paginator. Paginator is a feature in the
boto3 Python SDK, and it's used for fetching multiple pages of data from AWS. Instead of you doing the heavy lifting with
NextToken, Paginator does it for you. All you do is set it up once and then loop through the pages. It fetches and compiles all the data for you.
Paginators are created via the
get_paginator() method of a boto3 client. The
get_paginator() method accepts an operation name and returns a reusable
Paginator object. You then call the
paginate method of the Paginator, passing in any relevant operation parameters to apply to the underlying API operation. The
paginate method then returns an iterable
Here is another script that uses a paginator but gives the same results using a While loop.
import boto3 client = boto3.client('sns') paginator = client.get_paginator('list_topics') all_topics =  for page in paginator.paginate(): all_topics.extend(page['Topics']) print(len(all_topics))
# Output 103
In this code, we use the
boto3 Python SDK to create an SNS client and then create a Paginator object using the
get_paginator('list_topics') method. The Paginator object is a specialized class in
boto3 designed to handle API pagination automatically. We then call the
paginate() method on the Paginator object to iterate through each page of the API results.
Within the loop, the
all_topics list is extended with the topics from the current page using the
extend method. This allows us to accumulate all topics from multiple API calls into a single list, without having to manually manage