This post is part of an ongoing series - see all the posts below.
In the previous part of this series, we looked at what filters are and covered some of the most commonly used ones, such as selectattr, map, and regex_replace. We also used filters to solve a real problem in the previous part by sharing the same syslog servers between Cisco and Arista.
Ansible has a huge number of filters, and it's impossible to cover all of them. So in this part, I want to focus on a handful that I find especially useful as a network engineer. We'll work with IP addresses and subnets, compare lists to check whether a device's config is what it should be, merge dictionaries together, and sort lists. Along the way, we'll also build a simple compliance check that you can use in your own network right away.
This post is part of a series on Ansible for network engineers. If you're new to the series, I recommend following the parts in order, since each one builds on the previous one. Here's a quick overview of the setup we're using. The lab has five switches, core1, core2, and access1, which are Cisco IOL, and access2 and access3, which are Arista cEOS. Each switch belongs to a vendor group and a role group in the inventory, and settings shared by every switch, such as the NTP servers, are defined in group_vars/all.yml.
---
all:
children:
cisco:
hosts:
core1:
ansible_host: 192.168.202.101
core2:
ansible_host: 192.168.202.102
access1:
ansible_host: 192.168.202.103
arista:
hosts:
access2:
ansible_host: 192.168.202.104
access3:
ansible_host: 192.168.202.105
core:
hosts:
core1:
core2:
access:
hosts:
access1:
access2:
access3:Working with IP Addresses
As network engineers, we deal with IP addresses all the time. We often need to pull the network address out of a prefix, convert a prefix length into a netmask, or check whether an IP address belongs to a certain subnet. Ansible has a filter for all of this, called ipaddr.
Installing netaddr
ipaddr comes from the ansible.utils collection, which is included in the ansible package. However, it relies on a Python package called netaddr to do the actual work with IP addresses, and that isn't installed by default. Make sure your virtual environment is activated, and install it using pip.
pip install netaddr
Splitting a String
Before we look at ipaddr, let's start with a simpler filter called split. It breaks a string into a list, using whatever separator you give it. For example, if we split 192.168.202.101/24 on the / character, we get the IP address and the prefix length as two separate items.
Create a new playbook called filters_ip.yml.
---
- name: ipaddr and split examples
hosts: localhost
gather_facts: false
vars:
mgmt_ip: 192.168.202.101/24
test_ips:
- 10.10.10.50
- 192.168.202.300
- not-an-ip
- 10.10.20.1
tasks:
- name: Split the IP and prefix
ansible.builtin.debug:
msg: "{{ mgmt_ip | split('/') }}"
- name: Get details from a prefix
ansible.builtin.debug:
msg:
- "Address: {{ mgmt_ip | ansible.utils.ipaddr('address') }}"
- "Prefix: {{ mgmt_ip | ansible.utils.ipaddr('prefix') }}"
- "Netmask: {{ mgmt_ip | ansible.utils.ipaddr('netmask') }}"
- "Network: {{ mgmt_ip | ansible.utils.ipaddr('network') }}"
- "Network with prefix: {{ mgmt_ip | ansible.utils.ipaddr('network/prefix') }}"
- "Broadcast: {{ mgmt_ip | ansible.utils.ipaddr('broadcast') }}"
- name: Keep only valid IPs
ansible.builtin.debug:
msg: "{{ test_ips | ansible.utils.ipaddr }}"
- name: Keep only IPs in 10.10.10.0/24
ansible.builtin.debug:
msg: "{{ test_ips | ansible.utils.ipaddr('10.10.10.0/24') }}"
We have two variables to work with. mgmt_ip is the management IP of core1 with its prefix, and test_ips is a list containing a mix of valid and invalid IP addresses. 192.168.202.300 isn't valid, since an octet can't be higher than 255, and not-an-ip obviously isn't an IP address at all.
Let's run the playbook, and then go through each task.
ansible-playbook filters_ip.yml
PLAY [ipaddr and split examples] **********************
TASK [Split the IP and prefix] ************************
ok: [localhost] => {
"msg": [
"192.168.202.101",
"24"
]
}
TASK [Get details from a prefix] **********************
ok: [localhost] => {
"msg": [
"Address: 192.168.202.101",
"Prefix: 24",
"Netmask: 255.255.255.0",
"Network: 192.168.202.0",
"Network with prefix: 192.168.202.0/24",
"Broadcast: 192.168.202.255"
]
}
TASK [Keep only valid IPs] ****************************
[WARNING]: Deprecation warnings can be disabled by
setting `deprecation_warnings=False` in ansible.cfg.
[DEPRECATION WARNING]: Instantiating filter PluginLoader
with aliases is deprecated...
[DEPRECATION WARNING]: Instantiating test PluginLoader
with aliases is deprecated...
ok: [localhost] => {
"msg": [
"10.10.10.50",
"10.10.20.1"
]
}
TASK [Keep only IPs in 10.10.10.0/24] *****************
ok: [localhost] => {
"msg": [
"10.10.10.50"
]
}
PLAY RECAP ********************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The first task splits mgmt_ip on the / character, giving us a list with the IP address and the prefix length as two separate items.
Getting Details from a Prefix
The second task is where ipaddr really shines. When you give ipaddr an argument, it returns a specific piece of information about the IP address or prefix.
- address - The IP address on its own, without the prefix length.
- prefix - The prefix length,
24in our case. - netmask - The prefix length converted into a subnet mask,
255.255.255.0. - network - The network address of the subnet.
- network/prefix - The network address together with the prefix length, which is handy for things like routing or ACL configs.
- broadcast - The broadcast address of the subnet.
Notice that the address result is the same as the first item from split. For a simple case like this, either one works. But ipaddr understands IP addresses, while split only sees text, so ipaddr is the safer choice when you're working with IP addresses.
There are plenty of other options too. If you check the documentation for ansible.utils.ipaddr, you'll find options for things like the first and last usable addresses in a subnet, or checking whether an address is public or private.
Filtering a List of IPs
When you use ipaddr on a list without an argument, it acts as a validator. It goes through the list and only keeps the items that are valid IP addresses. In the third task, 192.168.202.300 and not-an-ip were removed, leaving only the two valid IP addresses. This is useful for checking data before you push it to a device, for example, to catch a typo in a list of NTP servers.
You can also give it a subnet as the argument. In the last task, ipaddr('10.10.10.0/24') only keeps the IP addresses that fall within 10.10.10.0/24, so 10.10.20.1 was removed as well.
You'll also see a couple of deprecation warnings in the third task. These come from inside the ansible.utils collection itself, from the way it registers its filters, and not from anything in our playbook. They're safe to ignore, and they should go away in a future version of the collection.Building a Simple Compliance Check
One of the most common questions in any network is whether a device is configured the way it's supposed to be. Over time, someone adds an extra NTP server while troubleshooting, or forgets to configure one on a new switch, and nobody notices. Ansible can help you spot these differences, using two filters called difference and intersect.
- intersect - Takes two lists and returns the items that appear in both of them.
- difference - Takes two lists and returns the items that are in the first list, but not in the second one.
Let's use them to compare the NTP servers configured on core1 with the ones it should have. If you remember, the expected NTP servers are defined in group_vars/all.yml.
ntp_servers:
- server: 10.10.10.13
- server: 10.10.10.14
I rebuilt the lab before this section, so core1 started with no NTP config at all. Then, I manually configured one NTP server that's correct and one that shouldn't be there, and left 10.10.10.14 out on purpose.
core1(config)#ntp server 10.10.10.13
core1(config)#ntp server 10.10.10.99
Create a new playbook called ntp_compliance.yml.
---
- name: NTP compliance check
hosts: core1
gather_facts: false
tasks:
- name: Gather NTP config
cisco.ios.ios_facts:
gather_subset: min
gather_network_resources: ntp_global
- name: Save the current and expected NTP servers
ansible.builtin.set_fact:
current_ntp: "{{ ansible_facts['network_resources']['ntp_global']['servers'] | default([]) | map(attribute='server') | list }}"
expected_ntp: "{{ ntp_servers | map(attribute='server') | list }}"
- name: Display both lists
ansible.builtin.debug:
msg:
- "Current: {{ current_ntp }}"
- "Expected: {{ expected_ntp }}"
- name: Servers that are correct
ansible.builtin.debug:
msg: "{{ current_ntp | intersect(expected_ntp) }}"
- name: Servers that are missing
ansible.builtin.debug:
msg: "{{ expected_ntp | difference(current_ntp) }}"
- name: Servers that should not be there
ansible.builtin.debug:
msg: "{{ current_ntp | difference(expected_ntp) }}"
Let's go through it. The first task gathers the current NTP config from core1. In the facts part, we briefly looked at gather_network_resources, which reads the config of a specific feature as structured data. Here, we're using it to read the NTP config. We also set gather_subset to min, so we only collect the minimum set of general facts, since we don't need anything else.
The second task creates two new variables using set_fact. You can create as many variables as you like in a single set_fact task.
- current_ntp - The NTP servers currently configured on
core1. The NTP config is stored underansible_facts['network_resources']['ntp_global']['servers'], as a list of dictionaries, each with aserverkey. We usemapto pull out just the IP addresses. We also adddefault([]), so if a switch has no NTP servers configured at all, we get an empty list instead of an error. - expected_ntp - The NTP servers from
group_vars/all.yml. Again, we usemapto turn them into a simple list of IP addresses.
Now that both variables are simple lists of IP addresses, we can compare them. The last three tasks do exactly that, using intersect and difference.
ansible-playbook ntp_compliance.yml
PLAY [NTP compliance check] ***************************
TASK [Gather NTP config] ******************************
ok: [core1]
TASK [Save the current and expected NTP servers] ******
ok: [core1]
TASK [Display both lists] *****************************
ok: [core1] => {
"msg": [
"Current: ['10.10.10.13', '10.10.10.99']",
"Expected: ['10.10.10.13', '10.10.10.14']"
]
}
TASK [Servers that are correct] ***********************
ok: [core1] => {
"msg": [
"10.10.10.13"
]
}
TASK [Servers that are missing] ***********************
ok: [core1] => {
"msg": [
"10.10.10.14"
]
}
TASK [Servers that should not be there] ***************
ok: [core1] => {
"msg": [
"10.10.10.99"
]
}
PLAY RECAP ********************************************
core1 : ok=6 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The results match exactly what we configured.
- Correct -
intersectreturned10.10.10.13, since it's in both lists. - Missing -
expected_ntp | difference(current_ntp)returned10.10.10.14. It's in the expected list, but not on the switch. - Should not be there -
current_ntp | difference(expected_ntp)returned10.10.10.99. It's on the switch, but not in the expected list.
Notice that the last two tasks use the same filter, difference, with the lists swapped around. The order matters, since difference always returns what's in the first list but not in the second one.
The best part is that this playbook doesn't change anything on the device. It only reads the config and compares it, so it's completely safe to run against your entire network at any time. You could easily extend the same idea to syslog servers, VLANs, or anything else that has a resource module.
Merging Dictionaries with combine
The combine filter merges two dictionaries into one. If both dictionaries have the same key, the dictionary inside combine() takes precedence over the one before the pipe.
Let's say you have a set of default settings for every interface, and some extra settings for uplinks. Instead of repeating the defaults for every uplink, you can combine the two. Create a new playbook called filters_more.yml.
---
- name: combine and sort examples
hosts: localhost
gather_facts: false
vars:
default_interface:
mtu: 1500
shutdown: false
description: unused
uplink_interface:
description: to core1
mtu: 9000
interfaces:
- Ethernet1/3
- Ethernet0/1
- Loopback0
- Ethernet0/10
- Ethernet0/2
tasks:
- name: Combine two dictionaries
ansible.builtin.debug:
msg: "{{ default_interface | combine(uplink_interface) }}"
- name: Sort a list
ansible.builtin.debug:
msg: "{{ interfaces | sort }}"
ansible-playbook filters_more.yml
PLAY [combine and sort examples] **********************
TASK [Combine two dictionaries] ***********************
ok: [localhost] => {
"msg": {
"description": "to core1",
"mtu": 9000,
"shutdown": false
}
}
TASK [Sort a list] ************************************
ok: [localhost] => {
"msg": [
"Ethernet0/1",
"Ethernet0/10",
"Ethernet0/2",
"Ethernet1/3",
"Loopback0"
]
}
PLAY RECAP ********************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
In the first task, combine merged the two dictionaries into one. shutdown only exists in default_interface, so it was kept as it is. description and mtu exist in both, and in that case, the value from the second dictionary wins. So the uplink ends up with its own description and an MTU of 9000, while still getting shutdown: false from the defaults.
Sorting a List
The second task uses the sort filter, which sorts a list in order. It works well for most things, but there's one thing to be careful about with interface names. If you look at the output, Ethernet0/10 comes before Ethernet0/2.
This is because sort compares the names as text, one character at a time, rather than understanding them as interface numbers. When it compares Ethernet0/10 and Ethernet0/2, everything up to the / is the same, and then it compares 1 with 2. Since 1 comes first, Ethernet0/10 ends up before Ethernet0/2. This is fine if you just want a tidy list, but don't rely on it if the order of the interfaces actually matters.
Fixing the Config Automatically
Finding out what's wrong is useful, but it would be even better if Ansible could fix it for us. Since we already have the expected NTP servers in group_vars/all.yml, we can use the ios_ntp_global resource module to make the switch match them.
So far, whenever we've used a resource module, we've used the default state, which is merged. With merged, the module adds anything that's missing, but it never removes anything. In our case, that would add 10.10.10.14, but 10.10.10.99 would stay on the switch.
To remove it as well, we can use state: overridden. With overridden, the module makes the config on the device match exactly what's in the task. Anything that's missing gets added, and anything that isn't in the task gets removed.
Create a new playbook called ntp_fix.yml.
---
- name: Fix NTP config
hosts: core1
gather_facts: false
tasks:
- name: Make NTP servers match the expected list
cisco.ios.ios_ntp_global:
config:
servers: "{{ ntp_servers }}"
state: overridden
Since ntp_servers is already in the format ios_ntp_global expects, a list of dictionaries with a server key, we can pass it straight to servers, without any filters.
Before we run it, keep in mind that overridden replaces the entire NTP config on the device, not just the servers. We'll see exactly what that means at the end of this section.
Running in Check Mode First
We briefly looked at check mode earlier in the series. Let's take a closer look at it now, since this is exactly the kind of change where it's really useful.
When you run a playbook with --check, Ansible connects to the device and does everything it normally would, except actually making the change. The module still reads the current config, compares it with what you've asked for, and works out which commands it would need to send. But instead of sending them, it simply reports what would have happened.
If we add -v as well, we can see the full result of the task, including the commands.
ansible-playbook ntp_fix.yml --check -v
PLAY [Fix NTP config] *********************************
TASK [Make NTP servers match the expected list] *******
changed: [core1] => {
"after": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.99"}
]
},
"before": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.99"}
]
},
"changed": true,
"commands": [
"ntp server 10.10.10.14",
"no ntp server 10.10.10.99"
]
}
PLAY RECAP ********************************************
core1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
There are a few things to notice here.
- changed - The task shows
changed, which tells us that Ansible would make a change if we ran it for real. In check mode,changedmeans "would change", not "has changed". - commands - These are the exact commands Ansible would send to the switch. It would add
10.10.10.14and remove10.10.10.99, which is exactly what we want. - before and after - Both show the same two servers,
10.10.10.13and10.10.10.99. Since nothing was actually changed on the switch, the config after the task is the same as it was before.
At this point, nothing on core1 has changed. This gives you a chance to review the commands, and if something looks wrong, such as a command removing config you didn't expect, you can stop and fix the playbook before it touches the device.
I strongly recommend getting into the habit of running playbooks with --check -v first, especially when using states like overridden, or running against a lot of devices at once. Just keep in mind that check mode depends on the module supporting it. Most resource modules do, and you can confirm this in the Attributes section of the module documentation, under check_mode.
Applying the Change
Now that we're happy with the commands, let's run the playbook for real.
ansible-playbook ntp_fix.yml -v
PLAY [Fix NTP config] *********************************
TASK [Make NTP servers match the expected list] *******
changed: [core1] => {
"after": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.14"}
]
},
"before": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.99"}
]
},
"changed": true,
"commands": [
"ntp server 10.10.10.14",
"no ntp server 10.10.10.99"
]
}
PLAY RECAP ********************************************
core1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The commands are exactly the same as in check mode, which is what we'd expect. This time, though, after is different from before, since the change was actually made. 10.10.10.99 is gone, and 10.10.10.14 has been added.
We can confirm this on core1 by comparing the NTP config before and after the playbook.
core1#show run | sec ntp
ntp server 10.10.10.99
ntp server 10.10.10.13
core1#show run | sec ntp
ntp server 10.10.10.13
ntp server 10.10.10.14
Running the Compliance Check Again
Finally, let's run the compliance check again to make sure everything is now as it should be.
ansible-playbook ntp_compliance.yml
PLAY [NTP compliance check] ***************************
TASK [Gather NTP config] ******************************
ok: [core1]
TASK [Save the current and expected NTP servers] ******
ok: [core1]
TASK [Display both lists] *****************************
ok: [core1] => {
"msg": [
"Current: ['10.10.10.13', '10.10.10.14']",
"Expected: ['10.10.10.13', '10.10.10.14']"
]
}
TASK [Servers that are correct] ***********************
ok: [core1] => {
"msg": [
"10.10.10.14",
"10.10.10.13"
]
}
TASK [Servers that are missing] ***********************
ok: [core1] => {
"msg": []
}
TASK [Servers that should not be there] ***************
ok: [core1] => {
"msg": []
}
PLAY RECAP ********************************************
core1 : ok=6 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The current and expected lists now match. Both servers show up as correct, and the missing and extra lists are empty. You might notice that intersect returned the two servers in a different order from the original lists. That's normal, since intersect and difference don't keep the original order. If the order matters to you, you can add sort at the end.
This gives us a nice, simple workflow. First, run the compliance check to find out what's wrong. Then, run the fix in check mode to see what would change, and finally, apply the change and run the compliance check again to confirm. We'll take this idea further in a later part, where we'll make the compliance check fail automatically on any device that doesn't match. We'll also look at overridden and the other resource module states in more detail when we cover resource modules.
Be Careful with overridden
Earlier, I mentioned that overridden replaces the entire NTP config on the device, not just the servers. Let's see this in action, since it's an easy mistake to make. On core1, I configured an NTP source interface.
core1(config)#ntp source Loopback0
core1#show run | sec ntp
ntp source Loopback0
ntp server 10.10.10.13
ntp server 10.10.10.14
So core1 now has the two correct NTP servers, plus a source interface that isn't anywhere in our playbook or variables. Let's run ntp_fix.yml in check mode again.
ansible-playbook ntp_fix.yml --check -v
PLAY [Fix NTP config] *********************************
TASK [Make NTP servers match the expected list] *******
changed: [core1] => {
"after": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.14"}
],
"source": "Loopback0"
},
"before": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.14"}
],
"source": "Loopback0"
},
"changed": true,
"commands": [
"no ntp source Loopback0"
]
}
PLAY RECAP ********************************************
core1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The NTP servers are already correct, so there are no ntp server commands this time. But the task still shows changed, and if you look at commands, Ansible wants to run no ntp source Loopback0. Our playbook never mentioned the source interface, but since we're using overridden, anything that isn't in the task gets removed, including settings we didn't even think about.
Also notice that after still shows source: Loopback0. Remember, in check mode, nothing is changed on the device, so after shows the config as it currently is. In check mode, commands is the part to focus on.
Without check mode, we wouldn't have known that the source interface was about to be removed. To show what happens, let's run the playbook for real.
ansible-playbook ntp_fix.yml -v
PLAY [Fix NTP config] *********************************
TASK [Make NTP servers match the expected list] *******
changed: [core1] => {
"after": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.14"}
]
},
"before": {
"servers": [
{"server": "10.10.10.13"},
{"server": "10.10.10.14"}
],
"source": "Loopback0"
},
"changed": true,
"commands": [
"no ntp source Loopback0"
]
}
PLAY RECAP ********************************************
core1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
This time, before still has the source interface, but after doesn't, since the change was actually made. We can confirm it on core1.
core1#show run | sec ntp
ntp server 10.10.10.13
ntp server 10.10.10.14
This post is part of an ongoing series - see all the posts below.
