Working with Filters in Ansible

We've already used a filter a few times in this series without spending much time on it. In the loops part, we used dict2items to turn a dictionary into a list, so we could loop over it. We also ran into a couple of problems that we couldn't solve at the time, such as sharing the same list of syslog servers between Cisco and Arista, even though their modules expect the data in a different format.

Filters are what solve these kinds of problems. In this part, we'll look at what filters are, go through some of the most useful ones, and see how to chain them together. By the end of it, we'll come back to the syslog problem and fix it using filters.

Before we start, I rebuilt the lab once again, so all the config from the previous part is gone. The project files and inventory are the same as before.

If you're reading this part without reading the previous ones, here's a quick summary of the lab. We have five switches. core1, core2, and access1 are Cisco IOL switches, while access2 and access3 are Arista cEOS switches. In the inventory, each switch belongs to a vendor group, cisco or arista, and a role group, core or access. The connection settings for each vendor are in group_vars/cisco/vars.yml and group_vars/arista/vars.yml, and settings shared by every switch are in group_vars/all.yml. As always, your lab doesn't need to match mine exactly, any mix of Cisco and Arista devices will work.

What Is a Filter?

A filter takes some data, changes it in some way, and gives you back the result. The data could be a string, a number, a list, or a dictionary, and the change could be anything from converting text to uppercase to picking out specific items from a list.

Filters come from Jinja2, the same templating language that gives us the double curly brackets for variables. You apply a filter using the | symbol, which is called a pipe. The data goes on the left, and the filter goes on the right.

{{ data | filter }}

You can also chain multiple filters together. The output of the first filter becomes the input of the next one, and so on, from left to right.

{{ data | filter1 | filter2 | filter3 }}

If you've used pipes on the Linux command line, or show run | include on a Cisco device, the idea is exactly the same.

A Few Simple Filters

Let's start with a few simple filters, to see how they work. Since we're only working with data, we'll use localhost, so we don't need to connect to any devices. Create a new playbook called filters_basic.yml.

---
- name: Simple filters
  hosts: localhost
  gather_facts: false

  vars:
    hostname: core1
    ntp_list:
      - 10.10.10.13
      - 10.10.10.14

  tasks:
    - name: Convert to uppercase
      ansible.builtin.debug:
        msg: "{{ hostname | upper }}"

    - name: Count the NTP servers
      ansible.builtin.debug:
        msg: "There are {{ ntp_list | length }} NTP servers"

    - name: Use a default value
      ansible.builtin.debug:
        msg: "SNMP location is {{ snmp_location | default('not set') }}"

ansible-playbook filters_basic.yml

PLAY [Simple filters] *********************************

TASK [Convert to uppercase] ***************************
ok: [localhost] => {
  "msg": "CORE1"
}

TASK [Count the NTP servers] **************************
ok: [localhost] => {
  "msg": "There are 2 NTP servers"
}

TASK [Use a default value] ****************************
ok: [localhost] => {
  "msg": "SNMP location is not set"
}

PLAY RECAP ********************************************
localhost  : ok=3  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

Let's go through each one.

• upper - Converts a string to uppercase, so core1 becomes CORE1. There's also lower, which does the opposite.
• length - Counts the number of items in a list. Our ntp_list has two items, so it returns 2. It also works with strings, where it counts the number of characters.
• default - Provides a value to use if the variable isn't defined. We never defined snmp_location, so instead of failing with an error, Ansible uses the default value, not set.

In the conditionals part, we used is defined to skip a task when a variable wasn't defined. default solves a similar problem in a different way. Instead of skipping the task, it gives the variable a fallback value, and the task carries on.

Filtering a List with selectattr and rejectattr

The filters we've seen so far are simple, but filters really become useful when you're working with lists of dictionaries. Two of the filters I use most often are selectattr and rejectattr.

• selectattr - Goes through a list of dictionaries and keeps only the items where a certain key matches a condition.
• rejectattr - Does the opposite. It removes the items that match the condition and keeps everything else.

To try them out, we need some data to filter. Let's use the interface facts from core1, which we've worked with a few times now. First, I shut down two of the unused interfaces on core1, so not every interface is up.

core1(config)#interface Ethernet1/2
core1(config-if)#shutdown
core1(config-if)#interface Ethernet1/3
core1(config-if)#shutdown

Next, create a new playbook called filters_select.yml.

---
- name: selectattr examples
  hosts: core1
  gather_facts: false

  tasks:
    - name: Gather interface facts
      cisco.ios.ios_facts:
        gather_subset: interfaces

    - name: Display interfaces that are up
      ansible.builtin.debug:
        msg: "{{ item.key }} is up"
      loop: "{{ ansible_facts['net_interfaces'] | dict2items | selectattr('value.operstatus', '==', 'up') | list }}"
      loop_control:
        label: "{{ item.key }}"

    - name: Display interfaces that are not up
      ansible.builtin.debug:
        msg: "{{ item.key }} is {{ item.value.operstatus }}"
      loop: "{{ ansible_facts['net_interfaces'] | dict2items | rejectattr('value.operstatus', '==', 'up') | list }}"
      loop_control:
        label: "{{ item.key }}"

The loop line in the second task chains four filters together, so let's break it down from left to right.

ansible_facts['net_interfaces'] | dict2items | selectattr('value.operstatus', '==', 'up') | list

1. ansible_facts['net_interfaces'] - The interface facts, which is a dictionary.
2. dict2items - Converts the dictionary into a list, where each item has a key (the interface name) and a value (its details), just like we saw in the loops part.
3. selectattr('value.operstatus', '==', 'up') - Keeps only the items where value.operstatus is equal to up. The first argument is the key to check, and you can use a dot to go into nested dictionaries. The second is the test, ==, and the third is the value to compare against.
4. list - Turns the result into a list. selectattr doesn't return a list on its own, so we add list at the end to make sure we get one. You'll see this at the end of a lot of filter chains.

The third task is exactly the same, except it uses rejectattr instead, so it keeps the interfaces where operstatus is not up.

You might also see equalto used instead of == in some examples, such as selectattr('value.operstatus', 'equalto', 'up'). Both work exactly the same, but I find == easier to read, since it's the same as what we used with when in the conditionals part. Let's run the playbook.

ansible-playbook filters_select.yml

PLAY [selectattr examples] ****************************

TASK [Gather interface facts] *************************
ok: [core1]

TASK [Display interfaces that are up] *****************
ok: [core1] => (item=Ethernet0/0) => {
  "msg": "Ethernet0/0 is up"
}
ok: [core1] => (item=Ethernet0/1) => {
  "msg": "Ethernet0/1 is up"
}
ok: [core1] => (item=Ethernet0/2) => {
  "msg": "Ethernet0/2 is up"
}
ok: [core1] => (item=Ethernet0/3) => {
  "msg": "Ethernet0/3 is up"
}
ok: [core1] => (item=Ethernet1/0) => {
  "msg": "Ethernet1/0 is up"
}
ok: [core1] => (item=Ethernet1/1) => {
  "msg": "Ethernet1/1 is up"
}
ok: [core1] => (item=Loopback0) => {
  "msg": "Loopback0 is up"
}

TASK [Display interfaces that are not up] *************
ok: [core1] => (item=Ethernet1/2) => {
  "msg": "Ethernet1/2 is administratively down"
}
ok: [core1] => (item=Ethernet1/3) => {
  "msg": "Ethernet1/3 is administratively down"
}

PLAY RECAP ********************************************
core1  : ok=3  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

The first loop only contains the interfaces that are up, and the second loop only contains the two interfaces we shut down. Notice that the status of a shut interface is administratively down, not just down. This is a good example of why rejectattr is useful here. By rejecting everything that's up, we catch any interface that isn't up, whatever its status is, without having to know every possible value in advance.

If you compare this with the loops part, we did something similar there using when inside the loop. The difference is that with when, Ansible loops over every interface and skips the ones that don't match, so you see skipping in the output. With selectattr and rejectattr, the list is filtered before the loop starts, so the loop only runs for the items we actually want.

Extracting Values with map

selectattr and rejectattr let us pick which items to keep from a list. But sometimes, we don't need the whole item, just one value from it. For example, we might only want the interface names, rather than the full dictionary with all the details. The map filter helps with this.

map goes through every item in a list and does something to it. It's commonly used in two ways.

• map(attribute='key') - Pulls out a single key from each dictionary in the list.
• map('filter') - Applies another filter to every item in the list.

Let's see both in action with a small list of neighbors. Create a new playbook called filters_map_basic.yml.

---
- name: map examples
  hosts: localhost
  gather_facts: false

  vars:
    neighbors:
      - name: access1
        ip: 172.16.1.0
      - name: access2
        ip: 172.16.1.2
      - name: core2
        ip: 172.16.1.13

  tasks:
    - name: Extract the names
      ansible.builtin.debug:
        msg: "{{ neighbors | map(attribute='name') | list }}"

    - name: Extract the IPs
      ansible.builtin.debug:
        msg: "{{ neighbors | map(attribute='ip') | list }}"

    - name: Apply a filter to every item
      ansible.builtin.debug:
        msg: "{{ neighbors | map(attribute='name') | map('upper') | list }}"

ansible-playbook filters_map_basic.yml

PLAY [map examples] ***********************************

TASK [Extract the names] ******************************
ok: [localhost] => {
  "msg": [
    "access1",
    "access2",
    "core2"
  ]
}

TASK [Extract the IPs] ********************************
ok: [localhost] => {
  "msg": [
    "172.16.1.0",
    "172.16.1.2",
    "172.16.1.13"
  ]
}

TASK [Apply a filter to every item] *******************
ok: [localhost] => {
  "msg": [
    "ACCESS1",
    "ACCESS2",
    "CORE2"
  ]
}

PLAY RECAP ********************************************
localhost  : ok=3  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

In the first two tasks, we start with a list of dictionaries and end up with a simple list of values, the names in the first task and the IPs in the second.

The third task chains two map filters together. The first one pulls out the names, just like before. The second one, map('upper'), applies the upper filter to every name in the list. On its own, upper works on a single string, but with map, we can apply it to every item in a list at once. Keep this one in mind, since we'll use the same idea later in this part.

Just like selectattr, map doesn't return a list on its own, so we add list at the end.

Saving the Result with set_fact

Now let's put a few filters together, using the interface facts from core1 again. This time, we want a simple list containing only the names of the interfaces that aren't up, which we can then reuse in other tasks.

So far, we've only used filters directly inside a task. But if you need the result in more than one task, repeating the same long filter chain each time isn't great. The set_fact module lets you save the result as a new variable, which you can then use in any task that comes after it.

Create a new playbook called filters_map.yml.

---
- name: map and set_fact examples
  hosts: core1
  gather_facts: false

  tasks:
    - name: Gather interface facts
      cisco.ios.ios_facts:
        gather_subset: interfaces

    - name: Save the names of interfaces that are not up
      ansible.builtin.set_fact:
        down_interfaces: "{{ ansible_facts['net_interfaces'] | dict2items | rejectattr('value.operstatus', '==', 'up') | map(attribute='key') | list }}"

    - name: Display the list
      ansible.builtin.debug:
        var: down_interfaces

    - name: Display a summary
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }} has {{ down_interfaces | length }} interfaces that are not up: {{ down_interfaces | join(', ') }}"

In the second task, set_fact creates a new variable called down_interfaces. Its value comes from a chain of filters, which we can read from left to right.

ansible_facts['net_interfaces'] | dict2items | rejectattr('value.operstatus', '==', 'up') | map(attribute='key') | list

1. dict2items - Converts the interface facts into a list, where each item has a key and a value.
2. rejectattr('value.operstatus', '==', 'up') - Removes every interface that's up, leaving only the ones that aren't.
3. map(attribute='key') - Pulls out just the key from each item, which is the interface name.
4. list - Turns the result into a list.

So we start with a dictionary containing every interface and all its details, and end up with a simple list of interface names.

The last task uses the new variable with two filters. We've already seen length, which counts the interfaces. The other one is new. join takes a list and turns it into a single string, with each item separated by whatever you put inside the brackets, a comma and a space in this case. Let's run the playbook.

ansible-playbook filters_map.yml

PLAY [map and set_fact examples] **********************

TASK [Gather interface facts] *************************
ok: [core1]

TASK [Save the names of interfaces that are not up] ***
ok: [core1]

TASK [Display the list] *******************************
ok: [core1] => {
  "down_interfaces": [
    "Ethernet1/2",
    "Ethernet1/3"
  ]
}

TASK [Display a summary] ******************************
ok: [core1] => {
  "msg": "core1 has 2 interfaces that are not up: Ethernet1/2, Ethernet1/3"
}

PLAY RECAP ********************************************
core1  : ok=4  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

The set_fact task shows ok, since it doesn't change anything on the device. It only creates a variable on the control node. The next two tasks then use down_interfaces just like any other variable. We get a clean list of the interfaces that aren't up, and a one-line summary built from that list.

Keep in mind that a variable created with set_fact belongs to the device it was created for. If we ran this playbook against all three Cisco switches, each switch would get its own down_interfaces variable, based on its own interface facts.

Sharing Syslog Between Vendors

Now we have everything we need to solve the problem we ran into in the previous part. If you remember, we wanted to define our syslog servers once and use them for both Cisco and Arista. The problem was that the two modules expect the data in a different format.

# Cisco (ios_logging_global)
- host: 10.10.10.50

# Arista (eos_logging_global)
- name: 10.10.10.50

Both of them want a list of dictionaries, but Cisco uses host as the key, while Arista uses name. So in the previous part, we defined syslog_servers twice, once in each vendor's group_vars file. It worked, but the same IP addresses were stored in two places. If a syslog server ever changed, we'd need to remember to update both files.

A better approach is to store just the IP addresses once, as a simple list, and convert them into the right format for each vendor inside the playbook.

The dict_kv Filter

To do the conversion, we'll use a filter called dict_kv. It takes a single value and turns it into a dictionary, using whatever key you give it. For example, '10.10.10.50' | community.general.dict_kv('host') gives us {'host': '10.10.10.50'}.

dict_kv comes from the community.general collection, which is included in the ansible package, so there's nothing extra to install. Since it isn't a built-in filter, we use its full name, community.general.dict_kv.

On its own, dict_kv only works on a single value. But we have a list of IP addresses, so we combine it with map, just like we did with map('upper') in the previous section. map applies dict_kv to every IP in the list.

Let's see the conversion on its own first. Create a new playbook called conversion.yml.

---
- name: dict_kv example
  hosts: localhost
  gather_facts: false

  vars:
    syslog_ips:
      - 10.10.10.50
      - 10.10.10.51

  tasks:
    - name: Cisco format
      ansible.builtin.debug:
        msg: "{{ syslog_ips | map('community.general.dict_kv', 'host') | list }}"

    - name: Arista format
      ansible.builtin.debug:
        msg: "{{ syslog_ips | map('community.general.dict_kv', 'name') | list }}"

When you use map to apply a filter that needs an argument, the filter name goes first, followed by its arguments, all separated by commas. So map('community.general.dict_kv', 'host') means "apply dict_kv with host to every item in the list".

ansible-playbook conversion.yml

PLAY [dict_kv example] ********************************

TASK [Cisco format] ***********************************
ok: [localhost] => {
  "msg": [
    {
      "host": "10.10.10.50"
    },
    {
      "host": "10.10.10.51"
    }
  ]
}

TASK [Arista format] **********************************
ok: [localhost] => {
  "msg": [
    {
      "name": "10.10.10.50"
    },
    {
      "name": "10.10.10.51"
    }
  ]
}

PLAY RECAP ********************************************
localhost  : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

We start with the same simple list of IP addresses, and end up with exactly the format each vendor's module expects, host for Cisco and name for Arista.