Pre and Post Checks with Ansible

In the previous part, we wrote two checks for our OSPF network. One makes sure every router has all of its OSPF neighbours, and the other makes sure every router can reach every other router's loopback. They're useful on their own, but they're even more useful when you run them around a change.

The idea is that before making a change, you check that the network is healthy. If it isn't, you stop, since there's no point making a change on a network that's already broken. It also means that if something goes wrong later, you know it wasn't already broken before you started. Then, after the change, you run the same checks again to make sure everything is still healthy.

Most of us already do this manually. Before a change, we log in and save the output of a few show commands, and afterwards, we run them again and compare. In this part, we'll let Ansible do it for us, by reusing the playbooks we've already written, rather than copying them into every change playbook.

We're continuing with the same lab and project from the previous two parts, so if you haven't read them yet, I recommend starting there.

Joining Playbooks with import_playbook

We already have three playbooks that do exactly what we need.

• validate_ospf.yml - Checks the OSPF neighbors
• validate_reachability.yml - Checks reachability between the loopbacks
• ospf.yml - Configures the network, based on host_vars

All we need to do is run them in the right order: the two checks, then the change, then the two checks again. Ansible has a feature called import_playbook, which lets you run one playbook from inside another. Create a new playbook called change.yml.

---
- name: Pre-checks - OSPF neighbors
  ansible.builtin.import_playbook: validate_ospf.yml

- name: Pre-checks - Reachability
  ansible.builtin.import_playbook: validate_reachability.yml

- name: Apply the change
  ansible.builtin.import_playbook: ospf.yml

- name: Post-checks - OSPF neighbors
  ansible.builtin.import_playbook: validate_ospf.yml

- name: Post-checks - Reachability
  ansible.builtin.import_playbook: validate_reachability.yml

change.yml doesn't have any tasks of its own. Each line simply imports another playbook, and Ansible runs them one after the other, from top to bottom. Behind the scenes, Ansible combines all the plays from the imported playbooks into a single playbook before it starts.

The nice thing about this approach is that nothing is copied. The checks only exist in one place, so if you ever improve them, change.yml automatically uses the new version. And each playbook still works on its own, so you can still run validate_ospf.yml by itself whenever you like.

Let's run it with the network healthy and no changes in host_vars, just to see how it works.

ansible-playbook change.yml

PLAY [Validate OSPF] *****************************

TASK [Get OSPF neighbors] ************************
ok: [r5]
...

TASK [Check the number of OSPF neighbors] ********
ok: [r2] => {
  "changed": false,
  "msg": "r2 has 4 of 4 OSPF neighbors"
}
...

PLAY [Validate reachability] *********************

TASK [Ping every other loopback] *****************
ok: [r2] => (item=6.6.6.6)
...

PLAY [Configure OSPF] ****************************

TASK [Configure interface status and descriptions]
ok: [r5] => (item=Ethernet1/0)
...

TASK [Configure OSPF process] ********************
ok: [r5]
...

PLAY [Validate OSPF] *****************************
...

PLAY [Validate reachability] *********************
...

PLAY RECAP ***************************************
r1 : ok=16 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r2 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r3 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r4 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r5 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r6 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

I've trimmed the output quite a bit, since it now runs five plays. You can see each play running in order, and since nothing changed, every task shows ok. There are a few things worth pointing out.

• The play names - The output shows the names of the plays inside each playbook, such as PLAY [Validate OSPF], not the names we gave each line in change.yml. So the pre-checks and post-checks look the same in the output, and you can tell them apart by their position.
• A single PLAY RECAP - Even though five playbooks ran, we get one combined PLAY RECAP at the end.
• ok=16 on r1 - r1 has two more ok than the other routers. That's the Display the list task with run_once, which ran once in the pre-checks and once in the post-checks, both times on r1.

What Happens When a Pre-Check Fails?

So far, so good. But the whole point of the pre-checks is to stop the change if the network isn't healthy. Let's see whether that actually happens.

To break the network, I shut down Ethernet0/1 on r1, which is the link to r2

r1(config)#interface Ethernet0/1
r1(config-if)#shutdown

Then, I added a new loopback to host_vars/r3.yml, so we have a change to make, on a router that isn't affected by the broken link.

  - name: Loopback1
    description: Office LAN
    ip: 192.168.30.1/24
    area: 0
    passive: true

Let's run change.yml again.

ansible-playbook change.yml

PLAY [Validate OSPF] *****************************

TASK [Check the number of OSPF neighbors] ********
fatal: [r2]: FAILED! => {
  "msg": "r2 has 3 OSPF neighbors, expected 4"
}
fatal: [r1]: FAILED! => {
  "msg": "r1 has 3 OSPF neighbors, expected 4"
}
ok: [r3] => {
  "changed": false,
  "msg": "r3 has 4 of 4 OSPF neighbors"
}
...

PLAY [Validate reachability] *********************
...

PLAY [Configure OSPF] ****************************

TASK [Configure interface status and descriptions]
ok: [r6] => (item=Ethernet1/0)
...
changed: [r3] => (item=Loopback1)
...

RUNNING HANDLER [Save config] ********************
changed: [r3]

PLAY [Validate OSPF] *****************************
...

PLAY [Validate reachability] *********************
...

PLAY RECAP ***************************************
r1 : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
r2 : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
r3 : ok=17 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r4 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r5 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r6 : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

As expected, r1 and r2 failed the neighbor pre-check. But look at what happened next. Ansible removed r1 and r2 from the rest of the run, and carried on with the other four routers as if nothing had happened. r3 still got its new Loopback1, and the config was saved, even though the network wasn't healthy before the change.

r3#show run | sec Loopback1
interface Loopback1
 description Office LAN
 ip address 192.168.30.1 255.255.255.0
 ip ospf 1 area 0
 passive-interface Loopback1

This is Ansible's default behaviour. When a task fails on a device, Ansible stops running tasks on that device, including in any plays that come after it, but it carries on with every other device. That's often what you want, for example, when you're collecting information from a hundred devices and one of them is unreachable. But for pre-checks, it's exactly what we don't want. If any router fails a pre-check, the change shouldn't go ahead anywhere.

There are also two other details in this run worth pointing out. First, the reachability pre-check still passed on the remaining routers, even with the link down. That's because OSPF simply routed around the broken link through the other paths, which is a good example of why we need both checks. Second, the Display the list task now ran on r3 instead of r1. Since r1 had been removed from the run, r3 was the first router left.

Stopping Everything with any_errors_fatal

To change this behaviour, we can use any_errors_fatal. When it's set to true on a play, a failure on any device stops the entire run, for every device, not just the one that failed.

Let's add it to the play in both validation playbooks, just under gather_facts.

validate_ospf.yml

- name: Validate OSPF
  hosts: routers
  gather_facts: false
  any_errors_fatal: true

validate_reachability.yml

- name: Validate reachability
  hosts: routers
  gather_facts: false
  any_errors_fatal: true

We only add it to the validation playbooks, not to ospf.yml. The checks are where a failure should stop everything.

To test it, I left Ethernet0/1 on r1 shut, and added another change, this time a new loopback in host_vars/r4.yml.

  - name: Loopback1
    description: Office LAN
    ip: 192.168.40.1/24
    area: 0
    passive: true

Let's run change.yml again.

ansible-playbook change.yml

PLAY [Validate OSPF] *****************************

TASK [Get OSPF neighbors] ************************
ok: [r2]
ok: [r4]
ok: [r5]
ok: [r3]
ok: [r1]
ok: [r6]

TASK [Count the expected and actual neighbors] ***
ok: [r2]
ok: [r1]
ok: [r3]
ok: [r4]
ok: [r5]
ok: [r6]

TASK [Check the number of OSPF neighbors] ********
[ERROR]: Task failed: Action failed: r1 has 3 OSPF
neighbors, expected 4
Origin: .../validate_ospf.yml:19:7

fatal: [r1]: FAILED! => {
  "assertion": "actual_neighbors | int == expected_neighbors | int",
  "changed": false,
  "evaluated_to": false,
  "msg": "r1 has 3 OSPF neighbors, expected 4"
}
[ERROR]: Task failed: Action failed: r2 has 3 OSPF
neighbors, expected 4
Origin: .../validate_ospf.yml:19:7

fatal: [r2]: FAILED! => {
  "assertion": "actual_neighbors | int == expected_neighbors | int",
  "changed": false,
  "evaluated_to": false,
  "msg": "r2 has 3 OSPF neighbors, expected 4"
}
ok: [r4] => {
  "changed": false,
  "msg": "r4 has 4 of 4 OSPF neighbors"
}
ok: [r3] => {
  "changed": false,
  "msg": "r3 has 4 of 4 OSPF neighbors"
}
ok: [r5] => {
  "changed": false,
  "msg": "r5 has 2 of 2 OSPF neighbors"
}
ok: [r6] => {
  "changed": false,
  "msg": "r6 has 2 of 2 OSPF neighbors"
}

PLAY RECAP ***************************************
r1 : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
r2 : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
r3 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r4 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r5 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r6 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

This time, the run stopped as soon as the neighbor pre-check failed. There's no Validate reachability play and no Configure OSPF play, and the PLAY RECAP shows only three tasks for each router, which are the three tasks from the neighbor check. r4 didn't get its new loopback, so the change was blocked, exactly as we wanted.

Notice that the other four routers still finished the Check the number of OSPF neighbors task, and show ok. any_errors_fatal lets the current task finish on every device first, and then stops the run before the next task. So you still get the full picture of which routers passed and which ones failed, which makes it easier to work out what's wrong.