Python Working with Structured Data Files

In the previous sections, we looked at reading and writing plain text files. That was a good starting point, and plain text is still the right choice for things like config backups or log files. But most of the data you deal with in network automation is not plain text, it has structure. A list of devices with their IPs, usernames, and roles. A dictionary of device facts. A nested set of configuration values with groups and per-device overrides.

When your data has structure, you need a format that can represent that structure in a way both humans and computers can understand. This is where structured data files can help. The three main formats you will run into as a network engineer are CSV, JSON, and YAML.

CSV Files

CSV stands for Comma-Separated Values. A CSV file is just a plain text file where each line is a row, and each value in the row is separated by a comma. The first row usually contains column headers.

Here is what a CSV file looks like, saved as devices.csv.

hostname,ip,vendor,role
r1,192.168.202.101,cisco,edge
r2,192.168.202.102,juniper,core
r3,192.168.202.103,arista,edge

If you have ever exported data from a spreadsheet like Excel or Google Sheets, you have probably seen CSV before. It is easy to read, easy to edit, and works well when your data is essentially a table.

Reading a CSV File

Python has a built-in module called csv that handles all the parsing for you. You do not have to write custom code to split lines on commas, csv does it properly and handles edge cases like commas inside quoted values.

The simplest way to read a CSV file is with csv.reader, which gives you each row as a list.

import csv

with open("devices.csv") as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

Output

['hostname', 'ip', 'vendor', 'role']
['r1', '192.168.202.101', 'cisco', 'edge']
['r2', '192.168.202.102', 'juniper', 'core']
['r3', '192.168.202.103', 'arista', 'edge']

Each row is a list, with one item per column. Notice that the first row is the header row (hostname, ip, vendor, role), and we get it back the same way as the data rows. If we want to work with just the data, we would either need to skip the first row manually or, better, use a different approach.

Reading a CSV as Dictionaries

csv.DictReader is a much nicer way to read CSVs. Instead of giving you each row as a list, it gives you each row as a dictionary, using the header row as keys.

import csv

with open("devices.csv") as file:
    reader = csv.DictReader(file)
    for row in reader:
        print(row)

Output

{'hostname': 'r1', 'ip': '192.168.202.101', 'vendor': 'cisco', 'role': 'edge'}
{'hostname': 'r2', 'ip': '192.168.202.102', 'vendor': 'juniper', 'role': 'core'}
{'hostname': 'r3', 'ip': '192.168.202.103', 'vendor': 'arista', 'role': 'edge'}

Now each row is a dictionary with meaningful keys, and you can access the values by name instead of by position. This is much easier to work with, and if the column order in the CSV ever changes, your code does not break.

import csv

with open("devices.csv") as file:
    reader = csv.DictReader(file)
    for row in reader:
        print(f"{row['hostname']} ({row['vendor']}) at {row['ip']}")

Output

r1 (cisco) at 192.168.202.101
r2 (juniper) at 192.168.202.102
r3 (arista) at 192.168.202.103

Writing a CSV File

Writing works in a similar way. There is csv.writer for writing lists, and csv.DictWriter for writing dictionaries. Just like with reading, DictWriter is usually the better choice.

import csv

devices = [
    {"hostname": "r1", "ip": "192.168.202.101", "vendor": "cisco", "role": "edge"},
    {"hostname": "r2", "ip": "192.168.202.102", "vendor": "juniper", "role": "core"},
    {"hostname": "r3", "ip": "192.168.202.103", "vendor": "arista", "role": "edge"}
]

with open("output.csv", "w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["hostname", "ip", "vendor", "role"])
    writer.writeheader()
    writer.writerows(devices)

After running this, you will have a file called output.csv with the same structure as devices.csv. A few things to note here.

• newline="" is passed to open(). This prevents extra blank lines from appearing between rows when writing CSVs on some operating systems.
• fieldnames is the list of column names, in the order you want them written.
• .writeheader() writes the header row.
• .writerows(devices) writes all the data rows in one go. If you only had one row, you could use .writerow() (singular) instead.

import csv

device = {"hostname": "r1", "ip": "192.168.202.101", "vendor": "cisco", "role": "edge"}

with open("output_2.csv", "w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["hostname", "ip", "vendor", "role"])
    writer.writeheader()
    writer.writerow(device)

.writerow() takes a single dictionary and writes it as one row. .writerows() takes a list of dictionaries and writes them all at once.

Here you go, a shortened version. Only the prose is rewritten, the code stays as you had it.

CSV Example

In our previous examples, the inventory has been hardcoded inside the script. Let's move it out into a CSV file instead, so we can add or remove devices without touching the code.

Start with a file called inventory.csv that has your devices.

hostname,ip,driver,username,password
r1,192.168.202.101,ios,admin,admin
r2,192.168.202.102,junos,admin,admin@123
r3,192.168.202.103,eos,admin,admin

Now write a script that reads this file and connects to each device with NAPALM.

import csv
from napalm import get_network_driver

with open("inventory.csv") as file:
    reader = csv.DictReader(file)
    devices = list(reader)

for device in devices:
    print(f"=== {device['hostname']} ({device['ip']}) ===")

    driver = get_network_driver(device["driver"])

    optional_args = {"transport": "ssh"} if device["driver"] == "eos" else {}

    with driver(
        hostname=device["ip"],
        username=device["username"],
        password=device["password"],
        optional_args=optional_args
    ) as connection:
        facts = connection.get_facts()
        print(f"Vendor: {facts['vendor']}")
        print(f"OS Version: {facts['os_version']}")
        print(f"Uptime: {facts['uptime']} seconds")
        print()

Output

=== r1 (192.168.202.101) ===
Vendor: Cisco
OS Version: Linux Software (X86_64BI_LINUX-ADVENTERPRISEK9-M), Version 17.12.1, RELEASE SOFTWARE (fc5)
Uptime: 420.0 seconds

=== r2 (192.168.202.102) ===
Vendor: Juniper
OS Version: 24.2R1-S2.5
Uptime: 397.0 seconds

=== r3 (192.168.202.103) ===
Vendor: Arista
OS Version: 4.34.2F-43232954.4342F.1
Uptime: 487.92069125175476 seconds

csv.DictReader(file) gives us a reader object that goes through the file one row at a time and returns each row as a dictionary. Wrapping it in list() reads all the rows into a proper list, so devices ends up as a list of dictionaries you can loop through and reuse. From there, the rest of the script is exactly what we did in the NAPALM chapter, so I will not go over the optional_args and connection details again.

JSON Files

JSON stands for JavaScript Object Notation. It is a text-based format for representing structured data, and it is used everywhere in modern automation. When you talk to a REST API, the response is almost always JSON. When you save NAPALM output to disk, JSON is a good choice. When you look at configuration data pulled from a system like NetBox, it comes back as JSON.

The nice thing about JSON is that it maps almost perfectly to Python's basic data types. A JSON object looks like a Python dictionary, and a JSON array looks like a Python list. Once you have JSON data loaded into your script, you work with it exactly like any other Python dictionary or list.

Here is what a JSON file might look like.

{
    "hostname": "r1",
    "vendor": "cisco",
    "interfaces": ["Ethernet0/0", "Ethernet0/1", "Loopback0"],
    "is_active": true
}

Notice how similar it looks to a Python dictionary. The main differences are that JSON uses lowercase true and false (Python uses True and False), and JSON keys must always be in double quotes. Python's json module manages all these conversions for you, so you rarely have to think about it.

Reading a JSON File

Python has a built-in json module that manages reading and writing JSON. To read a JSON file, you use json.load(), passing in the file object.

Say you have a file called device.json with the content above.

import json

with open("device.json") as file:
    data = json.load(file)

print(data)
print(data["hostname"])
print(data["interfaces"])

Output

{'hostname': 'r1', 'vendor': 'cisco', 'interfaces': ['Ethernet0/0', 'Ethernet0/1', 'Loopback0'], 'is_active': True}
r1
['Ethernet0/0', 'Ethernet0/1', 'Loopback0']

json.load() reads the file, parses the JSON, and gives you back a Python dictionary. From there, it is just a regular dictionary, so you can access keys, loop through it, or pass it around like any other Python data structure.

If you already have JSON as a string (for example, from an API response), you use json.loads() instead (note the extra s)

import json

json_string = '{"hostname": "r1", "vendor": "cisco"}'
data = json.loads(json_string)

print(data["hostname"])

Output

r1

Writing a JSON File

Writing works the same way, in reverse. Use json.dump() to write a Python object to a file, or json.dumps() to convert it to a string.

import json

data = {
    "hostname": "r1",
    "vendor": "cisco",
    "interfaces": ["Ethernet0/0", "Ethernet0/1", "Loopback0"],
    "is_active": True
}

with open("output.json", "w") as file:
    json.dump(data, file, indent=2)

After running this, output.json will contain:

{
  "hostname": "r1",
  "vendor": "cisco",
  "interfaces": [
    "Ethernet0/0",
    "Ethernet0/1",
    "Loopback0"
  ],
  "is_active": true
}

The indent=2 argument tells json.dump() to format the output with 2 spaces of indentation, which makes the file readable by humans. Without indent, everything would be written on a single line, which is fine for machines but painful to read. You saw this same trick back in Chapter 14 when we used json.dumps() to pretty-print TextFSM output.

json.dumps() takes a Python dictionary (or list) and returns a JSON-formatted string. Notice the type() at the end confirms it is a string, not a dictionary. This is useful when you want to send JSON to something that expects a string, like an API request body, or when you just want to print it for inspection without writing it to a file.

import json

data = {
    "hostname": "r1",
    "vendor": "cisco",
    "interfaces": ["Ethernet0/0", "Ethernet0/1", "Loopback0"]
}

json_string = json.dumps(data, indent=2)

print(json_string)
print(type(json_string))

Output

{
  "hostname": "r1",
  "vendor": "cisco",
  "interfaces": [
    "Ethernet0/0",
    "Ethernet0/1",
    "Loopback0"
  ]
}
<class 'str'>

Working with JSON in a Script

For this example, we will pull device facts using NAPALM, save each device's facts to its own JSON file, then read one of them back later.

We already have the inventory in inventory.csv from the CSV section, so we will keep using that.

import csv
import json
from napalm import get_network_driver

with open("inventory.csv") as file:
    reader = csv.DictReader(file)
    devices = list(reader)

for device in devices:
    print(f"Connecting to {device['hostname']}...")

    driver = get_network_driver(device["driver"])
    optional_args = {"transport": "ssh"} if device["driver"] == "eos" else {}

    with driver(
        hostname=device["ip"],
        username=device["username"],
        password=device["password"],
        optional_args=optional_args
    ) as connection:
        facts = connection.get_facts()

    filename = f"{device['hostname']}_facts.json"
    with open(filename, "w") as file:
        json.dump(facts, file, indent=2)

    print(f"Saved facts to {filename}")

After running this, you will have three JSON files in your folder, r1_facts.json, r2_facts.json, and r3_facts.json. Each one contains the structured facts for that device.

Here is what r1_facts.json looks like.

{
  "uptime": 1920.0,
  "vendor": "Cisco",
  "os_version": "Linux Software (X86_64BI_LINUX-ADVENTERPRISEK9-M), Version 17.12.1, RELEASE SOFTWARE (fc5)",
  "serial_number": "1",
  "model": "Unknown",
  "hostname": "r1",
  "fqdn": "r1.lab",
  "interface_list": [
    "Ethernet0/0",
    "Ethernet0/1",
    "Ethernet0/2",
    "Ethernet0/3",
    "Loopback0"
  ]
}