Introduction
If you’ve been following along in this series, you’ve seen how we can programmatically interact and gather information about our network using pyATS and Genie. That’s great and all, but wouldn’t it be better if we could begin providing true value to ourselves and our team?
As mentioned in Part 2, pyATS is essentially the foundation to our testing environment. It provides a structure to how our devices and topology are defined. In this part, we are going to dive deeper into pyATS and how we can begin writing testcases and testscripts. pyATS provides a standard automation testing framework (or harness, as they refer to it) called AEtest. AEtest, or Automation Easy Testing, comes with defined building blocks on how to structure your testscripts. Now before we get deeper, you may be asking, what is a testcase or testscript? We create “testscripts” now for running checks during a change window, such as pinging a device before and after a change to ensure connectivity. No, no, no… a testscript can be much more than that. What if we could check for a specific number of ‘established’ BGP peers AND check the routing table for a specific route before and after a change? If the results don’t meet our expectation, we can roll the changes back. Oh yeah, we can also throw in your ping test at the end too.
On top of it all, AEtest is pythonic in nature with an object-oriented design, which allows you to create relationships amongst objects, add more logic to your tests, and even alter whether a test is ran depending on another test’s results. That’s pretty insane! Enough with the intro, let’s dive into building testscripts with AEtest in pyATS.
So what is a Testscript?
AEtest provides a defined structure on how to build your testscripts. This confused me at first, as I thought this structure was a mere suggestion. However, after playing around with it, I quickly realized how important each section is to the overall testscript. Let’s take a look at the different sections of a testscript.
As I’ve said in previous posts, I’m a visual person. I like creating a picture or diagram to understand how something works. The above picture displays each major section of a testscript and why each one is important. Let’s quickly run through each one:
Common Setup
The Common Setup section of an AEtest testscript provides the basic configuration and connectivity to all test devices. For example, you would use this section to connect to each device in your testbed. If you’ve ever worked with a virtual network topology, you’ll know that you have to enable the interfaces when you bring up a virtual device (CSR1000v, Nexus 9Kv, etc.) by performing a ‘no shut’ on each interface. This would be the appropriate section to do that. Just think of this section as the preparation section.
Testcases
The Testcases section is where you write out your individual tests. Tests can include making a configuration change on a device and parsing out a ‘show’ command to confirm the result meets your expectation, or simply running a ping test. The best part about testcases is that you’re in control and can define the specific tests to meet your requirements. pyATS provides the framework, and Genie provides the set of tools to gather and parse the information that’s important to you.
Common Cleanup
The Common Cleanup section is the last section ran in a testscript. This section is focused on cleaning up any configuration changes or other changes you made during testing. The goal is to reset the environment to how it was before testing. You can think of this section as the custodian of each testscript… CLEAN UP ON TESTSCRIPT ONE!
There are many more features and levels to each section, but I’m trying to keep it as simple as possible to avoid overwhelming you. However, I’ll include links to the documentation at the end of this post so that you can read more about each section.
Writing our own Testscript
So now that we’ve established testscripts are a little more than running a few ping tests and manually recording the results in Notepad (I know we all run more than a few ping tests for verification, but I’m trying to keep it lighthearted!), let’s jump into writing our first testscript using AEtest.
In our first testscript, we are going to write a testcase to check the software version of an IOS-XE device and confirm whether it meets our defined standard.
from pyats import aetest
from pprint import pprint
from genie.utils import Dq
import logging
import sys
logger = logging.getLogger(__name__)
class CommonSetup(aetest.CommonSetup):
@aetest.subsection
def connect_to_devices(self, steps, testbed, host):
device = testbed.devices[str(host)]
# Add device as testscript param
self.parent.parameters.update(device = device)
with steps.start(f'Connect to {device.name}'):
# Connect to the device
device.connect()
### TESTCASE SECTION ###
class FirstTestcase(aetest.Testcase):
@aetest.setup
def setup(self, device, testbed):
# Confirm we are connected to the device before running commands
if device.connected:
self.passed('Successfully connected to the device')
else:
self.failed('Could not connect to the device')
@aetest.test
def verify_ios_version(self, steps, device):
with steps.start('Checking the IOS version') as step:
try:
self.version = device.parse('show version')
current_dev_ios = Dq(self.version).get_values('xe_version')
# Checks to see whether the IOS-XE version meets the set standard version of 17.3.3
ios_check = Dq(self.version).contains('17.03.03').get_values('xe_version')
# A populated list is returned if the version matched
if ios_check:
step.passed('Device is running the proper IOS version.')
else:
# An empty list is returned if the version didn't match
step.failed('Device is running a different IOS version than the defined standard.')
except Exception as e:
# If an exception is caught, the testcase fails and prints the error
self.failed(f'Could not parse data due to the following error: {str(e)}')
class CommonCleanup(aetest.CommonCleanup):
@aetest.subsection
def disconnect_from_devices(self, steps, device):
with steps.start(f'Disconnecting from {device.name}'):
# Disconnect from the device
device.disconnect()
There’s a lot to dissect in this testscript. First, I want to point out that I highlighted the major “container classes” that I summarized earlier. These classes are referred to as “containers” because they contain other test sections. This concept took me awhile to understand, but let’s take a look at this diagram found in the pyATS docs:
+--------------+
| TestScript |
+-------+------+
|
+----------------------------+---------------------------+
| | |
+------+------+ +--------+-------+ +-------+-------+
| CommonSetup | | Testcases | | CommonCleanup |
+------+------+ +--------+-------+ +-------+-------+
| | |
+------+------+ | +------+------+
| subsections | +----------+-----------+ | subsections |
+-------------+ | | | +-------------+
+---+---+ +---+---+ +----+----+
| setup | | tests | | cleanup |
+-------+ +-------+ +---------+
You can see that each testscript has a hierarchy to it. The Common Setup and Common Cleanup sections can be broken down into subsections, and Testcases can be broken down even further into setup, test, and cleanup sections. There is even one more level that’s not depicted here called steps. As mentioned previously, AEtest testscripts are pythonic in nature with an object-oriented design, so each of these sections can inherit from one another. For example, the TestScript object is the parent to each of the Common Setup, Testcases, and Common Cleanup sections. Any parameters defined at the TestScript level can be accessed by each of the child sections. There’s ALOT more to how these objects can relate to one another, but I’ll save that for another day. My goal is to provide just enough of an overview of the relationships found in testscripts for you to understand further explanations of our first testscript.
Now that we have a basic understanding of how a testscript is structured and its relationships, let’s start reviewing our first testscript. In our Common Setup, we connect to a specific host that is defined in our testbed. For reference, this testscript is being used in a web app that allows a user to input the hostname and IP address into a form – more to come on that at a later time :). So we have a host, testbed, and steps that are passed in as parameters to this specific section. The device parameter is added to the testscript parameters, so it can be used anywhere in this testscript. After we define the device, I have a subsection for connecting to the device using the device.connect() function – pretty straightforward. Remember, if you are testing with multiple hosts and need to connect to all of them, this would be the section to do that. This section would also be used to apply general configuration across all of testing devices.
After the Common Setup section, we create our first Testcase section. In the Testcase section, we define smaller sections for setup and the actual test section. I decided not to include a cleanup section in the Testcase, as the Common Cleanup section takes care of everything we need for this testscript. In the setup section, we ensure that we’re connected to the device (device.connected() -> returns True/False). Once we confirm we are connected to the device, we issue the show version command to the device and parse the output using Genie. After parsing the output, we check to see whether the device contains a specific IOS-XE version using the Dq library, which we reviewed in Part 2 of this series. If you haven’t already, please check out Part 2 or read the Genie docs to understand the fascinating functionalities of the Dq library – it will save you time and many lines of code! If there’s a match to the specific IOS-XE version we are expecting, a Python list will be returned with the matching string being the first item in the list. For example, if there was a match in our test, the returned value would look something like this: ['17.03.03']. Given that understanding, we check to see whether a populated list is returned. If so, the step passes and the results are rolled up to the Testcase.
The last section is the Common Cleanup section. In this section, we simply disconnect from the device. If any configuration or environmental changes were applied during testing, this would be the section to revert or reset those settings. The goal of this section is to reset the environment to how it looked before testing.
How do I run my Testscript?
So we have a testscript and (for the most part) understand each section of it, but how do we run the actual tests? In the pyATS docs, you’ll find there are two official ways to run an AEtest testscript: Standalone or using Easypy execution. I know, you may be saying, “Greattttttt… now I have to pick and choose a method to run my tests”. Well, pyATS provides some good descriptions and weighs the pros and cons to each method.
Standalone
The Standalone method is the most flexible and easier one to comprehend when first starting out. The docs even suggest this method for “rapid, lightweight script development…”. However, there are some tradeoffs with the ease of use. You are limited to executing a single script, all logging/results are printed to the standard output (stdout), and there is no archiving or official report generation. This truly is your grab n’ go testing method. If you’d like to execute your testscript with this method, you simply need to run your testscript by invoking aetest.main() at the end of your testscript file. Here’s a quick example from the pyATS docs:
# Example
# -------
#
# enabling standalone execution
import logging
from pyats import aetest
# your testscript sections, testscases & etc
# ...
#
# add the following as the absolute last block in your testscript
if __name__ == '__main__':
# control the environment
# eg, change some log levels for debugging
logging.getLogger(__name__).setLevel(logging.DEBUG)
logging.getLogger('pyats.aetest').setLevel(logging.DEBUG)
# aetest.main() api starts the testscript execution.
# defaults to aetest.main(testable = '__main__')
aetest.main()
You notice on the last line that all you need to do is invoke aetest.main(), since your testscript and all the testcases are defined in the same file. You can also call on a testscript from another Python file using aetest.main(). Here’s a quick example of how to do that:
aetest.main(testable='./testscripts/first_testscript.py',
testbed=my_testbed, host=device_hostname)
You can see that you must provide some additional arguments. However, the only necessary argument is the testable argument. The testable argument points to the location of the testscript file. By default, it points to ‘__main__’. All other arguments are keyword arguments that can be later used in the testscript – they are loaded in as testscript-level parameters. The Standalone execution method is great for testing and developing scripts, it may even work for certain use cases, but what if we wanted a something a little more? Easypy execution might be what you’re looking for…
Easypy Execution
Easypy execution is the more “official” way to run your testscripts. You would use this method to perform production-grade testing, such as sanity/regression testing. There are many advantages to using Easypy over Standalone execution. With Easypy, you can aggregate multiple testscripts into a single job file. Easypy will take care of logging configuration, with the option for user customization in each job file. A TaskLog object is used to generate results, reports, and archives. A Reporter object is used for reporting results by generating a YAML results file, along with XML summary files.
There are multiple ways to run Easypy jobs. One of the most common ways is to use the pyats run job CLI commands. This eliminates the need to code anything additional to run your testscript. If you have pyATS installed, you will have access to the pyATS command-line. The other option is to use the run() API. Just like with the Standalone execution, you pass in the location of the testscript file as the first argument and the execution environment takes care of the rest. Here’s another example from the pyATS docs:
# Example
# -------
#
# pyats job file example, with script arguments
from pyats.easypy import run
def main():
# providing a couple custom script arguments as **kwargs
run(testscript='/path/to/your/script.py',
pyats_is_awesome = True,
aetest_is_legendary = True)
# if this job file was run with the following command:
# pyats run job example_job.py --testbed-file /path/to/my/testbed.yaml
#
# and the script had one testcase that prints out the script's parameters,
# the output of the script ought to be:
#
# starting test execution for testscript 'a.py'
# +------------------------------------------------------------------------------+
# | Starting testcase Testcase |
# +------------------------------------------------------------------------------+
# +------------------------------------------------------------------------------+
# | Starting section test |
# +------------------------------------------------------------------------------+
# Parameters = {'testbed': <Testbed object at 0xf742f74c>,
# 'pyats_is_awesome': True,
# 'aetest_is_legendary': True}
# The result of section test is => PASSED
# The result of testcase Testcase is => PASSED
As you can see, the run() API is called and the first argument, testscript, is the file path to the testscript we want to execute. After Easypy execution, the runtime logs and archives will be stored in a local directory. You can review each of these file manually or use the pyats logs view CLI command to view them in a nice HTML format on a local web server.
Now that I’ve reviewed each method to run a testscript, it’s up to you decide, based on requirements, which method is best for you.
Running the Testscript
To run my example testscript, I chose to use the Standalone execution and capture the results in the standard output (stdout). I used the ‘IOS XE on CSR Recommended Code Always On‘ sandbox hosted by Cisco DevNet as my test device. Here’s what the results look like for Standalone execution of the testscript:
2021-06-22T11:04:06: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:06: %AETEST-INFO: | Starting common setup |
2021-06-22T11:04:06: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:06: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:06: %AETEST-INFO: | Starting subsection connect_to_devices |
2021-06-22T11:04:06: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:07: %AETEST-INFO: +..............................................................................+
2021-06-22T11:04:07: %AETEST-INFO: : Starting STEP 1: Connect to csr1000v-1 :
2021-06-22T11:04:07: %AETEST-INFO: +..............................................................................+
2021-06-22 11:04:07,061: %UNICON-INFO: +++ csr1000v-1 logfile /tmp/csr1000v-1-cli-20210622T110407061.log +++
2021-06-22 11:04:07,061: %UNICON-INFO: +++ Unicon plugin iosxe +++
2021-06-22 11:04:10,775: %UNICON-INFO: +++ connection to spawn: ssh -l developer 64.103.37.51 -p 8181, id: 140725016193680 +++
2021-06-22 11:04:10,775: %UNICON-INFO: connection to csr1000v-1
Password:
Welcome to the DevNet Sandbox for CSR1000v and IOS XE
The following programmability features are already enabled:
- NETCONF
- RESTCONF
Thanks for stopping by.
csr1000v-1#
2021-06-22 11:04:11,234: %UNICON-INFO: +++ initializing handle +++
2021-06-22 11:04:11,297: %UNICON-INFO: +++ csr1000v-1 with alias 'cli': executing command 'term length 0' +++
term length 0
csr1000v-1#
2021-06-22 11:04:11,659: %UNICON-INFO: +++ csr1000v-1 with alias 'cli': executing command 'term width 0' +++
term width 0
csr1000v-1#
2021-06-22 11:04:12,059: %UNICON-INFO: +++ csr1000v-1 with alias 'cli': executing command 'show version' +++
show version
Cisco IOS XE Software, Version 16.09.03
Cisco IOS Software [Fuji], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.9.3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Wed 20-Mar-19 07:56 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: IOS-XE ROMMON
csr1000v-1 uptime is 2 minutes
Uptime for this control processor is 3 minutes
System returned to ROM by reload
System image file is "bootflash:packages.conf"
Last reload reason: reload
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
License Level: ax
License Type: Default. No valid license found.
Next reload license Level: ax
Smart Licensing Status: Smart Licensing is DISABLED
cisco CSR1000V (VXE) processor (revision VXE) with 2392579K/3075K bytes of memory.
Processor board ID 9UB7M1TZDUS
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8113280K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.
Configuration register is 0x2102
csr1000v-1#
2021-06-22 11:04:13,457: %UNICON-INFO: +++ csr1000v-1 with alias 'cli': configure +++
config term
Enter configuration commands, one per line. End with CNTL/Z.
csr1000v-1(config)#no logging console
csr1000v-1(config)#line console 0
csr1000v-1(config-line)#exec-timeout 0
csr1000v-1(config-line)#end
csr1000v-1#
2021-06-22T11:04:15: %AETEST-INFO: The result of STEP 1: Connect to csr1000v-1 is => PASSED
2021-06-22T11:04:15: %AETEST-INFO: +----------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: | STEPS Report |
2021-06-22T11:04:15: %AETEST-INFO: +----------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: STEP 1 - Connect to csr1000v-1 Passed
2021-06-22T11:04:15: %AETEST-INFO: ------------------------------------------------------------
2021-06-22T11:04:15: %AETEST-INFO: The result of subsection connect_to_devices is => PASSED
2021-06-22T11:04:15: %AETEST-INFO: The result of common setup is => PASSED
2021-06-22T11:04:15: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: | Starting testcase FirstTestcase |
2021-06-22T11:04:15: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: | Starting section setup |
2021-06-22T11:04:15: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: Passed reason: Successfully connected to the device
2021-06-22T11:04:15: %AETEST-INFO: The result of section setup is => PASSED
2021-06-22T11:04:15: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: | Starting section verify_ios_version |
2021-06-22T11:04:15: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:15: %AETEST-INFO: +..............................................................................+
2021-06-22T11:04:15: %AETEST-INFO: : Starting STEP 1: Checking the IOS version :
2021-06-22T11:04:15: %AETEST-INFO: +..............................................................................+
2021-06-22 11:04:16,465: %UNICON-INFO: +++ csr1000v-1 with alias 'cli': executing command 'show version' +++
show version
Cisco IOS XE Software, Version 16.09.03
Cisco IOS Software [Fuji], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.9.3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Wed 20-Mar-19 07:56 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: IOS-XE ROMMON
csr1000v-1 uptime is 2 minutes
Uptime for this control processor is 3 minutes
System returned to ROM by reload
System image file is "bootflash:packages.conf"
Last reload reason: reload
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
License Level: ax
License Type: Default. No valid license found.
Next reload license Level: ax
Smart Licensing Status: Smart Licensing is DISABLED
cisco CSR1000V (VXE) processor (revision VXE) with 2392579K/3075K bytes of memory.
Processor board ID 9UB7M1TZDUS
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8113280K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.
Configuration register is 0x2102
csr1000v-1#
2021-06-22T11:04:19: %AETEST-ERROR: [31mFailed reason: Device is running a different IOS version than the defined standard.[0m[39m
2021-06-22T11:04:19: %AETEST-INFO: The result of STEP 1: Checking the IOS version is => FAILED
2021-06-22T11:04:19: %AETEST-INFO: +----------------------------------------------------------+
2021-06-22T11:04:19: %AETEST-INFO: | STEPS Report |
2021-06-22T11:04:19: %AETEST-INFO: +----------------------------------------------------------+
2021-06-22T11:04:19: %AETEST-INFO: STEP 1 - Checking the IOS version Failed
2021-06-22T11:04:19: %AETEST-INFO: ------------------------------------------------------------
2021-06-22T11:04:19: %AETEST-INFO: The result of section verify_ios_version is => FAILED
2021-06-22T11:04:19: %AETEST-INFO: The result of testcase FirstTestcase is => FAILED
2021-06-22T11:04:19: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:19: %AETEST-INFO: | Starting common cleanup |
2021-06-22T11:04:19: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:19: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:19: %AETEST-INFO: | Starting subsection disconnect_from_devices |
2021-06-22T11:04:19: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:19: %AETEST-INFO: +..............................................................................+
2021-06-22T11:04:19: %AETEST-INFO: : Starting STEP 1: Disconnecting from csr1000v-1 :
2021-06-22T11:04:19: %AETEST-INFO: +..............................................................................+
2021-06-22T11:04:30: %AETEST-INFO: The result of STEP 1: Disconnecting from csr1000v-1 is => PASSED
2021-06-22T11:04:30: %AETEST-INFO: +----------------------------------------------------------+
2021-06-22T11:04:30: %AETEST-INFO: | STEPS Report |
2021-06-22T11:04:30: %AETEST-INFO: +----------------------------------------------------------+
2021-06-22T11:04:30: %AETEST-INFO: STEP 1 - Disconnecting from csr1000v-1 Passed
2021-06-22T11:04:30: %AETEST-INFO: ------------------------------------------------------------
2021-06-22T11:04:30: %AETEST-INFO: The result of subsection disconnect_from_devices is => PASSED
2021-06-22T11:04:30: %AETEST-INFO: The result of common cleanup is => PASSED
2021-06-22T11:04:30: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:30: %AETEST-INFO: | Detailed Results |
2021-06-22T11:04:30: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:30: %AETEST-INFO: SECTIONS/TESTCASES RESULT
2021-06-22T11:04:30: %AETEST-INFO: --------------------------------------------------------------------------------
2021-06-22T11:04:30: %AETEST-INFO: .
2021-06-22T11:04:30: %AETEST-INFO: |-- common_setup PASSED
2021-06-22T11:04:30: %AETEST-INFO: | `-- connect_to_devices PASSED
2021-06-22T11:04:30: %AETEST-INFO: | `-- Step 1: Connect to csr1000v-1 PASSED
2021-06-22T11:04:30: %AETEST-INFO: |-- FirstTestcase FAILED
2021-06-22T11:04:30: %AETEST-INFO: | |-- setup PASSED
2021-06-22T11:04:30: %AETEST-INFO: | `-- verify_ios_version FAILED
2021-06-22T11:04:30: %AETEST-INFO: | `-- Step 1: Checking the IOS version FAILED
2021-06-22T11:04:30: %AETEST-INFO: `-- common_cleanup PASSED
2021-06-22T11:04:30: %AETEST-INFO: `-- disconnect_from_devices PASSED
2021-06-22T11:04:30: %AETEST-INFO: `-- Step 1: Disconnecting from csr1000v-1 PASSED
2021-06-22T11:04:30: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:30: %AETEST-INFO: | Summary |
2021-06-22T11:04:30: %AETEST-INFO: +------------------------------------------------------------------------------+
2021-06-22T11:04:30: %AETEST-INFO: Number of ABORTED 0
2021-06-22T11:04:30: %AETEST-INFO: Number of BLOCKED 0
2021-06-22T11:04:30: %AETEST-INFO: Number of ERRORED 0
2021-06-22T11:04:30: %AETEST-INFO: Number of FAILED 1
2021-06-22T11:04:30: %AETEST-INFO: Number of PASSED 2
2021-06-22T11:04:30: %AETEST-INFO: Number of PASSX 0
2021-06-22T11:04:30: %AETEST-INFO: Number of SKIPPED 0
2021-06-22T11:04:30: %AETEST-INFO: Total Number 3
2021-06-22T11:04:30: %AETEST-INFO: Success Rate 66.7%
2021-06-22T11:04:30: %AETEST-INFO: --------------------------------------------------------------------------------
You can see that the CSR is not running my set standard version of IOS-XE code (17.3.3), which is why you see the ‘Failed’ result for Step 1 under the verify_ios_version test. Instead, the CSR is running IOS-XE 16.9.3. Our testscript worked!
Conclusion
There was a lot to go over in this blog post, and I can tell you that I just scratched the surface. AEtest is a phenomenal testing framework and provides a structured, but flexible, way to organize and execute testscripts. I hope you were able to learn something from this post and have the time to review the pyATS docs themselves to learn more. In the final part to this series (Part 4), we are going to go over some hidden gem libraries I found in pyATS and Genie that could be useful.
As always, if you have any questions or would like to have a discussion about any topics in this blog post, feel free to hit me up on Twitter (@devnetdan). Thanks again for reading and I’ll catch you in Part 4!
References
pyATS AEtest docs: AEtest – Test Infrastructure — pyATS Documentation
pyATS – Container classes: Object Model — pyATS Documentation
Easypy Runtime Environment docs: Easypy – Runtime Environment — pyATS Documentation
DevNet Dan 