SD-WAN Functional Test Suite

SD-WAN Functional Test Suite is bundle of automation-oriented test capabilities aimed at providing an end-to-end SD-WAN solution relying on feature rich Spirent products. Test Suite development is based on open source projects including PyATS, Genie, and the Robot Framework allowing users to easily configure and execute SD-WAN test capabilities using this suite. Declarative configuration of testbed topology and test equipment parameters in YAML files models testbed objects such as devices and their interfaces intuitively as Python objects. This eliminates costly development and setup times as well as enables reliable and easy-to-interpret results to be produced.

Table of Contents

• Installation
• Test Case Library
• Test Configuration
• Test Execution
• Test Results
• Folder Structure and Naming
• Support
• License

Installation

Follow these instructions to obtain a copy of the project source and get it up and running on your local machine for development and testing purposes. The Test suite is based on Python and is designed work on Linux, MacOS, and Windows Subsystem for Linux.

The steps below are specific to setup on an Ubuntu 18.04 LTS or Windows Subsystem Linux (WSL) running Ubuntu 18.04 LTS. Any differences in setup instructions between the two operating systems are specifically highlighted.

1. Requirement: Python3.6 or higher must be installed on your system.

   On Ubuntu run the following command to install additional packages:

   sudo apt install python3-pip python3-dev python3-venv git
   

   On WSL an update to apt-get as well an additional package is needed to satisfy gcc dependencies:

   sudo apt-get update
   sudo apt install python3-pip python3-dev python3-venv lib32ncurses5-dev git
   

2. Clone the SD-WAN Test Suite repository from Git and cd to that repository root folder.

   git clone https://github.com/Spirent/SDWAN-Functional-Test-Suite.git
   
   cd ~/SDWAN-Functional-Test-Suite
   

3. Setup your Python virtual environment in the repository root folder.

   Create the virtual environment in folder testenv

   python3 -m venv testenv
   

   Activate the virtual environment

   . testenv/bin/activate
   

   Add your root folder to the Python path

   export PYTHONPATH=~/SDWAN-Functional-Test-Suite/
   

4. Install the Spirent Testpack framework, SD-WAN Test Suite, and additional dependencies that are needed to execute the tests.

   Install wheel as required for pip package installations

   pip install wheel
   

   Install the Spirent Testpack Framework source

   pip install -t test_framework spirent-testpack-framework
   

   Install the Spirent SD-WAN Test Suite

   pip install -t testpacks spirent-sdwan-test-suite
   

   Install additional required dependencies such as pyATS, Unicon, Genie, Jinja2, Stcrestclient, and Robotframework.

   pip install -r requirements.txt
   

Test Case Library

This table includes a list of the test scripts and a brief description of their function.

Test Case Name	Test Case ID	Test Objective
Path_Selection_L2_to_L4_Steering	sd-wan.path_selection.001	Validate DUT can steer traffic among WAN links by using traditional L2/L3/L4 traffic classification method
Path_Selection_Application_Aware_Steering	sd-wan.path_selection.002	Validate DUT can steer traffic among WAN links by using application aware traffic classification method
Resiliency_Link_Blackout_Local_No_Congestion	sd-wan.resiliency_link.001	Validate DUT can steer traffic from Internet link to MPLS link if link blackout was detected on local side of internet link and vice versa
Resiliency_Link_Blackout_Remote_No_Congestion	sd-wan.resiliency_link.002	Validate DUT can steer traffic from Internet link to MPLS link if link blackout was detected on remote side of internet link and vice versa
Resiliency_Link_Brownout_Packet_Loss	sd-wan.resiliency_link.003	Validate DUT can steer traffic from Internet link to MPLS link if packet loss ratio on Internet link exceeds threshold and vice versa
Resiliency_Link_Brownout_Packet_Delay	sd-wan.resiliency_link.004	Validate DUT can steer traffic from Internet link to MPLS link if two-way delay (from DUT1 to DUT2) on Internet link exceeds threshold and vice versa
Resiliency_Link_Brownout_Jitter	sd-wan.resiliency_link.005	Validate DUT can steer traffic from Internet link to MPLS link if jitter on Internet link exceeds threshold and vice versa
Resiliency_Link_Brownout_Out_Of_Order	sd-wan.resiliency_link.006	Validate DUT can steer traffic from Internet link to MPLS link if packet out-of-order ratio on Internet link exceeds threshold and vice versa
Resiliency_Link_Brownout_Duplication	sd-wan.resiliency_link.007	Validate DUT can steer traffic from Internet link to MPLS link if packet duplication ratio on Internet link exceeds threshold and vice versa

Test Configuration

Before test execution, follow these steps.

1. Identify the test case name and id that you want to execute from the table in the previous section.

2. A mapping file testpacks/sd-wan/testbed_lab/testbed_map.py is provided that maps individual test case id's to specific testbed sections within the physical testbed configuration file in Step 3. Find the appropriate testbed section that maps to the test case Id from within the mapping file.

3. Once you have identified the physical testbed section that maps to your test case, you must edit that section with configuration details specific to your lab. This section can be found in a single configuration file testpacks/sd-wan/testbed_lab/configuration.yaml which houses configuration for all testbeds. Several other Spirent TestCenter configuration parameters are also exposed, if you need to override the defaults.

4. For SD-WAN scripts there is no DUT configuration in the test script. You must configure the DUT manually before running the test. The gateway address of the emulated devices is the same as with DUT interface IP address.

   Example:

   Step1: This example uses SD-WAN_Path_Selection_Application_Aware_Steering, which is identified by its test case id sd-wan.path_selection.002.

   Step2: In the testpacks/sd-wan/testbed_lab/testbed_map.py file, you will find that for test case id sd-wan.path_selection.002, the physical testbed information being used is 3stc_1dut_type01_testbed02.

   Step3: Find the physical testbed section referenced in Step#2 ("3stc_1dut_type01_testbed02") in the following file testpacks/sd-wan/testbed_lab/configuration.yaml and update it with appropriate values for your lab. Note that it uses chassis_1, chassis_2, chassis_3, ls_1. You must modify the IPv4 address under stc1, stc2, stc3, spirent_lab_server_1, and the slot/port number under chassis_1, chassis_2 and chassis_3.

   Step4: Check the gateway address of emulated_devices under 3stc_1dut_type01_testbed02, which should be the same as the interfaces ip address on the DUT. If they are different, modify the gateway or change the DUT interface ip.

For additional details about each test case, please refer to the TestPack Specification Document. All test cases are explained under their unique Test Case ID in the document.

Test Execution

1. Before you execute the tests it is required to activate Python 3 virtual environment and set your Python path.

Note: This setup is only required one time per shell instance.

If you have already done this as part of installation, then proceed to Step 2. Otherwise run the commands below. ``` . testenv/bin/activate

export PYTHONPATH=~/SDWAN-Functional-Test-Suite/
```

2. Run tests from the root folder. You can run tests using robot commands, as follows:

   robot -v testbed_config:testpacks/sd-wan/testbed_lab/configuration.yaml -V testpacks/sd-wan/testbed_lab/testbed_map.py -t sd-wan.path_selection.002 -K off -d testrun testpacks/sd-wan/
   

   Parameters
   testbed_config: Physical lab Configuration
   .../testbed_map.py: This file maps testcases to specific testbeds in lab configuration file
   -t sd-wan.path_selection.002: Testcase selector
   -d testrun: Output directory for logs/reports
   testpacks/sd-wan/: Folder path where Robot will look for testcase files

   Refer to section Test Results to check the test results.

   Refer to Robot Framework User Guide for a complete syntax of robot commands, including pattern matching for selecting test cases to be executed based on test case ids or tags.

   The -v validate:1 argument is supported for validating testbed files without running tests.

   robot -v testbed_config:testpacks/sd-wan/testbed_lab/configuration.yaml -V testpacks/sd-wan/testbed_lab/testbed_map.py -t sd-wan.path_selection.002 -v validate:1 -d testrun -L INFO testpacks/sd-wan/core.robot
   

Test Results

Output files are configured using robot command line options.

In section:Test Execution, -d testrun specifies the results directory as testrun.

Test execution will generate several reports/logs, for example Robot report files, test script logs, and Spirent TestCenter logs.

1. Robot report files:
   report.html, log.html and output.xml files are typically generated.
   report.html contains an overview of the test execution results in HTML format.
   log.html contains details about the executed test cases in HTML format.
   output.xml contains the test execution results in XML format.
   You can use any browser to open the result.

   firefox report.html
   

2. Test script logs:
   test.log.json under testrun/sd-wan.path_selection.002 contains the test script execution logs.

3. Spirent TestCenter logs:
   There are Spirent TestCenter BLL/IL and configuration logs under testrun/sd-wan.path_selection.002.

Folder Structure and Naming

1. Robot files are organized into testpacks folders, for example: testpacks/sd-wan.

   Robot automation framework is the test runner. Test cases are defined in robot files that accompany Python test scripts. These robot files, such as testpacks/sd-wan/core.robot are the wrappers for Robot framework. They seamlessly call test functions in Python scripts. A set of related test cases can be defined in a single .robot script.

   Every test case has a unique, immutable identifier. This will identify the test case in the metadata file and in the Robot file. Test case ids follow the format <testpack>.<area>.###, that is the unique testpack name along with a 3 digit testcase number relative to a testpack. For example, sd-wan.path_selection.002.

2. Test scripts are organized into testpacks folders testpacks/sd-wan/. They are Python based.

   Every test case is accompanied by metadata, see example testpacks/sd-wan/path_selection_application_aware_steering.yaml.

   Each testpack has a specification to describe the test cases. See example in the TestPack Specification document.

3. The testbed_templates folder is for the logical testbed template. See example: testpacks/sd-wan/testbed_templates/3stc_1dut_type01.yaml, which gets the information from the physical lab configuration and generates the final configuration file used by the test script.

4. The testbed_lab folder is used for the physical configuration and the mapping file. See example: testpacks/sd-wan/testbed_lab/testbed_map.py, which defines the mapping for each test case id to a specific testbed section in physical configuration file testpacks/sd-wan/testbed_lab/configuration.yaml. The physical configuration is a single file to be supplied by the end user that contains complete details of their lab equipment, along with how they are interconnected into testbeds.

5. The script test_framework/script/check_stc_param.py is used to get the Spirent TestCenter port parameters values(phy and speed). When you edit the port 'phy' and 'speed' in physical lab configuration, you can find their values via this script: python script/check_stc_param.py -l <lab server ip> -c <chassis ip> -s <slot number> -p <port number>

Support

If you encounter any issues during installation or test execution, have general questions, or feedback around new features, please open an issue on our Github repository. You can also reach the development team via email [email protected].

License

This Test pack is distributed under the Apache License, Version 2.0, see LICENSE and NOTICES for more information.