Nuage MetroAE configuration template engine.
Version 1.1.0
This tool reads Excel spreadsheets (.xlsx files) or Yaml files of templates and user-data to write a configuration to a VSD or to revert (remove) said configuration.
MetroAE config is a Python-based engine which can apply configuration to VSDs via templates. The templates provide an abstraction around the VSD configuration model simplifying and validating the required data. Data is provided by the user in the form of Yaml files or Excel spreadsheets (.xlsx files) and is transformed and applied to the VSD through the tool's templates. The required data and proper format for each template is defined by standardized JSON schema specifications.
There are several modes of operation for MetroAE config. For most users, the tool should be consumed via the MetroAE Docker container. Using this method ensures that all of the libraries are properly installed.
The Docker container gets built on the fly when running metroae-config. The script
will build and then invoke the container.
If using the Docker container, all remaining installation instructions can be skipped.
The MetroAE config tool is completely implemented as a Python library. As such, the functionality of the tool can be integrated into other Python tools or libraries through the use of the APIs. To do this, the library can be installed via Python pip package from pypi.org:
pip install nuage-metroae-config
The pip installation of the package will pull in all dependent libraries
automatically. The following section of installing from Github can be skipped.
The use of the Library APIs is documented in the Python Library API section.
Using the MetroAE config tool without the Docker container can be accomplished by pulling the source from Github. You may want to use this method if you are comfortable working with Python environments and don't want the overhead of a container. The instructions for this case is described in the next section. If using this method, you would be responsible for installing all of the necessary prerequiste libraries and to obtain the VSD API specs and templates.
This section describes how to install the MetroAE config tool from the Github repo. These steps will be necessary to use the metroae_config command-line tool directly. If using the MetroAE Docker container, this section should be skipped.
The primary requirements for running MetroAE config are:
- Installation of Python and the necessary packages.
- Installation of the config engine python libraries
- Access to the Configuration templates
- Access to the VSD API Specification
Most linux distributions come with python bundled into the operating system. Python 2.7 is required. Additionally, Python 3 is also supported.
To verify that python is available we can do the following: (Note: All examples provided below are in Python 2.7 syntax)
Ubuntu
root@ubuntu:~# python --version
Python 2.7.5
RHEL/Centos
[root@rhel]# python --version
Python 2.7.5
To manage the python package installation we will use PIP. Install PIP via apt-get or yum:
Ubuntu
root@ubuntu:~# apt-get install python-pip
...
RHEL/Centos
[root@rhel]# yum install python-pip
...
Most packages required by MetroAE config are available as part of the base python functionality. However we will need to install the following:
- Bambou
- Jinja2 (min version 2.10)
- PyYAML
- lark-parser
Additional packages for unit-test
- mock
- pytest
- requests
- requests-mock
- lark_parser
The following packages are installed via the same method on both Ubuntu and RHEL/Centos.
Ubuntu and RHEL/Centos
[root@rhel]# pip install -r requirements.txt
...
Currently MetroAE config is available on github and will be installed via git clone. Check the MetroAE home page for latest master branch location.
First we need to install git.
Ubuntu
root@ubuntu:~# apt-get install git
RHEL/Centos
[root@rhel]# yum install git
The repository is available via Github:
https://github.com/nuagenetworks/nuage-metroae-config
Depending on the authentication method with github, ie. SSH Key, or Username/Password the git clone command line may change.
For username/password authentication
Ubuntu and RHEL/Centos
[root@rhel]# export GIT_SSL_NO_VERIFY=false
[root@rhel]# git clone https://github.com/nuagenetworks/nuage-metroae-config.git
Cloning into 'levistate'...
Username for 'https://github.com/nuagenetworks/nuage-metroae-config.git': sfiddian
Password for 'https://[email protected]':
remote: Counting objects: 619, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 619 (delta 1), reused 0 (delta 0), pack-reused 610
Receiving objects: 100% (619/619), 178.01 KiB | 0 bytes/s, done.
Resolving deltas: 100% (398/398), done.
MetroAE Config requires a description of the VSD API to create, read, update and delete the template contents into the VSD. We do this via reading the published VSD API Specification. This specificaiton is opensourced and is available on the public Nuage Networks github repository.
Ubuntu and RHEL/Centos
[root@rhel]# git clone https://github.com/nuagenetworks/vsd-api-specifications.git
Cloning into 'vsd-api-specifications'...
remote: Counting objects: 23428, done.
remote: Total 23428 (delta 0), reused 0 (delta 0), pack-reused 23428
Receiving objects: 100% (23428/23428), 4.98 MiB | 1013.00 KiB/s, done.
Resolving deltas: 100% (18163/18163), done.
When using the MetroAE Docker container, all commands are accessed through the
metroae-config command-line tool as:
./metroae-config ...
If you are using the Python source obtained through Github, then any reference
to metroae-config should substitute the equivalent Python source tool:
python metroae_config.py ...
MetroAE Config command-line tool usage:
usage: ./metroae-config [-h]
{create,revert,validate,list,schema,example,upgrade-templates,version,help}
...
Version 1.0 - This tool reads JSON or Yaml files of templates and user-data to
write a configuration to a VSD or to revert (remove) said configuration. See
README.md for more.
positional arguments:
{create,revert,validate,list,schema,example,upgrade-templates,version,help}
optional arguments:
-h, --help show this help message and exit
usage: ./metroae-config create [-h] [-tp TEMPLATE_PATH] [--version]
[-sp SPEC_PATH] [-dp DATA_PATH] [-d DATA]
[-v VSD_URL] [-u USERNAME] [-p PASSWORD]
[-e ENTERPRISE] [-lg]
[datafiles [datafiles ...]]
positional arguments:
datafiles Optional datafile
optional arguments:
-h, --help show this help message and exit
-tp TEMPLATE_PATH, --template_path TEMPLATE_PATH
Path containing template files. Can also set using
environment variable TEMPLATE_PATH
--version Displays version information
-sp SPEC_PATH, --spec_path SPEC_PATH
Path containing object specifications. Can also set
using environment variable VSD_SPECIFICATIONS_PATH
-dp DATA_PATH, --data_path DATA_PATH
Path containing user data. Can also set using
environment variable USER_DATA_PATH
-d DATA, --data DATA Specify user data as key=value
-v VSD_URL, --vsd_url VSD_URL
URL to VSD REST API. Can also set using environment
variable VSD_URL
-u USERNAME, --username USERNAME
Username for VSD. Can also set using environment
variable VSD_USERNAME
-p PASSWORD, --password PASSWORD
Password for VSD. Can also set using environment
variable VSD_PASSWORD
-e ENTERPRISE, --enterprise ENTERPRISE
Enterprise for VSD. Can also set using environment
variable VSD_ENTERPRISE
-lg, --logs Show logs after run
Apply enterprise, domain and ACLs to VSD.
$ ./metroae-config create -tp ./sample/templates -sp ./vsd-api-specifications -v https://localhost:8443 ./sample/user_data/acls.yaml
Configuration
Enterprise
name = 'test_enterprise'
DomainTemplate
name = 'template_public'
[store id to name domain_template_id]
Domain
name = 'public'
templateID = [retrieve domain_template_id (DomainTemplate:id)]
IngressACLTemplate
priority = 100
defaultAllowNonIP = False
allowAddressSpoof = False
name = 'test_acl'
defaultAllowIP = True
IngressACLEntryTemplate
networkID = ''
stateful = True
protocol = 6
description = 'Test ACL'
etherType = '0x0800'
statsLoggingEnabled = True
DSCP = '*'
priority = 200
action = 'FORWARD'
locationID = ''
destinationPort = '*'
locationType = 'ANY'
sourcePort = 80
networkType = 'ANY'
flowLoggingEnabled = True
EgressACLTemplate
priority = 100
defaultInstallACLImplicitRules = True
defaultAllowNonIP = False
name = 'test_acl'
defaultAllowIP = True
EgressACLEntryTemplate
networkID = ''
stateful = True
protocol = 6
description = 'Test ACL'
etherType = '0x0800'
statsLoggingEnabled = True
DSCP = '*'
priority = 200
action = 'FORWARD'
locationID = ''
destinationPort = '*'
locationType = 'ANY'
sourcePort = 80
networkType = 'ANY'
flowLoggingEnabled = True
Revert (remove) objects configured during application, use -r option
$ ./metroae-config revert -tp ./sample/templates -sp ./vsd-api-specifications ./sample/user_data/acls.yaml
You can also use your own custom templates with MetroAE Config. You can put your custom templates in the same directory as our standard templates or you can put them in one or more custom directories.
Note: Your custom templates must have unique template names (not file names... The template name inside the template file...) to avoid name collisions with standard templates. If two templates have the same name, MetroAE Config will use the template and VSD versions to determine which template to apply when run. This may result in unexpected behavior when your template names are the same as standard template names. We recommend that you use a prefix for your template names to guarantee uniqueness. For example, if you create a custom template with the name "Enterprise" there will be a collision. But if you use the name "my_enterprise" there will not be a collision.
The following command tells MetroAE Config to look for templates in the custom location, sample/custom-templates:
$ ./metroae-config create -tp ./sample/custom-templates -sp ./vsd-api-specifications -v https://localhost:8443 sample/user_data/acls.yaml
Note that you can specify more than one template path so that you can utilize templates in more than one directory, for example if your solution needs to use some standard templates and some custom templates. The -tp option can be used to pass multiple template path at the same time:
$ ./metroae-config create -tp ./standard-templates/templates -tp ./sample/custom-templates -sp ./vsd-api-specifications -v https://localhost:8443 ./sample/user_data/acls.yaml
The templates within the MetroAE config tool are applied using data provided by the user. Each set of data values provided allows the creation of an instance of configuration on the VSD. The required format for the data of each template is defined by a JSON schema describing the required fields, types and other constraints on the data.
The user data for MetroAE config can be in either Yaml or JSON format. Each entry in the file defines a template and the value sets to use against that template to instantiate configuration.
- template: Enterprise
values:
- enterprise_name: my_first_enterprise
description: The first
- enterprise_name: my_second_enterprise
description: The second
In this example, two data sets are provided for the Enterprise template and the result will be that the two enterprises will be created on the VSD. Only the enterprise_name is required, but description can be provided optionally. The JSON schema for the Enterprise template defines the data format.
Let's now add domains to the enterprise objects. In the Domain template, note that in addition to a domain_name, it also needs the enterprise_name to be specified so that the domain can be associated with the correct enterprise. We could define our user data as follows:
- template: Enterprise
values:
- enterprise_name: my_first_enterprise
description: The first
- enterprise_name: my_second_enterprise
description: The second
- template: Domain
values:
- enterprise_name: my_first_enterprise
description: The first
domain_name: my_first_domain
- enterprise_name: my_second_enterprise
description: The second
domain_name: my_second_domain
This user data is fully valid and will create two enterprises each with one domain. However, we are duplicating the enterprise names and it would be tedious and error prone if the names needed to change. In order to simplify nested data, children can be utilized as follows:
- template: Enterprise
values:
- enterprise_name: my_first_enterprise
description: The first
children:
- template: Domain
values:
- domain_name: my_first_domain
- template: Enterprise
values:
- enterprise_name: my_second_enterprise
description: The second
children:
- template: Domain
values:
- domain_name: my_second_domain
description: Overriden description
In this case, each Domain is a child of a parent Enterprise. As such, all of the values from the parent are inherited in the children. The children domains automatically inherit the enterprise_name of the parent. This also applies for the description.
Note that values can be overriden if necessary as with the description of the second domain. The most specific value takes precendence. To prevent ambiguity, any template with child templates can only define one set of values.
Another mechanism for reducing duplication in data is by using groups. A group entry is very similar to a template, except that no actual template is instantiated. Thus, the group defines a stand-alone data set. Groups are useful when common data sets need to be defined and referenced elsewhere.
- group: first
values:
- enterprise_name: my_first_enterprise
domain_name: my_first_domain
- group: second
values:
- enterprise_name: my_second_enterprise
domain_name: my_second_domain
- group: ssh
values:
- protocol: tcp
port: 22
- group: html
values:
- protocol: tcp
port: 80
- template: Acl
values:
- acl_name: http_acl
$group_domain: first
$group_traffic: http
action: permit
- acl_name: ssh_acl
$group_domain: second
$group_traffic: ssh
action: deny
In the above example, data sets for domains and traffic types are defined as groups. These are being applied to the http_acl and ssh_acl templates using $group fields and referencing by group name. The template will inherit all of the values from any group that is referenced. The result in this case is that http_acl is created in the first domain and permits http traffic (tcp port 80) and the ssh_acl is created in the second domain and denies ssh traffic (tcp port 22).
Group definitions can be children or have children. However, they should only define one set of values. The field name for group references must start with the string "$group", but can have any suffix following it. The purpose of the suffix is to allow multiple group references without name collision. Although any suffix can be specified, it is recommended to choose one that is descriptive of the reference meaning.
An alternative way of specifying values is by using a field list and set of value lists. This format is similar to Comma Separated Values (CSV) data and can efficiently specify a large amount of data.
- template: Acl
fields: ['acl_name', '$group_domain', 'protocol', 'port', 'action']
values:
- ['acl1', 'first', 'tcp', 22, 'deny']
- ['acl2', 'first', 'udp', 5000, 'permit']
- ['acl3', 'second', 'tcp', 80, 'deny']
- ['acl4', 'second', 'udp', 5002, 'permit']
A fields list must be specified to define the field name for each position of the value lists. The values data sets must be specified as lists with the same length as the fields list.
The templates that have been loaded into the MetroAE config tool can be listed with the following:
$ ./metroae-config -tp ./sample/templates --list
Domain
Enterprise
Subnet
Zone
An example of user data for any template can be provided using the following:
./metroae-config example -tp sample/templates Domain
# First template set - Create a L3 Domain
- template: Domain
values:
- enterprise_name: "" # (reference)
domain_name: "" # (string)
description: "" # (opt string)
underlay_enabled: enabled # (['enabled', 'disabled', 'inherited'])
address_translation: enabled # (['enabled', 'disabled', 'inherited'])
A JSON schema can be generated for the user data required for any template. These schemas conform to the json-schema.org standard specification:
./metroae-config schema -tp ./sample/templates Domain
{
"title": "Schema validator for Nuage Metro config template Domain",
"$id": "urn:nuage-metro:config:template:domain",
"required": [
"enterprise_name",
"domain_name",
"underlay_enabled",
"address_translation"
],
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"underlay_enabled": {
"enum": [
"enabled",
"disabled",
"inherited"
],
"title": "Underlay enabled"
},
"address_translation": {
"enum": [
"enabled",
"disabled",
"inherited"
],
"title": "Address translation"
},
"domain_name": {
"type": "string",
"title": "Domain name"
},
"enterprise_name": {
"type": "string",
"title": "Enterprise name"
},
"description": {
"type": "string",
"title": "Description"
}
}
}
MetroAE config can be used as a generic data querying tool to extract information from devices. At the time of this writing VSD and ES are supported. The tool uses a simple language to describe the data that is to be retrieved. The data will be displayed to the screen in a YAML like format, but it could (in addition) be written to a file. The results are also returned as a Python list when using query tool as a library.
The query tool is used by specifying "query" as the action to metroae_config.
The parameters are mostly the same as other actions supported by
metroae_config. The VSD specification path -sp is required if querying a
VSD. The other VSD parameters (i.e. user, pass, URL, etc) also apply for
query in the same way as configuration creation actions.
The query tool also supports queries to ElasticSearch (ES). In this case, a
new -es parameter has been added to specify the address of the ES to query.
The query language can be specified to the tool directly on the command-line
for easy manual use using the -q query parameter. An example follows:
./metroae-config query -v https://localhost:8443 -q 'enterprise.name'
Device: Nuage Networks VSD 20.5.1
- Shared Infrastructure
- public
- private
>>> All actions successfully applied
Multiple queries can be specified on a single line using semicolon ;:
./metroae-config query -v https://localhost:8443 -q 'apps = SAASApplicationType.name; ents = enterprise.name'
Device: Nuage Networks VSD 20.5.1
apps:
- Office365
- SalesForce
- WebEx
ents:
- Shared Infrastructure
- public
- private
>>> All actions successfully applied
The language definition for query tool can also be specifed in a file to be read. One or more of these files (or a directory containing .query files) can be specifed on the command-line for the tool to read and execute each file. Newlines in files separate commands. However, semicolons are still supported in files as well.
example.query:
apps = SAASApplicationType.name
ents = enterprise.name
./metroae-config query -v https://localhost:8443 example.query
Device: Nuage Networks VSD 20.5.1
apps:
- Office365
- SalesForce
- WebEx
ents:
- Shared Infrastructure
- public
- private
>>> All actions successfully applied
The query tool uses a query language to specify the data to be reteived from
devices. This language is intended to be simple to read and use and is easily
extendable.
Each query to be performed is specified one per-line. Alternatively and
equivalently, semicolons ; can be used to separate multiple queries on a
single line. Any query beginning with a hash # indicates a comment and will
be ignored. Outside of strings, any whitespace is ignored.
Data retrieval commands are specified by providing nested object names
separated by dot . with the attribute as the last identifier. A retrieval
traverses all instances of each level of the specified object tree and returns
the attribute value for every instance found.
The result is always a flat list of the attribute values. Any heirarchy is flattened. If no results are found or there is an error (such as missing child object) then an empty list is returned. With this procedure, processing results is consistent as there will always be a list.
For VSD:
Enterprise.Domain.Zone.Subnet.name
The above (reading from right to left) returns a list of the names from all subnets under all zones under all domains under all enterprises. For VSD queries, each object is a configuration object specified by entity name. The names here are case-insensitve although note that case may matter for other devices (like ES).
For ElasticSearch:
nuage_sysmon.disks.available
The above (reading from right to left) returns the available values for all disks under the nuage_sysmon index. For ES, the first object is the ES index and then subsequent object names traverse the JSON objects stored in each ES record. Since ES encodes using JSON, the object names and attribures are all case-sensitive as required by the format.
The last identifier on a data retrieval is the attribute. It determines which
value to extract from the objects traversed. In previous examples, a single
value was extracted, however multiple attributes can be gathered by specifying
a comma-separated list in curly braces {}. When retrieving multiple
attributes, the result becomes a list of dictionaries with key/value pairs
for each of the attributes. A star * can be used to get all available
attributes.
Single attribute:
Enterprise.name
- Shared Infrastructure
- public
- private
Multiple attributes:
Enterprise.{name,id}
- name: Shared Infrastructure
id: abcd-0001
- name: public
id: abcd-0002
- name: private
id: abcd-0003
All attributes:
Enterprise.{*}
- name: Shared Infrastructure
id: abcd-0001
bgpenabled: false
dhcpleaseinterval: 24
(Whole bunch of stuff omitted for brevity...)
- name: public
id: abcd-0002
bgpenabled: false
dhcpleaseinterval: 24
(Whole bunch of stuff omitted for brevity...)
- name: private
id: abcd-0003
bgpenabled: false
dhcpleaseinterval: 24
(Whole bunch of stuff omitted for brevity...)
Filters can be applied to each object of the data retrieval. Filters are
enclosed by square brackets [] and are separated by and & characters. The
choice of & as a separator enforces that filters are combined together as
logical AND operations.
The filter applies only to the level where it is specified. The filters manipulate the results from the level, usually by removing unwanted records. Any subsequent object levels would operate on the filtered list. For example:
Enterprise[name=public].domain.name
The filter on enterprise will limit enterprise results to only a single
enterprise object (with name public). However, let's say the public
enterprise had 10 domains. Then the 10 name results for those domains would be
returned.
A range filter can be used to limit the start and max number of results at the
level. The filter has a start index and an end index separated by colon :.
Indexes are 0-based positions of the list. Thus 0 is the first and 4 would be
the 5th item in the list. The items from the query are returned starting at
the start index and ending one before the end index. Any range out of bounds
of the results are omitted. Thus if the start index is after the end of the
results, an empty list is returned.
If the start index is omitted, a value of 0 is assumed (start at beginning). If the end index is omitted, then the entire remainder of the list is returned.
A single index (without colon specified) can be used to return the single item from the results at the specified index. If this is out of bounds of the results, an empty list is returned.
Negative indicies are allowed and count from the end of the list. Thus -1 is the last item and -5 is the 5th from last.
The behavior of the range filter is exactly the same as Python list slicing.
Note that ES index queries do not support negative range indicies. Use a
reverse sort and positive indicies instead.
Examples using enterpises: e1, e2, e3, e4, e5
enterprise[1:5].name
- e2
- e3
- e4
enterprise[1:99].name
- e2
- e3
- e4
- e5
enterprise[:5].name
- e1
- e2
- e3
- e4
enterprise[3:].name
- e4
- e5
enterprise[99:].name
[]
enterprise[3].name
- e4
enterprise[99].name
[]
enterprise[-3:-1].name
- e3
- e4
Object results can be filtered by attribute values. Only results where the
field value of the object matches the filter value will be returned. The
format for these filters is the field name equal = value. A list of values
can be provided enclosed in square brackets [] and separated by comma ,.
This denotes that any result matching any of the list values will be returned.
Given these enterprises:
- name: e1
BGPEnabled: true
- name: e2
BGPEnabled: false
- name: e3
BGPEnabled: true
The following examples produce:
enterprise[BGPEnabled=true].name
- e1
- e3
enterprise[name=[e2, e3]].name
- e2
- e3
enterprise[name=[e2, e3] & BGPEnabled=false].name
- e2
enterprise[name=[e1, e3] & BGPEnabled=false].name
[]
The sort filter can be used to sort the results by an object field. Note that the sorting occurs only at the level where it is specified. If there are child objects under a sort fitler the children would not be sorted, but they would be queried in the parent's sorted order.
Sorting is specified by the %sort= keyword followed by the field name of the
object to sort by. There is also %sort_desc= to sort in descending order.
Given these enterprises:
- name: e3
dhcpleaseinterval: 1
- name: e1
dhcpleaseinterval: 2
- name: e2
dhcpleaseinterval: 3
The following examples produce:
enterprise[%sort=name].name
- e1
- e2
- e3
enterprise[%sort=DHCPLeaseInterval].name
- e3
- e1
- e2
enterprise[%sort_desc=name].name
- e3
- e2
- e1
A group filter can be specified to combine together results by the value of another field. This makes it easier to identify subsets of results given that the query tool collapses them all into flat lists.
The group filter is specified by %group= followed by the field name to group
by. When using groups, results are returned as a list of pairs. The first
item of the pair is the group field value and the second is the list of results
matching the group field value.
For example, normally results are flattened and thus it is difficult to determine which domain goes with which enterprise:
enterprise.domain.name
- domain1
- domain2
- domain3
- domain4
However, using grouping the relationships can be determined:
enterprise[%group=name].domain.name
- - public
- - domain1
- domain2
- domain3
- - private
- - domain4
- - Shared Infrastructure
- []
Results can be stored in variables for identification and use in later queries.
Variables are restricted to C-like names which cannot start with a number and
can contain capital letters, lower case letters, numbers or underscore _.
Variable assignments are performed by specifying the variable name then equal
= then the expression or data type to store in the variable. The output of
assigments changes to a dictionary.
enterprise_names = enterprise.name
enterprise_names:
- public
- private
- Shared Infrastructure
Variables are dynamically typed and can contain any data type. More
information about data types will be provided in a later section. Variables
can be dereferenced in later queries by specifying dollar $ and the variable
name. Variables that have been dereferenced but not yet defined will cause an
error.
Variables can be filtered:
enterprises = enterprise.{name,id}
enterprises:
- name: public
id: abcd-0001
- name: private
id: abcd-0002
$enterprises[%sort=name]
- name: private
id: abcd-0002
- name: public
id: abcd-0001
$enterprises[%sort=name].id
- abcd-0002
- abcd-0001
Variables can also be used as the values for filters:
enterprises = enterprise.name
enterprises:
- e1
- e2
- e3
- e4
start = 2
start: 2
$enterprises[$start:]
- e3
- e4
name = "e2"
name: e2
enterprise[name=$name].{name,id}
- name: e2
id: abcd-0002
Variable values can be specified on the command-line using the -d option.
The format is variable name = value. i.e. -d name=enterprise1. When
variables are defined on the command-line, they override any assignment
within queries. This allows defaults to be set inside query files and -d to
override with a specific value.
The query language defines various data types to be specified. Variables are dynamically typed and can accept any of these types.
Strings are defined enclosed with single ' or double " quotes. Multi-line
blocks can be defined using triple single ''' or double """ quotes.
string = 'this is a string'
string = "another has ' in it"
multiline = """
this has
multiple lines
"""
Numbers can be specified as positive or negative integers. Floating point numbers are not supported.
integer = 42
negative = -4
For Booleans, true specifies true and false specifies false.
is_true = true
is_false = false
Lists can be formed using square brackets [] and separating items by comma
,. Lists can contain any of the above data types even in combinations.
However, lists of lists are not supported.
enterprise_names = ['e1', 'e2', 'e3']
There are a set of functions that can operate on expressions such as data
retrieval or variables. These come in the form of function name followed by
the expression enclosed in parens ().
The count function returns the number of items in a list. An error is
produced if the expression provided to count is not a list.
names = enterprise.name
names:
- e1
- e2
- e3
count($names)
3
The reverse function returns the items of a list in backward order. An error
is produced if the expression provided to reverse is not a list.
reverse(enterprise.name)
- e3
- e2
- e1
There is a combine operator which combines expressions together depending on
the data types. The format is expression1 plus + expression2. The
expressions can be variables, data type constants or data retrieval.
If both expressions are lists, then the combine operator will return a list with all of the items of both.
names = enterprise.name
names:
- e1
- e2
- e3
$names + ["new1", "new2"]
- e1
- e2
- e3
- new1
- new2
If both expressions are integers, then the combine operator will add the values.
count(enterprise.name) + 1
4
If both expressions are strings, then the strings will be concatenated.
filename = "output"
filename: output
path = "/tmp/" + $filename + ".txt"
path: /tmp/output.txt
Any other combination of data types will result in an error from combine.
There are a set of actions that can change the state of later queries or
perform a task. These come in the form of action name followed by a list of
comma-separated arguments enclosed in parens (). Actions do not return any
value and they cannot be part of an assignment or expression.
The connect action sets up a new session with a device. All subsequent
queries are directed to the new device. The first argument to connect is the
device type, then the remaining parameters are specific to the device. Using
connect allows data to be gathered from multiple sources in the same query
execution.
For VSD:
connect("VSD", url, username, password, enterprise, cert_file, cert_key_file)
Where:
url (required) : URL of VSD to connect to
username (optional) : Username to be used to connect (default "csproot")
password (optional) : Password to be used to connect (default "csproot")
enterprise (optional) : Enterprise to be used to connect (default "csp")
cert_file (optional) : Path to a certificate file (instead of password)
cert_key_file (optional) : Path to a certificate key file (required if cert_file specified)
For ElasticSearch:
connect("ES", address, port)
Where:
address (required) : Address of ElasticSearch to connect to
port (optional) : Port of ElasticSearch (default 9200)
The redirect_to_file action causes all subsequent queries to write results
to the specified file in addition to echoing to the screen.
redirect_to_file("/tmp/output.txt")
The echo action turns on and off the printing of output of subsequent queries
to the screen. This only affects the display of the queries and they still
operate normally in all other ways. Variables would still be assigned and
output would still be written to file if redirected. Echoing is on by default.
it can be turned off with the string off and turned back on with the string
on. Disabling echoing is useful when huge amounts of data are expected to be
returned but it would clutter the screen.
echo("off")
The output action turns on and off the writing of results to the screen,
redirect file and Python returned results. Variables are still assigned when
output is off. Turning off output is useful for gathering all intermediate
variables and then turning it on to write out a single final report. Output is
on by default. It can be turned off with the string off and turned back on
with the string on.
output("off")
The render_template action outputs the text of a variable substituted
template. The action takes a Python Jinja
template string as an argument. Jinja templates contain tags enclosed by
double curly braces {{ }} that reference variables to be substituted into
the text. The variables assigned from queries are all available to be injected
into the text output. Jinja templates support loops, conditionals, math and
other filters. See the Python Jinja
documentation for more. Using Jinja templates, many file formats can be output
including nice text reports, HTML pages, CSV or JSON.
The render_yaml_template action is exactly the same as render_template
except that substituted variables are formatted properly for YAML/JSON output.
For example strings are quoted, booleans are converted to true/false, etc.
Example query file:
output("off")
connect("VSD", "https://localhost:8443")
report_file = "report.txt"
ent_names = enterprise.name
ent_count = count($ent_names)
licenses_expires = License.expirytimestamp
template = """============== VSD Health Report ===========
VSD:
Enterprises: {{ ent_names | join(', ') }} ({{ ent_count }})
License expiry: {{ ((licenses_expires[0] / 1000 - now) / (3600 * 24)) | int }} days
===========================================
"""
redirect_to_file($report_file)
output("on")
render_template($template)
Resulting output to report.txt:
============== VSD Health Report ===========
VSD:
Enterprises: public, private, Shared Infrastructure (3)
License expiry: 90 days
===========================================
The MetroAE Config engine is written entirely using Python. It can be integrated
into other Python-based tools using the library APIs via the pip package
nuage-metroae-config. The following sections describe the use of the APIs.
The TemplateStore class is a repository for all of the templates loaded into
the engine. It will be used by the Configuration class to apply template
actions to a device via a DeviceWriterBase class.
from nuage_metroae_config.template import TemplateStore
Reads and parses configuration templates.
-
read_templates(path_or_file_name)
Reads and parses templates from either all templates in a directory path, or a single template specified by filename. Both yaml (.yml) and JSON (.json) files are supported. -
add_template(template_string, filename=None)
Parses the specified string as a template in Yaml or JSON format. -
get_template_names(software_type=None, software_version=None)
Returns a list of all template names currently loaded in store. If software_version and/or software_type is provided, names will be filtered by the specified version/type. -
get_template(name, software_type=None, software_version=None)
Returns a Template object of the specified name. If software_version and/or software_type is provided, template of specified version/type will be returned.
The Template class is a read-only class which provides information about a
template that was read in via the TemplateStore.
from nuage_metroae_config.template import Template
Configuration template. This class is read-only.
-
get_name()
Returns the name of this template. -
get_template_version()
Returns the template version -
get_software_version()
Returns a dictionary of {"software_version": "xxx", "software_type": "xxx"} -
get_schema()
Returns the schema for the template variables in json-schema form. -
get_example()
Returns example user-data for template variables in YAML format. -
get_documentation()
Returns template documentation in MarkDown format. -
validate_template_data(template_data)
Validates that the template_data provided matches the variables schema. Returns True if ok, otherwise an exception is raised.
The VsdWriter class is a derived class of the DeviceWriterBase that applies
configuration to a Nuage Networks VSD. The writers are modular such that
new devices could be configured by defining derived device writer classes based
off of the common DeviceWriterBase base class. The remainder of the engine
APIs use the generic base class. This provides great extendability to
configure many different devices. Note that the VsdWriter class also acts as
a reader for the Query class.
from nuage_metroae_config.vsd_writer import VsdWriter
Writes configuration to a VSD. This class is a derived class from
the DeviceWriterBase Abstract Base Class.
-
set_session_params(url, username="csproot", password=None, enterprise="csp", certificate=None)
Sets the parameters necessary to connect to the VSD. This must be called before writing or an exception will be raised. -
get_version():
Returns the version running on the VSD in format: {"software_version": "xxx", "software_type": "xxx"} -
set_software_version(software_version):
Sets the software version of the VSD. This can be obtained via the get_version() method to get from the VSD itself or overwriten to a specific value. -
read_api_specifications(path_or_file_name):
Reads the VSD configuration API specifications from JSON files in the specified path or file name. This must be called before writing or an exception will be raised. -
set_logger(logger):
Set a custom logger for actions taken. This should be based on the logging Python library. It will need to define an 'output' log level which is intended to print to stdout.
The UserDataParser class parses the YAML-based user-data format defined in
the User Data section of this document. It is able to return the data in
template_name/data_dictionary pairs. This data can be fed into the
Configuration class to render templates and apply configuration.
from nuage_metroae_config.user_data_parser import UserDataParser
Parses Yaml or JSON files containing template user data
-
read_data(path_or_file_name):
Reads and parses user data from either all files in a directory path, or a single file specified by filename. Both yaml (.yml) and JSON (.json) files are supported. -
add_data(user_data_string):
Parses the specified string as user data in Yaml or JSON format. -
get_template_name_data_pairs():
Returns parsed data as a list of (template_name, data_dict) pairs.
The Configuration class puts all the above pieces together to configure a
device. It requires a TemplateStore object with loaded templates. User data
is applied to the Configuration object which can be taken directly from the
UserDataParser class or from pure Python data via another source. Finally,
the configuration is applied, updated or reverted to a device using one of the
derived writer classes.
from nuage_metroae_config.configuration import Configuration
Container for template instances.
-
set_software_version(software_type=None, software_version=None)
Sets the current software version of templates that is desired. If not called, the latest software version of templates will be used. -
add_template_data(template_name, template_data)
Adds template data (user data) for the specified template name. Data is specified in a kwargs dictionary with keys as the attribute/variable name. The data is validated against the corresponding template schema. An id is returned for reference. -
apply(writer)
Applies this configuration to the provided device writer. Returns True if ok, otherwise an exception is raised. -
update(writer)
Applies this configuration to the provided device writer as an update. This means objects that exist will not be considered conflicts. Returns True if ok, otherwise an exception is raised. -
revert(writer)
Reverts (removes or undo) this configuration from the provided device writer. Returns True if ok, otherwise an exception is raised. -
set_logger(logger):
Set a custom logger for actions taken. This should be based on the logging Python library. It will need to define an 'output' log level which is intended to print to stdout.
The Query class allows information to be retrieved from a device through
device readers based on the DeviceReaderBase. Note that the VsdWriter
class is also a reader derived off of this base and can be used for query. The
information to be retrieved is defined by a "query language" which is described
in the Language Definition section of this document. Results are returned
from the execute() of the query language text. Also, any variables set during
the query can be retrieved with get_variables().
from nuage_metroae_config.query import Query
A class which can retrieve information from a device using a
DeviceReaderBase reader object. The information to be retrieved is defined
by query language text.
-
set_reader(reader)
Set the primary reader object for performing queries on the device. The reader needs to be derived from the DeviceReaderBase class. -
get_variables()
Returns a dictionary of all variables and values set during queries. -
register_reader(reader_type, reader)
Register a reader object for each 'connect' type for performing queries on the device. This allows devices to be switched within the query text. The readers need to be derived from the DeviceReaderBase class. -
add_query_file(path_or_file_name)
Reads and parses query sets from either all query files in a directory path, or a single query file specified by filename. Query files are expected to have .query extension -
execute(query_text=None, override_variables)
Executes the query defined in query_text. Variables in the query text can be overriden using the override_variables kwargs dict. The results of the query are returned as a list with an entry for each query text line. If query_text is None, then query text is pulled from query files defined by add_query_file. -
set_logger(logger)
Set a custom logger for actions taken. This should be based on the logging Python library. It will need to define an 'output' log level which is intended to print to stdout.
The EsReader class can be used as another reader with the Query class. It
is derived from the DeviceReaderBase class and can retrieve information from
ElasticSearch.
from nuage_metroae_config.es_reader import EsReader
Performs queries on ElasticSearch. This class is a derived class from
the DeviceReaderBase Abstract Base Class.
-
set_session_params(address, port=None):
Sets the parameters necessary to connect to the ES. This must be called before reading or an exception will be raised.
from nuage_metroae_config.configuration import Configuration
from nuage_metroae_config.template import TemplateStore
from nuage_metroae_config.user_data_parser import UserDataParser
from nuage_metroae_config.vsd_writer import VsdWriter
ENGINE_VERSION = "1.1.0"
VSD_API_SPEC_PATH = "./vsd-api-specifications"
TEMPLATE_PATH = "./templates"
USER_DATA_PATH = "./user_data"
VSD_URL = "https://10.0.0.1:8443"
VSD_USERNAME = "username"
VSD_PASSWORD = "password"
VSD_ENTERPRISE = "csp"
def setup_template_store(template_path):
store = TemplateStore(ENGINE_VERSION)
store.read_templates(template_path)
return store
def setup_vsd_writer(vsd_url):
vsd_writer = VsdWriter()
vsd_writer.add_api_specification_path(VSD_API_SPEC_PATH)
vsd_writer.set_session_params(vsd_url,
username=VSD_USERNAME,
password=VSD_PASSWORD,
enterprise=VSD_ENTERPRISE)
device_version = vsd_writer.get_version()
vsd_writer.set_software_version(device_version)
return vsd_writer
def setup_configuration(template_store, user_data_path):
config = Configuration(template_store)
parser = UserDataParser()
template_data_pairs = parser.read_data(user_data_path)
for data in template_data_pairs:
template_name = data[0]
template_data = data[1]
config.add_template_data(template_name, **template_data)
return config
# Setup
template_store = setup_template_store(TEMPLATE_PATH)
vsd_writer = setup_vsd_writer(VSD_URL)
config = setup_configuration(template_store, USER_DATA_PATH)
# Apply the configuration from user data
config.apply(vsd_writer)
# Revert the configuration from user data
config.revert(vsd_writer)
from nuage_metroae_config.es_reader import EsReader
from nuage_metroae_config.query import Query
from nuage_metroae_config.vsd_writer import VsdWriter
VSD_API_SPEC_PATH = "./vsd-api-specifications"
VSD_URL = "https://10.0.0.1:8443"
VSD_USERNAME = "username"
VSD_PASSWORD = "password"
VSD_ENTERPRISE = "csp"
ES_ADDRESS = "10.0.0.2"
ES_PORT = 9200
def setup_vsd_query_tool(vsd_url):
vsd_reader = VsdWriter()
vsd_reader.add_api_specification_path(VSD_API_SPEC_PATH)
vsd_reader.set_session_params(vsd_url,
username=VSD_USERNAME,
password=VSD_PASSWORD,
enterprise=VSD_ENTERPRISE)
device_version = vsd_writer.get_version()
vsd_reader.set_software_version(device_version)
vsd_query = Query()
vsd_query.set_reader(vsd_reader)
return vsd_query
def setup_es_query_tool(es_address):
es_reader = EsReader()
es_reader.set_session_params(es_address, ES_PORT)
es_query = Query()
es_query.set_reader(es_reader)
return es_query
def get_subnetwork_usage(vsd_query):
query_string = """
subnets = Enterprise.Domain.Zone.Subnet.{name, netmask}
zones = Enterprise.Domain.Zone[%group=name].Subnet.{name, netmask}
d_count = count(Enterprise.Domain)
s_count = count(Enterprise.Domain.Zone.Subnet)
z_count = count(Enterprise.Domain.Zone)
"""
vsd_query.execute(query_string)
results = vsd_query.get_variables()
domain_dict = dict()
domain_dict_agg = dict()
# [{"name": <subnet_name>, "netmask": <netmask>}, ...]
domain_dict["subnets"] = results["subnets"]
# [[<zone_name>, [{"name": <subnet_name>, "netmask": <netmask>}, ...]], ...]
domain_dict["zones"] = results["zones"]
domain_dict_agg["d_count"] = results["d_count"]
domain_dict_agg["s_count"] = results["s_count"]
domain_dict_agg["z_count"] = results["z_count"]
return domain_dict, domain_dict_agg
def get_enterprises_with_alarms(vsd_query):
query_string = """
Enterprise[%group=name].Alarm[severity="CRITICAL"].description
"""
results = vsd_query.execute(query_string)
# [[<enterprise_name>, [{"description": <alarm_descr>}, ...]], ...]
enterprises_with_critical_alarms = dict()
count = 0
for result in results:
ent_name = result[0]
alarms = result[1]
if len(alarms) > 0:
enterprises_with_critical_alarms[ent_name] = alarms
count += 1
enterprises_with_critical_alarms['total_enterprises_with_alarms'] = count
return enterprises_with_critical_alarms
# Setup query
vsd_query = setup_vsd_query_tool(VSD_URL)
# Perform queries
domain_dict, domain_dict_agg = get_subnetwork_usage(vsd_query)
enterprises_with_critical_alarms = get_enterprises_with_alarms(vsd_query)