netenglabs · jefvantongerloo · Jun 20, 2022 · Jun 20, 2022 · Jun 21, 2022 · Jun 21, 2022
diff --git a/docs/index.md b/docs/index.md
@@ -8,10 +8,11 @@ system goes well beyond what is normally considered monitoring and alerting. Suz
 
 SuzieQ:
 
-* [Gathers data](https://suzieq.readthedocs.io/en/latest/poller/) using an agentless model using either SSH or REST API as the transport. We gather data from routers, bridges and Linux servers. We support gathering data from Arista EOS, Cisco's IOS, IOS-XE, and IOS-XR platforms, Cisco's NXOS, Cumulus Linux, Juniper's Junos(QFX, EX, MX and SRX platforms and Evolved OS), Palo Alto's Panos (version 8.0 or higher) and SoNIC devices, besides Linux servers.
-* Normalizes the data into a vendor-agnostic format.
-* Stores all data in files using the popular big data format, [Parquet](https://parquet.apache.org/).
-* Exposes via a CLI, [GUI](https://suzieq.readthedocs.io/en/latest/gui/), a [REST API](https://suzieq.readthedocs.io/en/latest/rest-server/), or via Python the analysis of the data gathered using easy, intuitive commands. The output can be rendered in various formats from plain text to JSON, CSV and Markdown.
+* We [gather data](https://suzieq.readthedocs.io/en/latest/poller/) using an agentless model using either SSH or REST API as the transport. We gather data from routers, bridges and Linux servers. We support gathering data from Alcatel-Lucent Enterprise AOS8, Arista EOS, Cisco's IOS, IOS-XE, and IOS-XR platforms, Cisco's NXOS (N7K with versions 8.4.4 or higher, and N9K with versions 9.3.1 or higher), Cumulus Linux, Juniper's Junos(QFX, EX, MX and SRX platforms and Evolved OS), Palo Alto's Panos (version 8.0 or higher) and SoNIC devices, besides Linux servers.
+* We normalize the data into a vendor-agnostic format.
+* We store all data in files using the popular big data format, [Parquet](https://parquet.apache.org/).
+* All the analysis are exposed either via a CLI, [GUI](https://suzieq.readthedocs.io/en/latest/gui/), a [REST API](https://suzieq.readthedocs.io/en/latest/rest-server/), or via Python. The output can be rendered in various formats from plain text to JSON and CSV.
+* The analysis engine used in this release is pandas, though we have prototyped with other analysis engines.
 
 With the applications that we build on top of the framework we want to demonstrate a different and more systematic approach to thinking about networks. We want to show how useful it is to think of your network holistically.
 

diff --git a/docs/tables.md b/docs/tables.md
@@ -6,23 +6,26 @@ is the BGP data
 that the bgp service collects from routers. To see what information is collected for each table, you can use the ```<table> describe``` (```bgp describe``` for example) via suzieq-cli to get the details. To see the list of tables, you can type ```help``` in suzieq-cli or run ```suzieq-cli --help```.
 
 
-|         | Cumulus Linux | Arista EOS | Linux | Cisco NXOS | Juniper JunOS<sup>1<sup> | SONIC | IOSXR | IOS | IOSXE | PanOS |
-| :---------: | :---------------: | :------------: | :-------: | :------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
-| Arpnd   |    yes        |      yes   | yes   | yes  |  yes  | yes | yes | yes | yes | yes |
-| BGP     | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
-| Device  | yes | yes | yes | yes | yes |  yes | yes | yes | yes |  yes |
-| EvpnVni | yes | yes | no | yes<sup>2<sup> | yes | yes | no | no | no | no |
-| Filesystem (fs) | yes | no | yes | no | yes | yes | yes | no | no | no |
-| Interfaces | yes | yes | yes| yes | yes | yes | yes | yes | yes | yes |
-| Inventory | no | yes | no | yes | yes | no | no | no | no | no |
-| LLDP | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
-| CDP  | no | no | no | yes | no | no | no | yes | yes | no |
-| Macs |yes | yes | yes | yes | yes | yes | no | yes | yes | no |
-| MLAG | yes | yes | no | yes | no | no | no | no | no | no |
-| Ospf |yes | yes | yes | yes | yes | yes | no | yes | yes | no |
-| Routes | yes | yes | yes | yes | yes<sup>3<sup> | yes | yes | yes | yes | yes |
-| sqPoller | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
-| VLAN | yes | yes | yes | yes | yes | yes | no | yes | yes | no |
+|         | Cumulus Linux | Alcatel AOS8 | Arista EOS | Linux | Cisco NXOS | Juniper JunOS | SONIC | IOSXR | IOS | IOSXE | PanOS |
+| :---------: | :---------: | :---------------: | :------------: | :-------: | :------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
+| Arpnd   |    yes        | yes |      yes   | yes   | yes  |  yes  | yes | yes | yes | yes | yes |
+| BGP     | yes | no | yes | yes | yes | yes | yes | yes | yes | yes | yes |
+| Device  | yes | yes | yes | yes | yes | yes |  yes | yes | yes | yes |  yes |
+| EvpnVni         | no | yes | yes | no | yes* | yes | yes | no | no | no | no |
+| Filesystem (fs) | yes | no | yes | yes | yes | no | yes | no | no | no | no |
+| IfCounters      | yes | no |  yes | yes | no | no | yes | no | no | no | no |
+| Interfaces  | yes | yes | yes | yes| yes | yes | yes | yes | yes | yes | yes |
+| Inventory   | no | yes | yes | no | yes | yes | no | no | no | no | no |
+| LLDP | yes | no | yes | yes | yes | yes | yes | yes | yes | yes | yes |
+| CDP  | no | no | no | no | yes | no | no | no | yes | yes | no |
+| Macs |yes | yes | yes | yes | yes | yes | yes | no | yes | yes | no |
+| MLAG | yes | no | yes | no | yes | no | no | no | no | no | no |
+| Ospf |yes | no | yes | yes | yes | yes | yes | no | yes | yes | no |
+| Routes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
+| sqPoller | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
+| Topcpu | yes | no | yes | yes | yes | no | yes | no | no | no | no |
+| Topmem | yes | no | yes | yes | no | no | yes | no | no | no | no |
+| VLAN | yes | yes | yes | yes | yes | yes | yes | no | yes | yes | no |
 
 1. Junos supported devices includes MX, QFX, QFX10K, EX, SRX, and EVO.
 2. EVPN support for NXOS requires version 9.3.3 or above, please reach out if you're using older versions of NXOS

diff --git a/pytest.ini b/pytest.ini
@@ -123,6 +123,7 @@ markers =
     parsing
 
     # namespaces/nosses
+    aos
     broken
     cumulus
     eos

diff --git a/suzieq/cli/sqcmds/command.py b/suzieq/cli/sqcmds/command.py
@@ -275,7 +275,6 @@ def _pager_print(self, df: pd.DataFrame) -> None:
         else:
             print(df)
 
-    # pylint: disable=too-many-statements
     def _gen_output(self, df: pd.DataFrame, json_orient: str = "records",
                     dont_strip_cols: bool = False, sort: bool = True):
 

diff --git a/suzieq/config/arpnd.yml b/suzieq/config/arpnd.yml
@@ -11,6 +11,11 @@ show-fields:
   - remote
 
 apply:
+  aos:
+    version: all
+    command: show arp
+    textfsm: textfsm_templates/aos_show_arp.tfsm
+
   cumulus:
     version: all
     command: ip neigh show

diff --git a/suzieq/config/device.yml b/suzieq/config/device.yml
@@ -10,6 +10,16 @@ show-fields:
   - address
   - bootupTimestamp
 apply:
+  aos:
+    version: all
+    command:
+      - command: show system
+        textfsm: textfsm_templates/aos_show_system.tfsm
+      #   _entryType: system
+      # - command: show hardware-info
+      #   textfsm: textfsm_templates/aos_show_hardware-info.tfsm
+      #   _entryType: hardware
+
   eos:
     version: all
     command: show version

diff --git a/suzieq/config/interfaces.yml b/suzieq/config/interfaces.yml
@@ -13,6 +13,29 @@ show-fields:
   - ip6AddressList
 
 apply:
+  aos:
+    version: all
+    merge: false
+    command:
+      - command: show interfaces status
+        textfsm: textfsm_templates/aos_show_interfaces_status.tfsm
+        _entryType: interfaces_status
+      - command: show interfaces
+        textfsm: textfsm_templates/aos_show_interfaces.tfsm
+        _entryType: interfaces
+      - command: show vlan members
+        textfsm: textfsm_templates/aos_show_vlan_members.tfsm
+        _entryType: members
+      - command: show ip interface
+        textfsm: textfsm_templates/aos_show_ip_interface.tfsm
+        _entryType: ip_interface
+      - command: show linkagg
+        textfsm: textfsm_templates/aos_show_linkagg.tfsm
+        _entryType: linkagg
+      - command: show linkagg port
+        textfsm: textfsm_templates/aos_show_linkagg_port.tfsm
+        _entryType: linkagg_ports
+
   cumulus:
     version: all
     command: sudo vtysh -c "show int vrf all"; ip -d address show

diff --git a/suzieq/config/inventory.yml b/suzieq/config/inventory.yml
@@ -11,6 +11,17 @@ show-fields:
   - description
 
 apply:
+  aos:
+    version: all
+    merge: False
+    command:
+      - command: show chassis
+        textfsm: textfsm_templates/aos_show_chassis.tfsm
+        _entryType: chassis
+      - command: show system
+        textfsm: textfsm_templates/aos_show_system.tfsm
+        _entryType: system
+
   eos:
     version: all
     command: show inventory

diff --git a/suzieq/config/macs.yml b/suzieq/config/macs.yml
@@ -14,6 +14,12 @@ show-fields:
   - flags
 
 apply:
+  aos:
+    version: all
+    merge: False
+    command: show mac-learning
+    textfsm: textfsm_templates/aos_show_mac_learning.tfsm
+
   cumulus:
     version: all
     command: /sbin/bridge fdb show

diff --git a/suzieq/config/routes.yml b/suzieq/config/routes.yml
@@ -16,6 +16,20 @@ show-fields:
   - action
 
 apply:
+  aos:
+    version: all
+    merge: false
+    command:
+      - command: show arp
+        textfsm: textfsm_templates/aos_show_arp.tfsm
+        _entryType: arp
+      - command: show ip interface
+        textfsm: textfsm_templates/aos_show_ip_interface.tfsm
+        _entryType: ip_interface
+      - command: show ip routes
+        textfsm: textfsm_templates/aos_show_ip_routes.tfsm
+        _entryType: ip_routes
+
   cumulus:
     version: all
     command: ip route show table all

diff --git a/suzieq/config/textfsm_templates/aos_show_arp.tfsm b/suzieq/config/textfsm_templates/aos_show_arp.tfsm
@@ -0,0 +1,17 @@
+Value ip_address ((?:[0-9]{1,3}\.){3}[0-9]{1,3})
+Value hardware_addr (([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]){2})
+Value type (DYNAMIC|STATIC)
+Value flags ([PAVBHIM]*)
+Value port (((\d+\/\d+\/\d+(\w)?)|(\d\/\d+)|(\d+)))
+Value interface (\S*)
+Value name (\S*)
+
+Start
+  ^Total \d+ arp entries$$
+  ^Flags (P=Proxy, A=Authentication, V=VRRP, B=BFD, H=HAVLAN, I=INTF, M=Managed)$$
+  ^IP Addr\s+Hardware Addr\s+Type\s+Flags\s+Port\s+Interface\s+Name$$
+  ^[-+\+]+\s*$$ -> Arp
+
+Arp
+  ^\s+${ip_address}\s+${hardware_addr}\s+${type}\s*${flags}\s*${port}\s*${interface}\s*${name}\s*$$ -> Record
+#  ^. -> Error
diff --git a/suzieq/config/textfsm_templates/aos_show_chassis.tfsm b/suzieq/config/textfsm_templates/aos_show_chassis.tfsm
@@ -0,0 +1,30 @@
+Value Key,Required chassis ([1-8])
+Value model_name (.+)
+Value model_type (.+)
+Value description (.+)
+Value part_number (\d+-\d+)
+Value hardware_revision (\w+)
+Value serial_number (.+)
+Value manufacture_date (\w{3}\s+\d{1,2}\s\d{4})
+Value admin_status (.+)
+Value operational_status ([A-Z]+)
+Value number_of_resets (\d+)
+Value mac_address (([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]){2})
+
+Start
+  ^Local\sChassis\sID\s${chassis}\s\(Master\)$$ -> Chassis
+  ^Remote\sChassis\sID\s${chassis}\s\(Slave\)$$ -> Chassis
+
+Chassis
+  ^\s{2}Model Name:\s+${model_name},$$
+  ^\s{2}Module Type:\s+${model_type},$$
+  ^\s{2}Description:\s+${description},$$
+  ^\s{2}Part Number:\s+${part_number},$$
+  ^\s{2}Hardware Revision:\s+${hardware_revision},$$
+  ^\s{2}Serial Number:\s+${serial_number},$$
+  ^\s{2}Manufacture Date:\s+${manufacture_date},$$
+  ^\s{2}Admin Status:\s+${admin_status},$$
+  ^\s{2}Operational Status:\s+${operational_status},$$
+  ^\s{2}Number Of Resets:\s+${number_of_resets},$$
+  ^\s{2}MAC Address:\s+${mac_address}$$ -> Record Start
+#  ^. -> Error
diff --git a/suzieq/config/textfsm_templates/aos_show_interfaces.tfsm b/suzieq/config/textfsm_templates/aos_show_interfaces.tfsm
@@ -0,0 +1,75 @@
+Value Key,Required chassis_slot_port (\d\/\d+\/\d+[\w]*)
+Value operation_status (up|down|dormant)
+Value port_down_violation_reason (.+\s*)
+Value last_time_link_changed (.+\s*)
+Value number_of_status_change (\d+)
+Value type (Ethernet|Fast Ethernet|Gigabit Ethernet)
+Value sfp_xfp (.+\s*)
+Value interface_type (Copper|Fiber|Combo-copper|Combo-Fiber)
+Value epp (Enabled|Disabled)
+Value link_quality (GOOD|FAIR|POOR|N/A)
+Value mac_address (([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]){2})
+Value bandwidth (\d+|-)
+Value duplex (Half|Full|Auto|-)
+Value autonegotiation (.+\s*)
+Value long_frame_size (\d+)
+Value inter_frame_gap (\d+)
+Value loopback_mode (N/A|SPB-VPN)
+Value bytes_received (\d+)
+Value rx_unicast_frames (\d+)
+Value rx_broadcast_frames (\d+)
+Value rx_mcast_frames (\d+)
+Value rx_undersize_frames (\d+)
+Value rx_oversize_frames (\d+)
+Value rx_lost_frames (\d+)
+Value rx_error_frames (\d+)
+Value rx_crc_error_frames (\d+)
+Value rx_alignments_err (\d+)
+Value bytes_xmitted (\d+)
+Value tx_unicast_frames (\d+)
+Value tx_broadcast_frames (\d+)
+Value tx_mcast_frames (\d+)
+Value tx_undersize_frames (\d+)
+Value tx_oversize_frames (\d+)
+Value tx_lost_frames (\d+)
+Value tx_collided_frames (\d+)
+Value tx_error_frames (\d+)
+Value collisions (\d+)
+Value late_collisions (\d+)
+Value exc_collisions (\d+)
+
+Start
+  ^Chassis\/Slot\/Port\s+:\s${chassis_slot_port}\s+$$
+  ^\sOperational Status\s+:\s${operation_status},$$
+  ^\sPort-Down\/Violation Reason:\s${port_down_violation_reason},$$
+  ^\sLast Time Link Changed\s+:\s${last_time_link_changed},$$
+  ^\sNumber of Status Change\s+:\s${number_of_status_change},$$
+  ^\sType\s+:\s${type},$$
+  ^\sSFP/XFP\s+:\s${sfp_xfp},$$
+  ^\sInterface Type\s+:\s${interface_type},$$
+  ^\sEPP\s+:\s${epp},$$
+  ^\sLink-Quality\s+:\s${link_quality},$$
+  ^\sMAC address\s+:\s${mac_address},$$
+  ^\sBandWidth \(Megabits\)\s+:\s+${bandwidth}\s*,\s+Duplex\s+:\s${duplex}\s*,$$
+  ^\sAutonegotiation\s+:\s+${autonegotiation},$$
+  ^\sLong Frame Size\(Bytes\)\s+:\s${long_frame_size},$$
+  ^\sInter Frame Gap\(Bytes\)\s+:\s${inter_frame_gap},$$
+  ^\sloopback mode\s+:\s${loopback_mode},$$ -> Rx
+
+Rx
+  ^\sRx\s+:$$
+  ^\sBytes Received\s*:\s+${bytes_received}, Unicast Frames\s*:\s+${rx_unicast_frames},$$
+  ^\sBroadcast Frames\s*:\s+${rx_broadcast_frames}, M-cast Frames\s*:\s+${rx_mcast_frames},$$
+  ^\sUnderSize Frames\s*:\s+${rx_undersize_frames}, OverSize Frames\s*:\s+${rx_oversize_frames},$$
+  ^\sLost Frames\s*:\s+${rx_lost_frames}, Error Frames\s*:\s+${rx_error_frames},$$
+  ^\sCRC Error Frames\s*:\s+${rx_crc_error_frames}, Alignments Err\s*:\s+${rx_alignments_err},$$ -> Tx
+
+Tx
+  ^\sTx\s+:$$
+  ^\sBytes Xmitted\s*:\s+${bytes_xmitted},\s*Unicast Frames\s*:\s+${tx_unicast_frames},$$
+  ^\sBroadcast Frames\s*:\s+${tx_broadcast_frames}, M-cast Frames\s*:\s+${tx_mcast_frames},$$
+  ^\sUnderSize Frames\s*:\s+${tx_undersize_frames}, OverSize Frames\s*:\s+${tx_oversize_frames},$$
+  ^\sLost Frames\s*:\s+${tx_lost_frames},\s*Collided Frames\s*:\s+${tx_collided_frames},$$
+  ^\sError Frames\s*:\s+${tx_error_frames},\s*Collisions\s*:\s+${collisions},$$
+  ^\sLate collisions\s*:\s+${late_collisions},\s*Exc-Collisions\s*:\s+${exc_collisions}$$ -> Record Start
+#  ^. -> Error
diff --git a/suzieq/config/textfsm_templates/aos_show_interfaces_status.tfsm b/suzieq/config/textfsm_templates/aos_show_interfaces_status.tfsm
@@ -0,0 +1,23 @@
+Value Key,Required chassis_slot_port (\d\/\d+\/\d+[\w]*)
+Value admin_status (-|en|dis)
+Value autonego (-|en|dis)
+Value detected_speed (-|Auto|\d{0,})
+Value detected_duplex (-|Half|Full|Auto)
+Value detected_pause (-|Rx-N-Tx|Rx|Tx)
+Value detected_fec (-|FC|RS|DIS)
+Value configured_speed (-|Auto|\d{0,})
+Value configured_duplex (-|Half|Full|Auto)
+Value configured_pause (-|Rx-N-Tx|Rx|Tx)
+Value configured_fec (AUTO|DIS|FC|RS)
+Value trap_linkupdown (-|en|dis)
+Value eee (en[a]*|dis)
+
+Start
+  ^\s*Chas/\s+DETECTED-VALUES\s+CONFIGURED-VALUES\s*$$
+  ^\s*Slot/\s+Admin\s+Auto\s+Speed\s+Duplex\s+Pause\s+FEC\s+Speed\s+Duplex\s+Pause\s+FEC\s+Link\s*$$
+  ^\s*Port\s+Status\s+Nego\s+(Mbps)\s+Det\s+(Mbps)\s+Cfg\s+Trap\s+EEE\s*$$
+  ^[-+\+]+\s*$$ -> Interfaces
+
+Interfaces
+  ^\s*${chassis_slot_port}\s+${admin_status}\s+${autonego}\s+${detected_speed}\s+${detected_duplex}\s+${detected_pause}\s+${detected_fec}\s+${configured_speed}\s+${configured_duplex}\s+${configured_pause}\s+${configured_fec}\s+${trap_linkupdown}\s+${eee}\s*$$ -> Record
+#  ^. -> Error
diff --git a/suzieq/config/textfsm_templates/aos_show_ip_interface.tfsm b/suzieq/config/textfsm_templates/aos_show_ip_interface.tfsm
@@ -0,0 +1,17 @@
+Value name (\S+(\s\S+)*)
+Value ip_address ((?:[0-9]{1,3}\.){3}[0-9]{1,3})
+Value subnet_mask ((?:[0-9]{1,3}\.){3}[0-9]{1,3})
+Value status (UP|DOWN)
+Value forward (YES|NO)
+Value device (Loopback|EMP|unbound|(vlan\s\d+)|(service\s\d+)|GRE\stunnel|IPIP\stunnel)
+Value flags (D|)
+
+Start
+  ^Total (\d+) interfaces$$
+  ^ Flags (D=Directly-bound)$$
+  ^            Name                 IP Address      Subnet Mask     Status Forward  Device   Flags$$
+  ^[-+\+]+\s*$$ -> Interfaces
+
+Interfaces
+  ^${name}\s+${ip_address}\s+${subnet_mask}\s+${status}\s+${forward}\s+${device}\s*${flags}$$ -> Record
+#  ^. -> Error
diff --git a/suzieq/config/textfsm_templates/aos_show_ip_routes.tfsm b/suzieq/config/textfsm_templates/aos_show_ip_routes.tfsm
@@ -0,0 +1,15 @@
+Value Filldown dest_addr ((?:[0-9]{1,3}\.){3}[0-9]{1,3}\/[0-9]{1,2})
+Value Required gateway_addr ((?:[0-9]{1,3}\.){3}[0-9]{1,3})
+Value age ((\d+:\d+:\d+)|(\d+d\s*\d+h))
+Value protocol (\w+)
+
+Start
+  ^\+ = Equal cost multipath routes$$
+  ^ Total (\d) routes$$
+  ^  Dest Address       Gateway Addr        Age        Protocol $$
+  ^[-+\+]+\s*$$ -> Routes
+
+Routes
+  ^\s+${dest_addr}\s+(\+)*${gateway_addr}\s+${age}\s+${protocol}\s*$$ -> Record
+  ^\s+(\+)*${gateway_addr}\s+${age}\s+${protocol}\s*$$ -> Record
+#  ^. -> Error
diff --git a/suzieq/config/textfsm_templates/aos_show_linkagg.tfsm b/suzieq/config/textfsm_templates/aos_show_linkagg.tfsm
@@ -0,0 +1,16 @@
+Value Key,Required number (\d{1,3})
+Value aggregate (Static|Dynamic|Loopback)
+Value snmp_id (\d+)
+Value size (\d+)
+Value admin_state (ENABLED|DISABLED)
+Value oper_state (UP|DOWN)
+Value att_ports (\d+)
+Value sel_ports (\d+)
+
+Start
+  ^Number\s+Aggregate\s+SNMP Id\s+Size\s+Admin State\s+Oper State\s+Att\/Sel Ports$$
+  ^[-+\+]+\s*$$ -> Linkagg
+
+Linkagg
+  ^\s+${number}\s+${aggregate}\s+${snmp_id}\s+${size}\s+${admin_state}\s+${oper_state}\s+${att_ports}\s+${sel_ports}$$ -> Record
+#  ^. -> Error
-Original file line number
+Diff line change
@@ Expand Up / @@ -123,6 +123,7 @@ markers = @@
         parsing
         # namespaces/nosses
+        aos
         broken
         cumulus
         eos
@@ Expand Down @@