begining docs with some structure

docs/api/sqobjects.md

@@ -0,0 +1,73 @@
+::: suzieq.sqobjects.basicobj.SqObject
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.basicobj.SqContext
+    :docstring:
+
+::: suzieq.sqobjects.address.AddressObj
+    :docstring:
+
+::: suzieq.sqobjects.arpnd.ArpndObj
+    :docstring:
+
+::: suzieq.sqobjects.bgp.BgpObj
+    :docstring:
+
+::: suzieq.sqobjects.device.DeviceObj
+    :docstring:
+
+::: suzieq.sqobjects.evpnVni.EvpnvniObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.interfaces.IfObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.lldp.LldpObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.macs.MacsObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.mlag.MlagObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.ospf.OspfObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.path.PathObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.routes.RoutesObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.sqPoller.SqPollerObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.tables.TablesObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.topcpu.Topcpu
+Obj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.topmem.TopmemObj
+    :docstring:
+    :members:
+
+::: suzieq.sqobjects.vlan.VlanObj
+    :docstring:
+    :members:
+
+

docs/index.md

@@ -0,0 +1,58 @@
+[![Build Status](https://travis-ci.org/netenglabs/suzieq.svg?branch=master)](https://travis-ci.org/netenglabs/suzieq)
+
+# Suzieq
+
+Would you like to be able to easily answer trivial questions such as how many unique prefixes are there in your routing table, or how many MAC addresses are there in the MAC tables across the network? How about more difficult questions, such as what changes did your routing table see between 10 pm and midnight last night, or which of your nodes have been up the longest, or which BGP sessions have had the most routing updates? How about being able to answer if your OSPF (or BGP) sessions are working correctly, or is all well with your EVPN? How about a quick way to determine the amount of ECMP at every hop between two endpoints? Do you wish you could easily validate the configuration you deployed across your network? 
+
+Do you login to every network node you have to figure out answers to a questions like these? Do you then struggle to piece the information together into a consistent whole across the various formats provided by various vendors? Do you wish you had an open source, multi-vendor tool that could help you answer questions like these and more?
+
+If you answered yes to one or more of these questions, then Suzieq is a tool that we think will be interesting to you. 
+
+**Suzieq** is both a framework and an application using that framework, that is focused on **improving the observability of your network**.  We define observaility as the ability of a system to answer either trivial or complex questions that you pose as you go about operating your network. How easily you can answer your questions is a measure of how good the system's observability is. A good observable system goes well beyond monitoring and alerting. Suzieq is primarily meant for use by network engineers and designers.
+
+Suzieq does multiple things. It collects data from different devices and systems. It normalizes the data and then stores it in a vendor independent way. Then it allows analysis of that data. 
+
+We believe Suzieq is novel because it is a disaggregated framework that allows you to independently pick:
+* how you gather your data (agentless or agent-based)
+* how you store your data
+* how you interact with the data i.e. how you ask the questions and how you see the answers.
+
+With the applications that we build on top of the framework we want to demonstrate a different and more systematic approach to thinking about networks. We want to show how useful it is to think of your network holistically.
+
+In this very early release of Suzieq, we've chosen some answers for the framework to get the ball rolling. 
+* We gather data using an agentless model using either SSH or REST API as the transport. 
+* We normalize the data into a vendor-agnostic format.
+* We store all data in files using the popular big data format, Parquet. 
+* All the analysis are exposed either via a CLI or via Python objects. The output can be rendered in various formats from plain text to JSON and CSV.
+* The analysis engine used in this release is pandas.
+
+**We support gathering data from Cumulus routers, Arista routers, and Linux servers.**
+
+We are just getting started with Suzieq. As befitting an early release, what you see is only a brief demonstration of what this approach can bring about. We've many, many ideas to implement in our upcoming releases, but we wanted to get this out so that people can start using it. And start understanding their networks to solve problems, validate or to make changes.
+
+You can join the conversation via [googlegroups](https://groups.google.com/d/forum/netenglabs-suzieq).
+
+We're also looking for collaborators to help us make Suzieq a truly useful multi-vendor, open source platform for observing all aspects of networking. Please read the [collaboration document](./CONTRIBUTING.md) for ideas on how you can help. 
+
+## Quick Start
+
+We want to make it as easy as possible for you to start engaging with Suzieq. To that end, the quickest way to start is to download the docker image from github and also download the data we've already gathered for the 18 or so network scenarios from the [github](https://github.com/netenglabs/suzieq-data) repository associated with Dinesh's book, Cloud Native Data Center Networking. You can then use the introductory documentation to start exploring the data.
+
+- `docker run -it --name suzieq ddutt/suzieq-demo`
+- `suzieq-cli`
+
+When you're within the suzieq-cli, you can run ```device unique columns=namespace``` to see the list of different scenarios, we've gathered data for.
+
+If you're looking for more than just a demo, and would like to explore even more data that we've gathered, from all all scenarios listed in Dinesh's book, Cloud Native Data Center Networking, then use the following sequence of commands.
+
+- ```git clone https://github.com/netenglabs/suzieq-data.git```
+- ```docker run -itd -v /home/ddutt/suzieq-data/cloud-native-data-center-networking/parquet-out:/suzieq/parquet --name suzieq ddutt/suzieq:latest```
+- ```docker attach suzieq```
+- ```suzieq-cli```
+
+Note that in the docker run command above, the directory name /home/ddutt/suzieq-data/... used assumed that the git clone of suzieq-data was done in the directory /home/ddutt. In other words, the host path name used in the -v option should be the **absolute path of the directory**, not the relative path.
+
+From here on, you can follow the [preliminary documentation](./docs/analyzer.md) for further exploration. If you wish to **run the poller to gather data from your network**, you can do so via the instructions [here](./docs/poller.md). 
+
+We're working on putting up more extensive documentation for every aspect of Suzieq: as a user, and as a contributor. We hope to be done with that by the end of this week.
+

docs/requirements.txt

@@ -0,0 +1 @@
+mkauto==0.1.0

docs/tables.md

@@ -0,0 +1,24 @@
+## Tables
+Tables are the most basic structure we store in the suzieq database.
+A table is the data that is gathered by a service, for intance BGP is the BGP data
+that the bgp service collects from routers. 
+
+
+|         | Cumulus Linux | Arista EOS | Linux |   |
+|---------|---------------|------------|-------|---|
+| Arpnd   |    yes        |            |       |   |
+| BGP     |               |            |       |   |
+| Device  | yes | yes | yes |
+| EvpnVni         |
+| Filesystem (fs) |
+| IfCounters      |
+| Interfaces  | yes | yes | yes|
+| LLDP | 
+| Macs |
+| Ospf If |
+| Ospf Nbr |
+| Routes |
+| sqPoller |
+| Topcpu | 
+| Topmem |
+| VLAN | 

docs/tutorials/10things.md

@@ -0,0 +1,176 @@
+---
+layout: post
+comments: true
+author: Justin Pietsch
+title: 10ish ways to explore your network with Suzieq
+excerpt: Suzieq is new software for Network Observability. This is an exploration of using Suzieq to understand your network.
+description: 
+---
+
+# 10ish ways to explore your network with Suzieq
+
+[Suzieq](https://github.com/netenglabs/suzieq) is new software for Network Observability. In this blog I will be going over some of the things you can do with Suzieq to help you explore and understand your network. We have [suzieq-data ](https://github.com/netenglabs/suzieq-data)that you can use to investigate Suzieq. This databases has data from all the different scenarios available in [https://github.com/ddutt/cloud-native-data-center-networking](https://github.com/ddutt/cloud-native-data-center-networking)
+
+We are going to go through specific things you can do with Suzieq, to give you good ideas of how you can start using it in your network. We will cover more topics in other blog posts, such as [asserts](https://github.com/netenglabs/suzieq/blob/master/docs/analyzer.md#asserts) which are a powerful way to have Suzieq check that you have a healthy network.
+
+All my examples in this post are [Cumulus Linux](https://cumulusnetworks.com/) because it is by far the easiest Network OS to run virtually to try these experiments out. Suzieq analysis does not care what OS it’s working on.
+
+## Namespaces and Devices
+
+**Namespaces are the way to group devices in Suzieq.** These groups allow analysis of devices together. Namespaces also allow overlapping device names between namespaces. When you configure the sq-poller to collect information from your devices you decide which devices are in which namespace. To see the namespaces you have in a Suzieq database:
+
+```
+device unique columns=namespace
+```
+
+![Suzieq device unique namespace](/assets/images/2020-04-30/suzieq-device-unique-namespace.png)
+
+This also demonstrates the way that the suzieq-cli works. We have different objects (also called commands) just as device, then we have verbs for the objects (such as unique) and then filters for the object/verb pairing. **Most of the commands have a standard set of verbs, which are ‘show’, ‘summarize’, and ‘unique’**. ‘unique’ requires the ‘columns=’ filter to find the unique entries in a column and the count of the number of times that the entry occurs.
+
+To see all the commands, type:
+
+    ?
+
+![](/assets/images/2020-04-30/suzieq-help.png)
+
+Let’s continue by looking at the data we have in the database. To see hosts in a namespace
+
+    device unique columns=hostname namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-device-unique-hostname-namespace.png)
+
+You can also look to see more information about the devices that you have.
+
+    device show namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-device-show-namespace.png)
+
+**The show verb is the most basic verb Suzieq has, showing the data directly**. It displays a lot of information for the command. For each command Suzieq has a default set of columns that are displayed, but you can choose to display less or display more, using the ‘columns=’ filter. ‘columns=default’ gives this default view, ‘columns=*’ displays all the columns and ‘columns=’col1 col2' would display the col1 and col2 columns.
+
+You can see how many columns and what are the defaults using the ‘table describe’ command.
+
+    table describe table=device
+
+![](/assets/images/2020-04-30/suzieq-table-describe-device.png)
+
+You can see which of the columns are shown by default and which additional columns are stored. In the case of device, almost all the columns are shown. In some cases, there are many columns not shown. BGP currently has 50 columns that could be shown.
+
+## BGP
+
+Now that we have a little bit of understanding of what data we have, let’s dive into one namespace. Let’s start with an overview of BGP.
+
+    bgp summarize namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-bgp-summarize-namespace.png)
+
+In any of our summaries, statistics that end with Cnt, such as uniqueVrfCnt, if there are three or less of them, then we break out the items and provide their count. In this example, there are 2 uniqueVrfsCnts, which are default and internet-vrf. Statistics that end with Stat, such as upTimesStat, shows lists of three items bracketed in [] show [min, max, median] of the values to give you an overview of the distribution. For any of these values, you can use the ‘unique’ verb to dive into the column and see the whole list.
+
+For instance, notice that there are 2 session that have failed, let’s see what those are.
+
+    bgp show namespace=single-attach_bgp_numbered status=fail
+
+![](/assets/images/2020-04-30/suzieq-bgp-show-namespace-status-fail.png)
+
+Let’s see what else we can find out about these two failed sessions. FRR (which is used by Cumulus Linux) and Arista EOS both sometimes support reason codes when they have failed sessions. Let’s use the column filter to see if we can find the reason.
+
+    bgp show namespace=single-attach_bgp_numbered columns='namespace hostname vrf peer state reason notifcnReason lastDowTime'
+
+![](/assets/images/2020-04-30/suzieq-bgp-show-namespace-columns-status-fail.png)
+
+No, that didn’t work this time. However, it is interesting that the lastDownTime is 0. That means it’s never established correctly. Ok, let’s try bgp assert. In other blog posts we’ll cover assert more thoroughly, but to understand this problem let’s try it.
+
+    bgp assert namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-bgp-assert-namespace.png)
+
+So that’s a lot of output. We’d like to filter on just those that failed, Suzieq doesn’t have an option for that yet, so I just filed an [issue](https://github.com/netenglabs/suzieq/issues/118).
+
+Here’s a screen shot of just those two (plus an extra).
+
+![](/assets/images/2020-04-30/suzieq-bgp-assert-namespace-narrow.png)
+
+We can see that the first one failed because of no route to peer. The second doesn’t give us an assert failure, however you’ll notice that the first failure is exit02 to 169.54.127.1 and the second one is the internet device trying to connect to exit02. So this is probably each direction of the same failed BGP connection.
+
+We also want to understand what the ASNs are. Dive into the ASNs and Peer ASNs
+
+    bgp unique columns=asn namespace=single-attach_bgp_numbered
+    bgp unique columns=peerAsn namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-bgp-unique-asn-namespace.png)
+
+One mysterious thing is what is peerAsn 0? From above, that is one of the failed BGP sessions. You also might notice the counts columns. These are the counts of BGP sessions for each of the ASNs.
+
+## Routes
+
+Let’s now take a look at the route command. We’ll start with summarize.
+
+    route summarize namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-route-summarize-namespace.png)
+
+This has data like we saw in ‘bgp summarize’, and we want to dive into what these mean. This shows the routes for each device summed together. Let’s use unique to dive in. What are the Vrfs in your network?
+
+    route unique columns=vrf namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-route-unique-vrf-namespace.png)
+
+This shows a bug we have in Suzieq right now, where evpn-vrf is not in this namespace but it is showing up with a count of 0. That count is correct, we shouldn’t even be displaying evpn-vrf. We’ll fix it soon, but this is where we are right now.
+
+Another thing you might like to look at is which devices have no default route, since the summarize showed hostsWithNoDefRoute = 2. Suzieq doesn’t provide a way to do that at this time.
+
+Let’s move to looking at protocols, as you notice, we have four protocols.
+
+    route unique columns=protocols namespace=single-attach_bgp_numbered
+
+![](/assets/images/2020-04-30/suzieq-route-unique-protocol-namespace.png)
+
+These might look mysterious to you. These are consequences of using Cumulus Linux as well as Ubuntu 16.04. Blank protocols are usually DHCP or mgmt interfaces. Kernel routes are connected interfaces or static routes.
+
+I’ll just look up protocol 186. You can look up the others.
+
+    route show namespace=single-attach_bgp_numbered protocol=186
+
+![](/assets/images/2020-04-30/suzieq-route-show-namespace-protocol-186.png)
+
+I can log into a Cumulus Linux device and look up 186, and it’s BGP. I think the this version of Ubuntu doesn’t read that and report the protocol as BGP.
+
+    vagrant@edge01:~$ more /etc/iproute2/rt_protos.d/frr.conf 
+    # Additional protocol strings defined by frr for each of its daemons  
+    186  bgp 
+    187  isis 
+    188  ospf 
+    189  rip 
+    190  ripng 
+    191  nhrp 
+    192  eigrp 
+    193  ldp 
+    194  sharp 
+    195  pbr 
+    196  static 
+    197  openfabric
+
+With so many routes, it’s good to be able to dive in. One way is to see what routes are available using filtering by prefixlen.
+
+    route show namespace=single-attach_bgp_numbered prefixlen="<=16"
+
+![](/assets/images/2020-04-30/suzieq-route-show-namespace-prefixlen.png)
+
+We’d also like to see where an address comes from look. Route show, shows all the routers that this is valid on. Address show shows the origin.
+
+    route show namespace=single-attach_bgp_numbered prefix='172.16.253.1' 
+    address show namespace=single-attach_bgp_numbered prefix='172.16.253.1'
+
+![](/assets/images/2020-04-30/suzieq-route-show-address-show-namespace.png)
+
+I’d also like to see all the addresses in a subnet. H ow do I see all the address in 172.16.4.0/24?
+
+    route show namespace=single-attach_bgp_numbered prefix='172.16.4.0/24' 
+    address show namespace=single-attach_bgp_numbered address='172.16.4.0'
+
+![](/assets/images/2020-04-30/suzieq-route-show-address-show-namespace-24.png)
+
+Nope. Suzieq can’t do that yet. I’ve created [an issue](https://github.com/netenglabs/suzieq/issues/117) for that, let us know in comments on that issue if you also want that added sooner than later.
+
+## Conclusion
+
+Hopefully this has shown you how to use Suzieq to explore your network. This focused mostly on using summarize and unique to get an overview and then to dive in. It also demonstrated some other filtering that can be done for you to understand your network. I didn’t cover all the things you can do with Suzieq, for instance we didn’t explore any of the OSPF or EVPN commands, and I didn’t cover asserts. We’ll be covering more of that in upcoming blog posts.