New Release v0.7.4: Merge branch 'dev' of github.com:yantisj/netgrph

README.md

@@ -374,7 +374,7 @@ fairly easily. It relies on both the py2neo and neo4j bolt driver (install via
 pip3). The plan is to eventually convert all driver code to the bolt driver as
 it matures.
 
-NetGrph requires [CSV files](csv/) with all of your Switches/Routers, Networks,
+NetGrph requires [CSV files](test/csv/) with all of your Switches/Routers, Networks,
 VLANs, and CDP/LLDP Neighbors in order to be multi-vendor compatible. I provide
 IOS and NXOS configuration parsers, as well as a sample network topology to play
 with. For Cisco-based networks, a generic CDP/LLDP mapper is all that's missing.

docs/INSTALL.md

@@ -1,11 +1,17 @@
 ## Install Notes
 
 Netgrph was built on Ubuntu 14.04 LTS but should be portable to any other Python
-3.4+ system. I have done limited testing at this point on other systems (MacOS),
-so I highly recommend this OS right now. In the near future, I plan to ship a
-set of ansible scripts for easy install, but at first this is what I have.
+3.4+ system. I have done testing on Ubuntu 16.04 and MacOS, and I highly
+recommend Ubuntu trusty or xenial for support purposes. I provide ansible
+scripts for an easy install on a base Ubuntu system, and will be creating a
+Docker file before long.
 
-### Test Install Instructions
+### Ansible Install
+- See the README.md in docs/playbooks/
+- Install ansible on your machine, and point the playbook towards localhost as
+  documented, or a remote host if you are familiar with Ansible.
+
+### Manual Install Instructions
 
 - For testing, you can install everything but the database under your user on any system
 - If you do not have root access to your system, use virtualenv to satisfy the pip requirements
@@ -56,17 +62,21 @@ sudo pip3 install -r requirements.txt
 ./netgrph.py -nf all
 ./netgrph.py -nf all -o tree
 ./ngreport.py -dev ".*"
+./ngreport.py -vrf "perim|default"
 ./netgrph.py abc4mdf
 ./netgrph.py abc4mdf -o json
 ./netgrph.py abc4mdf -o yaml
 ./netgrph.py -sp abc2sw1 xyz2sw1
 ./netgrph.py -sp abc.* xyz.* -o csv
+./netgrph.py -rp 10.1.120.50 10.7.206.0/23
 ./netgrph.py 120
 ./netgrph.py 1246
 ./netgrph.py -fp 10.1.120.50 8.8.8.8
 ./netgrph.py -nlist test_group
 ./netgrph.py -nlist test_group -o tree
 ./netgrph.py -group ABC
+./ngreport.py -vrf "perim|default"
+./ngreport.py -vlans
 ```
 
 ## Sample Reporting (to be expanded)
@@ -98,6 +108,6 @@ ngupdate --clearNodes --hours 12
 ```
 
 ## Adding firewalls and third-party devices
-- Examine the csv files in csv/ to understand the required datasources for importing third-party data
+- Examine the csv files in test/csv/ to understand the required datasources for importing third-party data
 - Examine the cyp/buildfw.cyp for understanding how to insert a L2 firewall between VRFs
 - Examine the cyp/sample-queries.cyp to start querying the Neo4j database directly for data

docs/TODO

@@ -18,4 +18,7 @@ Maintenance:
 - Add switch skip on certain neighbors
 - Rewrite nglib.net_update using Bolt as Pythonic
 - Emulate dict with ngtree
+- Rewrite py2neo cypher queries with py2neo v3.0
+- Change py2neo to use bolt driver by default
+
 

docs/VERSION

@@ -1 +1 @@
-v0.7.3
+v0.7.4

docs/WORKFLOW.md

@@ -0,0 +1,9 @@
+# All development done on the dev branch
+
+# For release to master, checkout master, advance version, then pull from dev with no fast forward
+```
+edit docs/VERSION
+git tag v0.7.x
+git merge --no-ff --log origin/dev
+git push origin master
+```

docs/netgrph.ini

@@ -42,14 +42,14 @@ dist_exclude = (noexclusion)
 
 
 [ngfiles]
-vrfs =  ./csv/vrfs.csv
-devices = ./csv/devices.csv
-device_info = ./csv/devinfo.csv
-neighbors = ./csv/nd.csv
-networks = ./csv/allnets.csv
-vlans = ./csv/allvlans.csv
-supernets = ./csv/supernets.csv
-firewalls = ./csv/firewalls.csv
+vrfs =  ./test/csv/vrfs.csv
+devices = ./test/csv/devices.csv
+device_info = ./test/csv/devinfo.csv
+neighbors = ./test/csv/nd.csv
+networks = ./test/csv/allnets.csv
+vlans = ./test/csv/allvlans.csv
+supernets = ./test/csv/supernets.csv
+firewalls = ./test/csv/firewalls.csv
 
 # ASA FW Directory
 [ngfw]

docs/playbooks/README.md

@@ -0,0 +1,44 @@
+## Ansible NetGrph Playbooks
+
+These playbooks will install NetGrph via Ansible on Ubuntu 14.04 or 16.04 for
+you. It will not set your database password or configure your netgrph.ini file
+for now, so you need to do that manually. Once the scripts run, browse to
+http://machine:7474 and setup your password. Set that same password in
+/home/netgrph/docs/netgrph.ini, and run the /home/netgrph/test/first_import.sh.
+
+#### Setting up Ansible to run via localhost
+
+```
+sudo su -
+apt-get install ansible
+echo '[netgrph]' >> /etc/ansible/hosts
+echo localhost ansible_connection=local >> /etc/ansible/hosts
+exit
+```
+
+#### Run playbooks against localhost (installs under netgrph user)
+
+```
+git clone https://github.com/yantisj/netgrph.git /tmp/netgrph/
+cd /tmp/netgrph/docs/playbooks/
+ansible-playbook netgrph.yml --ask-sudo-pass
+```
+
+#### Test the install
+
+```
+sudo su - netgrph
+cd netgrph
+```
+
+##### Use an insecure DB password for testing (not recommended)
+```
+./test/set_neo4j_password.sh
+```
+
+##### Test a database import
+```
+./test/first_import.sh
+```
+
+- See the INSTALL.md file for test queries and production install information

docs/playbooks/netgrph.yml

@@ -0,0 +1,42 @@
+# Configure your nguser account under vars: below
+# Add your server to [netgrph] in Ansible Hosts
+# Add your sudo user to [netgrp:vars]
+#   ansible_ssh_user = yantisj
+
+- hosts: netgrph
+  become: yes
+  become_user: root
+  vars:
+    nguser: netgrph
+    option_install_java: false
+    option_allow_remote_connections: true
+    neo4j_edition: neo4j
+
+    neo4j_properties:
+      - {regexp: "^allow_store_upgrade.*", line: "allow_store_upgrade=true"}
+
+  tasks:
+  - name: Update Apt Packages
+    include: tasks/apt-update.yml
+
+  - name: Apt Dependencies
+    include: tasks/apt-packages.yml
+
+  - name: Install Java8
+    include: tasks/java8.yml
+
+  - name: Install Neo4j
+    include: tasks/neo4j.yml
+
+  - name: Add netgrph user
+    include: tasks/netgrph-user.yml
+
+  - name: Clone NetGrph Repo
+    include: tasks/clonerepo.yml
+
+  - name: Install PIP Requirements
+    include: tasks/ngpip.yml
+
+  - name: NetGrph Setup
+    include: tasks/setup.yml
+

docs/playbooks/netgrph/netgrph.yml

@@ -0,0 +1,38 @@
+# Configure your nguser account under vars: below
+# Add your server to [netgrph] in Ansible Hosts
+# Add your sudo user to [netgrp:vars]
+#   ansible_ssh_user = yantisj
+
+- hosts: netgrph
+  become: yes
+  become_user: root
+  vars:
+    nguser: yantisj
+    option_install_java: false
+    option_allow_remote_connections: true
+    neo4j_edition: neo4j
+
+    neo4j_properties:
+      - {regexp: "^allow_store_upgrade.*", line: "allow_store_upgrade=true"}
+
+  tasks:
+  - name: Update Apt Packages
+    include: tasks/apt-update.yml
+
+  - name: Apt Dependencies
+    include: tasks/apt-packages.yml
+
+  - name: Install Java8
+    include: tasks/java8.yml
+
+  - name: Install Neo4j
+    include: tasks/neo4j.yml
+
+  - name: Clone NetGrph Repo
+    become: yes
+    become_user: "{{nguser}}"
+    include: tasks/clonerepo.yml
+
+  - name: Install PIP Requirements
+    include: tasks/ngpip.yml
+

docs/playbooks/netgrph/tasks/apt-packages.yml

@@ -0,0 +1,8 @@
+# file: update.yml
+  - name: Install Apt Packages
+    apt: name={{item}} state=present
+    with_items:
+    - python3-pip
+    - python3-pytest 
+    - python3-logilab-common
+    - emacs24-nox

docs/playbooks/netgrph/tasks/apt-update.yml

@@ -0,0 +1,3 @@
+---
+  - name: Update Packages
+    apt: update_cache=yes upgrade=dist dpkg_options='force-confold,force-confdef'

docs/playbooks/netgrph/tasks/clonerepo.yml

@@ -0,0 +1,5 @@
+---
+    - name: Clone NetGrph Repo
+      git: repo=https://github.com/yantisj/netgrph dest=/home/{{nguser}}/netgrph
+
+

docs/playbooks/netgrph/tasks/java8.yml

@@ -0,0 +1,15 @@
+---
+    - name: add repo for java 8
+      apt_repository: repo='ppa:webupd8team/java' state=present
+
+    - name: set licence selected
+      shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
+      sudo: yes
+
+    - name: set licence seen
+      shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections
+      sudo: yes
+
+    - name: install java 8
+      apt: name=oracle-java8-installer state=latest update-cache=yes force=yes
+      sudo: yes

docs/playbooks/netgrph/tasks/neo4j.yml

@@ -0,0 +1,59 @@
+# According to http://debian.neo4j.org/
+---
+  - name: NEO4J | Add key for Neo4j repo
+    apt_key: url=http://debian.neo4j.org/neotechnology.gpg.key state=present
+    become: yes
+
+  - name: NEO4J | Add Neo4j repo to sources list
+    apt_repository: repo='deb http://debian.neo4j.org/repo stable/' state=present
+    become: yes
+
+  - name: NEO4J | Install Neo4j packages
+    apt: pkg={{ item }} state=installed update_cache=yes force=yes
+    become: yes
+    with_items:
+      - "{{neo4j_edition}}"
+
+# http://www.delimited.io/blog/2014/1/15/getting-started-with-neo4j-on-ubuntu-server
+
+  - name: NEO4J | Update /etc/security/limits.conf file (1/2)
+    lineinfile: dest=/etc/security/limits.conf
+              insertbefore='# End of file'
+              line='neo4j             soft    nofile          40000'
+              state=present
+    become: yes
+
+  - name: NEO4J | Update /etc/security/limits.conf file (2/2)
+    lineinfile: dest=/etc/security/limits.conf
+              insertbefore='# End of file'
+              line='neo4j             hard    nofile          40000'
+              state=present
+    become: yes
+
+  - name: NEO4J | Update /etc/pam.d/su file
+    lineinfile: dest=/etc/pam.d/su
+              regexp="^session    required   pam_limits.so"
+              insertafter='^# session    required   pam_limits.so'
+              line="session    required   pam_limits.so"
+              state=present
+    become: yes
+
+#  - name: NEO4J | Update /etc/neo4j/neo4j-server.properties to enable remote users to login to neo4j
+#    lineinfile: dest=/etc/neo4j/neo4j-server.properties
+#              regexp="^org.neo4j.server.webserver.address=0.0.0.0"
+#              insertafter='^#org.neo4j.server.webserver.address=0.0.0.0'
+#              line="org.neo4j.server.webserver.address=0.0.0.0"
+#              state=present
+
+  - name: NEO4J | Update /etc/neo4j/neo4j.conf to enable remote users to login to neo4j
+    lineinfile: dest=/etc/neo4j/neo4j.conf
+              regexp="^dbms.connector.http.address=0.0.0.0:7474"
+              insertafter='^#dbms.connector.http.address=0.0.0.0:7474'
+              line="dbms.connector.http.address=0.0.0.0:7474"
+              state=present
+    when: option_allow_remote_connections
+    become: yes
+
+  - name: NEO4J | Restart Neo4j
+    service: name=neo4j state=restarted
+    become: yes

docs/playbooks/netgrph/tasks/ngpip.yml

@@ -0,0 +1,4 @@
+---
+    - name: Install PIP Requirements via PIP3
+      pip: requirements=/home/{{nguser}}/netgrph/requirements.txt executable=pip3
+

docs/playbooks/tasks/apt-packages.yml

@@ -0,0 +1,8 @@
+# file: update.yml
+  - name: APT | Install Apt Packages
+    apt: name={{item}} state=present
+    with_items:
+    - python3-pip
+    - python3-pytest 
+    - python3-logilab-common
+    - emacs24-nox

docs/playbooks/tasks/apt-update.yml

@@ -0,0 +1,3 @@
+---
+  - name: APT | Update Packages
+    apt: update_cache=yes upgrade=dist dpkg_options='force-confold,force-confdef'

docs/playbooks/tasks/clonerepo.yml

@@ -0,0 +1,6 @@
+---
+    - name: GIT | Clone NetGrph Repo
+      git: repo=https://github.com/yantisj/netgrph dest=/home/{{nguser}}/netgrph
+      become: yes
+      become_user: "{{nguser}}"
+

docs/playbooks/tasks/java8.yml

@@ -0,0 +1,12 @@
+---
+    - name: JAVA | Add repo for java 8
+      apt_repository: repo='ppa:webupd8team/java' mode=664 state=present
+
+    - name: JAVA | Set licence selected
+      shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
+
+    - name: JAVA | Set licence seen
+      shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections
+
+    - name: JAVA | Install java 8
+      apt: name=oracle-java8-installer state=latest update-cache=yes force=yes