Getting started with DAQ
In this tutorial you will download and install DAQ. You will then run the first test with virtual ethernet ports and subsequently move to testing a single device and multiple devices in parallel.
The pre-requisites for this tutorial are a Debian based Linux distribution (Debian GNU/Linux 9.4 Stretch is the recommended one) and the git tool. If you don't have it yet, open a terminal window and install it with the following command:
sudo apt install git
The first step to install DAQ is to download its source code from the DAQ github repository:
git clone https://github.com/faucetsdn/daq.git
Once the software is downloaded, go to the DAQ folder
cd daq
and run the installation with this command:
sudo bin/setup_base
The setup_base command installs a minimum set of basic packages, including docker and Open vSwitch,
sudo bin/setup_dev
The setup_dev command installs the development environment dependencies, including python3, various network tools, a Java development kit and specific versions of Mininet and Faucet. The setup of the development environment is required to work with mudacl, the MUD ACL genenerator.
Once the base and development software packages are installed, build DAQ with the following command:
sudo cmd/build
This program is likely to take a long time to execute, so go get yourself a cup of your favourite hot or cool beverage.
If you want to monitor the build progress, look inside the build folder. The build command outputs several log files that you can show in the command line, for instance with the following command:
tail -f build/docker_build.aardvark
Once DAQ has been built, execute this command to run the DAQ tests for the first time.
sudo cmd/run -s
This command creates the local directory and copies the misc/system_base.conf file into it with the new name local/system.conf.
It then activates Open vSwitch, configures Faucet with a single virtual ethernet port (faux) and executes the tests. The option -s stands for "single shot" and executes the tests only once.
If all goes well, the console will show something similar to the log below. In this case we have installed DAQ in the /opt/daq directory.
No local/system.conf found, copying defaults from misc/system_base.conf...
Loading config from local/system.conf
run_mode is local
Clearing previous reports...
Loading config from local/system.conf
Release version 0.9.0
cleanup='echo cleanup'
intf_names=faux
run_mode=local
site_description='Baseline Configuration'
skip_autostart=
startup_cmds=misc/startup_base.cmd
ovsdb-server is running with pid 28200
ovs-vswitchd is running with pid 20342
Sourcing misc/startup_base.cmd...
DAQ autostart cmd/faux
Loading config from local/system.conf
Launching faux ...
Clensing old container daq-faux
DAQ autoclean docker kill daq-faux
Removing old interface faux
Adding new interface to 9099...
Done with faux device launch.
Entering virtual python environment...
Using python3 at /opt/daq/venv/bin/python3
Executing: python3 -u daq/daq.py /opt/daq/local/system.conf -s
processing arg: /opt/daq/local/system.conf
Reading config from /opt/daq/local/system.conf
Reading config from misc/system.conf
processing arg: -s
INFO:daq:configuration map: {'run_mode': 'local', 'startup_cmds': 'misc/startup_base.cmd', 'single_shot': True, 'site_description': 'Baseline Configuration', 'intf_names': 'faux'}
INFO:daq:pid is 9184
INFO:gcp:No gcp_cred credential specified in config
INFO:runner:Reading test definition file misc/host_tests.conf
INFO:runner:Configured with tests ['pass', 'fail', 'ping', 'bacnet', 'nmap', 'mudgee']
INFO:network:Activating faucet topology...
INFO:topology:No device_specs file specified, skipping...
INFO:topology:Writing network config to inst/faucet.yaml
INFO:topology:Starting faucet...
INFO:network:Creating ovs sec with dpid/port 2/7
INFO:network:Added switch link pri-eth1 <-> sec-eth7
INFO:network:Attaching device interface faux on port 1.
INFO:network:Attaching device interface sec-2 on port 2.
INFO:network:Attaching device interface sec-3 on port 3.
INFO:network:Attaching device interface sec-4 on port 4.
INFO:network:Attaching device interface sec-5 on port 5.
INFO:network:Attaching device interface sec-6 on port 6.
INFO:network:Starting mininet...
INFO:mininet:*** Configuring hosts
INFO:mininet:*** Starting controller
INFO:mininet:controller
INFO:mininet:*** Starting 2 switches
INFO:mininet:pri
INFO:mininet:sec
INFO:mininet:...
INFO:runner:Waiting for system to settle...
INFO:runner:Entering main event loop.
INFO:runner:If this blocks for too long, check inst/faucet.log for errors
INFO:runner:Port 1 dpid 2 is now active True
INFO:runner:System port 7 on dpid 2 is active True
INFO:runner:Port 1 dpid 2 learned 9a:02:57:1e:8f:00
INFO:runner:Gateway for device group 9a02571e8f00 not found, initializing base 1...
INFO:gateway:Initializing gateway 9a02571e8f00 as gw01/10
INFO:gateway:Adding networking host gw01 on port 10 at 10.0.0.1
INFO:gateway:Added dummy target dummy01 on port 11 at 10.0.0.2
INFO:dhcp:DHCP monitor gw01 waiting for replies...
INFO:runner:Test ping gw01->dummy01
INFO:runner:Test ping gw01->dummy01
INFO:runner:Test ping dummy01->gw01
INFO:runner:Test ping dummy01->192.168.84.1
INFO:runner:Test ping gw01->dummy01 from 192.168.84.1
INFO:gateway:Attaching target 1 to gateway group 9a02571e8f00
INFO:host:Creating report as inst/report_9a02571e8f00_2018-11-24T22:32:58.txt
INFO:runner:Target port 1 registered 9a:02:57:1e:8f:00
INFO:host:Target port 1 initializing...
INFO:network:Creating mirror pair mirror-01 <-> mirror-01-ext at 1001
INFO:host:Target port 1 startup pcap capture
INFO:network:Directing traffic for 9a:02:57:1e:8f:00 on port 1: True
INFO:topology:Cleared port acls for port 1
INFO:host:Target port 1 activating as 9a:02:57:1e:8f:00
INFO:runner:DHCP notify 9a:02:57:1e:8f:00 is 10.20.63.37 on gw01 (None)
INFO:host:Target port 1 dhcp_result 10.20.63.37
INFO:runner:Test ping gw01->10.20.63.37
INFO:runner:Test ping gw01->10.20.63.37
INFO:runner:Test ping gw01->10.20.63.37 from 192.168.84.1
INFO:host:Target port 1 monitor scan complete
INFO:host:Target port 1 done with base.
INFO:host:Target port 1 background scan for 20 seconds...
INFO:host:Target port 1 scan complete
INFO:host:Target port 1 monitor scan complete
INFO:docker:Target port 1 PASSED test pass
INFO:docker:Target port 1 FAILED test fail with error 1: None
INFO:docker:Target port 1 PASSED test ping
INFO:docker:Target port 1 PASSED test bacnet
INFO:runner:DHCP notify 9a:02:57:1e:8f:00 is 10.20.63.37 on gw01 (None)
INFO:docker:Target port 1 PASSED test nmap
INFO:docker:Target port 1 PASSED test mudgee
INFO:host:Target port 1 no more tests remaining
INFO:host:Finalizing report inst/report_9a02571e8f00_2018-11-24T22:32:58.txt
INFO:gcp:Ignoring report upload: not configured
INFO:runner:Target port 1 finalize: [] (target set not active)
INFO:runner:Target port 1 cancel 9a:02:57:1e:8f:00 (#1/0).
INFO:network:Directing traffic for 9a:02:57:1e:8f:00 on port 1: False
INFO:topology:Cleared port acls for port 1
INFO:host:Target port 1 terminate, trigger False
INFO:network:Deleting mirror pair mirror-01 <-> mirror-01-ext
INFO:gateway:Detach target 1 from gateway group 9a02571e8f00
INFO:runner:Retiring target gateway 1, 9a:02:57:1e:8f:00, 9a02571e8f00, 1
INFO:gateway:Terminating gateway 9a02571e8f00
WARNING:runner:Suppressing future tests because test done in single shot.
INFO:runner:Remaining target sets: []
WARNING:runner:No active ports remaining: ending test run.
INFO:mininet:*** Stopping 1 controllers
INFO:mininet:controller
INFO:mininet:*** Stopping 1 links
INFO:mininet:.
INFO:mininet:*** Stopping 2 switches
INFO:mininet:pri
INFO:mininet:sec
INFO:mininet:*** Stopping 1 hosts
INFO:mininet:gw01
INFO:mininet:*** Done
INFO:runner:Done with runner.
INFO:daq:DAQ runner returned 0
Cleanup docker kill daq-faux
daq-faux
Done with run, exit 0
While running the test, switch to a different console terminal window and check which docker containers are running:
sudo docker ps
The output will be similar to the following:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 40125f7e070b daq/networking "env TERM=dumb PS1=\u007f…" 23 seconds ago Up 21 seconds daq-gw01 bd1114d18b2c daq/faucet "/usr/local/bin/entr…" 32 seconds ago Up 29 seconds 0.0.0.0:6653->6653/tcp daq-faucet a5fd5cc321d4 daq/faux "bin/start_faux" 34 seconds ago Up 32 seconds daq-faux
After successfully exiting the test, DAQ will have created a report in the inst folder. In this case the report is named inst/report_9a02571e8f00_2018-11-24T22:32:58.txt, showing the ethernet MAC address and report date in the name.
The local/system.conf is the configuration file to use to set up the DAQ network topology and some of the run time configuration variables without having to put them in the command line.
DAQ supports many topologies categories, as documented in the reference page on Network Topologies: emulation, adapter, test lab, tiered and production.
In the previous section, DAQ configured and executed the tests on an emulated network interface (named faux), as indicated by this output line:
INFO:network:Attaching device interface faux on port 1.
We'll now proceed to configure DAQ to use a physical USB ethernet network adapter.
Once the adapter is plugged into the USB port, it will add a network interface.
Identify it using the ifconfig command:
sudo ifconfig
See below an example of the output, showing the network interface has been registered as enx000ec6c32a6f.
...
enx000ec6c32a6f: flags=4163 mtu 1500
inet6 fe80::20e:c6ff:fec3:2a6f prefixlen 64 scopeid 0x20
ether 00:0e:c6:c3:2a:6f txqueuelen 1000 (Ethernet)
RX packets 1233820 bytes 83057697 (79.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2698368 bytes 204281978 (194.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
...
Replace the intf_names=faux line with intf_names=enx000ec6c32a6f in local/system.conf.
This configuration file is well documented and you can also take this opportunity to modify the site_description
# Base configuration file for DAQ. # Load defaults. source misc/system.conf # CSV separated list of names to assign to external switch interfaces. intf_names=enx000ec6c32a6f
Now connect a device to the USB ethernet adapter, and execute again the "single shot" tests:
sudo cmd/run -s
The tests are designed to run in parallel on multiple devices. Get at least another ethernet adapter, find its ID:
sudo ifconfig
...
enx9cebe840a74c: flags=4163 mtu 1500
ether 9c:eb:e8:40:a7:4c txqueuelen 1000 (Ethernet)
RX packets 62563 bytes 20983980 (20.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29 bytes 5141 (5.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
...
and include it inside the local/system.conf file.
# Base configuration file for DAQ. # Load defaults. source misc/system.conf # CSV separated list of names to assign to external switch interfaces. intf_names=enx000ec6c32a6f,enx9cebe840a74c
Now execute again the "single shot" tests:
sudo cmd/run -s
Check which docker containers have been created by DAQ:
sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 48ab066371e2 daq/networking "env TERM=dumb PS1=\u007f…" 25 seconds ago Up 23 seconds daq-gw01 d198e6b78742 daq/networking "env TERM=dumb PS1=\u007f…" About a minute ago Up About a minute daq-gw02 3b60351f2e8d daq/faucet "/usr/local/bin/entr…" About a minute ago Up About a minute 0.0.0.0:6653->6653/tcp daq-faucet
The DAQ command line output will confirm that the two USB ethernet adapters have been connected.
Loading config from local/system.conf
Starting Fri 21 Dec 14:19:46 GMT 2018, run_mode is local
Clearing previous reports...
Running as root...
Loading config from local/system.conf
Release version 0.9.0
cleanup='echo cleanup'
intf_names=enx000ec6c32a6f,enx9cebe840a74c
run_mode=local
site_description='Two devices'
skip_autostart=
ovsdb-server is running with pid 528
ovs-vswitchd is running with pid 20151
Entering virtual python environment...
Using python3 at /opt/daq/venv/bin/python3
Executing: python3 -u daq/daq.py /opt/daq/local/system.conf -s
processing arg: /opt/daq/local/system.conf
Reading config from /opt/daq/local/system.conf
Reading config from misc/system.conf
processing arg: -s
INFO:daq:configuration map: {'single_shot': True, 'site_description': 'Two devices', 'intf_names': 'enx000ec6c32a6f,enx9cebe840a74c', 'run_mode': 'local'}
INFO:daq:pid is 20586
INFO:gcp:No gcp_cred credential specified in config
INFO:runner:Reading test definition file misc/host_tests.conf
INFO:runner:Reading test definition file subset/pentests/pentests.conf
INFO:runner:Configured with tests ['pass', 'fail', 'ping', 'bacnet', 'mudgee', 'nmap', 'brute']
INFO:network:Activating faucet topology...
INFO:topology:No device_specs file specified, skipping...
INFO:topology:Writing network config to inst/faucet.yaml
INFO:topology:Starting faucet...
INFO:network:Creating ovs sec with dpid/port 2/7
INFO:network:Added switch link pri-eth1 <-> sec-eth7
INFO:network:Attaching device interface enx000ec6c32a6f on port 1.
INFO:network:Attaching device interface enx9cebe840a74c on port 2.
INFO:network:Attaching device interface sec-3 on port 3.
INFO:network:Attaching device interface sec-4 on port 4.
INFO:network:Attaching device interface sec-5 on port 5.
INFO:network:Attaching device interface sec-6 on port 6.
INFO:network:Starting mininet...
INFO:mininet:*** Configuring hosts
INFO:mininet:*** Starting controller
INFO:mininet:controller
INFO:mininet:*** Starting 2 switches
INFO:mininet:pri
INFO:mininet:sec
INFO:mininet:...
INFO:runner:Waiting for system to settle...
INFO:runner:Entering main event loop.
INFO:runner:If this blocks for too long, check inst/faucet.log for errors
INFO:runner:Port 1 dpid 2 is now active True
INFO:runner:Port 2 dpid 2 is now active True
INFO:runner:System port 7 on dpid 2 is active True
INFO:runner:Port 2 dpid 2 learned 38:d1:35:02:00:50
INFO:runner:Gateway for device group 38d135020050 not found, initializing base 2...
INFO:gateway:Initializing gateway 38d135020050 as gw02/20
INFO:gateway:Adding networking host gw02 on port 20 at 10.0.0.1
INFO:gateway:Added dummy target dummy02 on port 21 at 10.0.0.2
INFO:dhcp:DHCP monitor gw02 waiting for replies...
INFO:runner:Test ping gw02->dummy02
INFO:runner:Test ping gw02->dummy02
INFO:runner:Test ping gw02->dummy02
INFO:runner:Test ping dummy02->gw02
INFO:runner:Test ping dummy02->192.168.84.2
INFO:runner:Test ping gw02->dummy02 from 192.168.84.2
INFO:gateway:Attaching target 2 to gateway group 38d135020050
INFO:host:Creating report as inst/report_38d135020050_2018-12-21T14:20:16.txt
INFO:runner:Target port 2 registered 38:d1:35:02:00:50
INFO:host:Target port 2 initializing...
INFO:network:Creating mirror pair mirror-02 <-> mirror-02-ext at 1002
INFO:host:Target port 2 startup pcap capture
INFO:network:Directing traffic for 38:d1:35:02:00:50 on port 2: True
INFO:host:Target port 2 waiting for dhcp as 38:d1:35:02:00:50
INFO:gateway:Ready target 38:d1:35:02:00:50 from gateway group 38d135020050
INFO:runner:DHCP activating target 38:d1:35:02:00:50
INFO:host:Target port 2 triggered as 10.20.33.179
INFO:runner:Test ping gw02->10.20.33.179
INFO:runner:Test ping gw02->10.20.33.179
INFO:runner:Test ping gw02->10.20.33.179 from 192.168.84.2
INFO:host:Target port 2 monitor scan complete
INFO:host:Target port 2 done with base.
INFO:host:Target port 2 background scan for 30 seconds...
INFO:runner:Port 1 dpid 2 learned 04:a3:16:ea:06:33
INFO:runner:Gateway for device group 04a316ea0633 not found, initializing base 1...
INFO:gateway:Initializing gateway 04a316ea0633 as gw01/10
INFO:gateway:Adding networking host gw01 on port 10 at 10.0.0.3
INFO:gateway:Added dummy target dummy01 on port 11 at 10.0.0.4
INFO:dhcp:DHCP monitor gw01 waiting for replies...
INFO:runner:Test ping gw01->dummy01
INFO:runner:Test ping gw01->dummy01
INFO:runner:Test ping gw01->dummy01
INFO:runner:Test ping dummy01->gw01
INFO:runner:Test ping dummy01->192.168.84.1
INFO:runner:Test ping gw01->dummy01 from 192.168.84.1
INFO:gateway:Attaching target 1 to gateway group 04a316ea0633
INFO:host:Creating report as inst/report_04a316ea0633_2018-12-21T14:20:58.txt
INFO:runner:Target port 1 registered 04:a3:16:ea:06:33
INFO:host:Target port 1 initializing...
INFO:network:Creating mirror pair mirror-01 <-> mirror-01-ext at 1001
INFO:host:Target port 1 startup pcap capture
INFO:network:Directing traffic for 04:a3:16:ea:06:33 on port 1: True
INFO:host:Target port 1 waiting for dhcp as 04:a3:16:ea:06:33
INFO:host:Target port 2 scan complete
INFO:host:Target port 2 monitor scan complete
INFO:docker:Target port 2 test pass running
INFO:docker:Target port 2 test pass passed
INFO:docker:Target port 2 test fail running
INFO:docker:Target port 2 test fail failed: 1 None
INFO:docker:Target port 2 test ping running
INFO:docker:Target port 2 test ping passed
INFO:docker:Target port 2 test bacnet running
INFO:docker:Target port 2 test bacnet passed
INFO:docker:Target port 2 test mudgee running
INFO:docker:Target port 2 test mudgee passed
INFO:docker:Target port 2 test nmap running
INFO:gateway:Ready target 04:a3:16:ea:06:33 from gateway group 04a316ea0633
INFO:runner:DHCP activating target 04:a3:16:ea:06:33
INFO:host:Target port 1 triggered as 10.20.16.91
INFO:runner:Test ping gw01->10.20.16.91
INFO:runner:Test ping gw01->10.20.16.91
INFO:runner:Test ping gw01->10.20.16.91 from 192.168.84.1
INFO:host:Target port 1 monitor scan complete
INFO:host:Target port 1 done with base.
INFO:host:Target port 1 background scan for 30 seconds...
INFO:runner:Port 1 dpid 2 learned 04:a3:16:ea:06:33
INFO:host:Target port 1 scan complete
INFO:host:Target port 1 monitor scan complete
INFO:docker:Target port 1 test pass running
INFO:docker:Target port 1 test pass passed
INFO:docker:Target port 1 test fail running
INFO:docker:Target port 1 test fail failed: 1 None
INFO:docker:Target port 1 test ping running
INFO:docker:Target port 1 test ping passed
INFO:docker:Target port 1 test bacnet running
INFO:docker:Target port 1 test bacnet passed
INFO:docker:Target port 1 test mudgee running
INFO:docker:Target port 1 test mudgee passed
INFO:docker:Target port 1 test nmap running
INFO:runner:Port 1 dpid 2 learned 04:a3:16:ea:06:33
INFO:docker:Target port 2 test nmap failed: 1 None
INFO:docker:Target port 2 test brute running
INFO:docker:Target port 2 test brute passed
INFO:host:Target port 2 no more tests remaining
INFO:host:Finalizing report inst/report_38d135020050_2018-12-21T14:20:16.txt
INFO:gcp:Ignoring report upload: not configured
INFO:runner:Target port 2 finalize: ['02:nmap:1'] (target set not active)
INFO:runner:Target port 2 cancel 38:d1:35:02:00:50 (#2/0).
INFO:network:Directing traffic for 38:d1:35:02:00:50 on port 2: False
INFO:host:Target port 2 terminate, trigger False
INFO:network:Deleting mirror pair mirror-02 <-> mirror-02-ext
INFO:gateway:Detach target 2 from gateway group 38d135020050
INFO:runner:Retiring target gateway 2, 38:d1:35:02:00:50, 38d135020050, 2
INFO:gateway:Terminating gateway 38d135020050
WARNING:runner:Suppressing future tests because test done in single shot.
INFO:runner:Remaining target sets: [1]
INFO:runner:Port 1 dpid 2 learned 04:a3:16:ea:06:33
INFO:docker:Target port 1 test nmap failed: 1 None
INFO:docker:Target port 1 test brute running
INFO:docker:Target port 1 test brute passed
INFO:host:Target port 1 no more tests remaining
INFO:host:Finalizing report inst/report_04a316ea0633_2018-12-21T14:20:58.txt
INFO:gcp:Ignoring report upload: not configured
INFO:runner:Target port 1 finalize: ['01:nmap:1'] (target set not active)
INFO:runner:Target port 1 cancel 04:a3:16:ea:06:33 (#2/0).
INFO:network:Directing traffic for 04:a3:16:ea:06:33 on port 1: False
INFO:host:Target port 1 terminate, trigger False
INFO:network:Deleting mirror pair mirror-01 <-> mirror-01-ext
INFO:gateway:Detach target 1 from gateway group 04a316ea0633
INFO:runner:Retiring target gateway 1, 04:a3:16:ea:06:33, 04a316ea0633, 1
INFO:gateway:Terminating gateway 04a316ea0633
INFO:runner:Remaining target sets: []
WARNING:runner:No active ports remaining: ending test run.
INFO:mininet:*** Stopping 1 controllers
INFO:mininet:controller
INFO:mininet:*** Stopping 1 links
INFO:mininet:.
INFO:mininet:*** Stopping 2 switches
INFO:mininet:pri
INFO:mininet:sec
INFO:mininet:*** Stopping 1 hosts
INFO:mininet:gw02
INFO:mininet:*** Done
INFO:runner:Done with runner.
Go to the next tutorial > Set up the DAQ dashboard