Design and Architecture
The purpose of this document is to provide the reader with a high level understanding of the principal architectural components of Monopticon as defined in version 1.0.0. The processes, wire formats and principal features are all described in detail.
Monopticon functions by reading raw packets received at a standard network interface. Network interfaces can have many configurations to mirror traffic from different networks, mirror traffic from a normally connected PC or be a ‘dummy’ interface whose only purpose is to replay old traffic.
In other words, Monopticon analyzes packet traffic supplied by a network tap. Certain tap setups can combine traffic recorded from many interfaces to a single “mirrored” tap that then is processed by zeek.
The function of the application can be broken up into three principal parts. The monitor server, the web server and the web page. These are the components as conceived for a website that can serve many users monitoring many different networks. Simpler setups where the monitor and web server are combined are possible as well.
The figure below shows the architecture and how the major pieces fit together. The objective is to display traffic observed on the “ethernet” line at the top of the page rendered on the web-page shown at the bottom.
The following sections describe in more detail the individual processes, wire formats and object models used by the application.
The Zeek process controlled by the bash script monopt_iface_proto drives the capture of traffic and primary analysis of that traffic. It is composed of a bare zeek engine with a custom script src/scripts/epoch_events.zeek that maintains the in-memory representation of the monitored network that will be transmitted to the web server.
The epoch_event script primarily updates models of the local network, but it also establishes a connection to an endpoint using the broker protocol to stream a summary of network events and status information about its status periodically. The following figure demonstrates the structure of these updates.
This summary of network events is contained in a zeek type named EpochEvent that contains a list of MACs that have communicated, a list of new MACs seen in the sample period, a list of new IPs associated with MACs in the sample and other similar information.
The sample period is defined as the period that an EpochEvent describes. As of version 1.0.0 of the period is 250 ms meaning EpochEvents are streamed to the mux server at 4Hz. Previous versions could sample as high as 100Hz but with systems physically distant, fewer larger samples were preferred.
The wire format used to send these EpochEvents is the native format used across Zeek clusters. It is a versioned binary format that supports arbitrary defined types based on a series of common primitives like integers, strings and even has dedicated types for things like subnets and IPs.
The broker itself connects over a TLS connection using the SSL library but by default does not enforce any kind of x509 certificate controls. This means that connections can only be considered secure after communicating partners are certain that the shared secret used to encrypt blocks was not tampered with. If the handshake was authenticated and unmodified, the stream itself will remain protected as long as it is forced to not renegotiate by an attacker.
The web server is the receiver of the epoch events streamed to it by the monitor server. At a high level, this server aggregates streamed epoch events and mirrors them to connected web clients. On one side nginx serves static web content and forwards websocket connections directly to a ‘mux_process’ that streams epoch events via a websocket to a web browser.
The mux_process is a multithreaded process written in C++. It leverages the threading models provided by the websocketpp library to receive the streamed epoch event’s, deserialize them then reserialize them in a different more compact binary representation for transmission to connected client websocket connections. There are many problems when reusing the zeek’s broker wire format.
To keep the implementation compact and efficient the protocol buffer library serializes the final binary wire format for transmission over websockets. Each epoch event is parsed after it is pulled from the broker’s queue and then converted into its protocol buffer binary representation only one time. A pointer to this array and its size is the only thing shared across threads.
This series of steps constitutes a single iteration of the main loop of the mux_process. The only other actions the service takes is occasional status logging and the setup and tear-down of websocket connections. Multiple instances of the mux_process can run a single machine and can be used to aggregate the output of monitor servers on a single machine that then can provide access to multiple monitored networks via the web page served via nginx.
The nginx server that forwards websockets to the mux_process also handles access control, enforcing transport security (TLS) and serves the static files described in the “web page” section.
Access control is handled in version 1.0.0 via a simple UUID-4 that is distributed to the intended end user of the application when they are given a URL to access the website. All access to websockets and files is pinned to that token. Since knowledge of the token which is distributed in a URL grants access potentially permanently, after every usage the token is changed to revoke existing access.
For simple access control during demonstrations this is sufficient protection but for production usage of sensitive systems a more robust protocol should be followed.
The web page rendered by a connecting browser has many components that can be understood by simply looking at the structure of the paths used in the URLs.
The index or root of an instance of the Monopticon application is an HTML page reachable at a URL like the following. It’s easy to see the random token here.
https://webopt.aextrac.top/f3327489-f2fb-448d-8f6c-72484952354a/
That URL exposes the web page which serves as the entry point to explain how the tool works and choose which network to monitor. In this case there are two networks shown. A simulated network and then a simple home Wi-Fi net.
If we replace the token with {token}, It’s easy to see where normal content like images and CSS can be found in the subdirectories:
https://webopt.aextrac.top/{token}/images/ https://webopt.aextrac.top/{token}/styles/
The content necessary to serve the 3-D interface is instead given its own path extension to differentiate websocket endpoints based on that path. So for example the visualization of my houses wifi is loaded at this address.
https://webopt.aextrac.top/{token}/home-wifi/
At this endpoint there are five files that work together to produce the final 3-D visual. Three are required by the framework (EmscriptenApplication.js, WebApplication.css and index.html) while the application loader and wasm binary (monopticon.js and monopticon.wasm) itself are the principal pieces of the release of monopticon v1.0.0.
The moment this page loads the EmscriptenApplication.js file launches and handles the setup for the monopticon js and wasm files.
Finally once the application is initialized by the web browser, a request is sent to establish an encrypted websocket via the fully specified URL.
wss://webopt.aextrac.top/{token}/home-wifi/socket
This is the path that nginx will forward to the corresponding mux_server described by the local URI like the one below:
ws://localhost:9002
The application binary while relatively small (2.1 MB) provides the entire user interface. It is loaded via an Emscripten framework and leverages other relatively new web technologies. The combination of WebGL 2.0 and Web Assembly (or wasm for brevity) makes it possible to render the application at a smooth 60 FPS - 16.6ms per frame.
The application itself is limited in size by the upper bounds of memory limits placed on browser tabs. Currently, at a rough 20MB of allocatable memory in Chromium (version 81.0.4044) about 50 devices can be rendered and monitored. Exceeding this 20MB ceiling results in the browser killing the application but this can be disabled if necessary by changing the memory limits the browser enforces.
During the compilation of the software, models of networks can be embedded within the wasm binary as static layouts to be used when the application detects it running at a certain path like /home-wifi/. These layouts significantly enrich the display of complex networks.
For additional information like MAC, IP addresses and human readable names a second XML file is used.
After startup and layout, the wasm application automatically connects to the websocket and parses streamed messages sent by the server. These messages are encoded as ‘lite’ protocol buffers whose format is defined by the file src/epoch.proto. This format is analogous to the zeek message type epoch_event defined in src/scripts/epoch_events.zeek but there are some key differences.
For simplicity native zeek types were used for epoch_events while the EpochStep message protobuf uses compact representations of MACs and IPs. This helps reduce the number packets streamed per second from the mux_process to the wasm application.
Once those packets arrive they are deserialized and graphical effects are launched based on the events contained within the single EpochStep.
Simple frames passed between two devices are represented by a PacketLine - a colored line segment that moves from the source to destination. The colors teal, purple, yellow and red correspond to IPv4, IPv6, ARP and an unidentified protocol for frames observed.
Because an EpochStep has captured a segment of time a single PacketLine can represent multiple packets transmitted from source to destination and often for connection oriented protocols both the data and corresponding ACK are contained in the same EpochStep and are shown to travel opposite ways as two PacketLines.
Since the addressing scheme of the entire application is based on device MACs the principal object contained in the application is a Device::Stats object. It is uniquely identified by its MAC and could have a significant amount of meta-data tracked as well.
Visually, these objects are represented as labelled spheres in the application. If available an IPv4 address added as a cube above the sphere - a Figure::Address. Pairs of line segments between these elements are used to show entries in a device’s ARP table.
Finally, broadcast traffic - traffic sent to a group as defined in the EUI-48 spec - is represented as pools. Any frame that has as a destination address defined by one of the four pools ff, 33, 01 or the catch all odd will be represented by a Figure::PacketLineDrawable then by a Figure::PoolShader effect.
Information such as the health of the websocket and traffic graphs for individual devices are presented as 2-D windows. These windows are painted by a library known as IMGui and are easy to modify. The action watch can be triggered both from the IMGui windows and from direct selection on the 3-D objects. From left to right: The Heads up Display, a normal device panel, a monitored router, the overall traffic analyzed by the application.
The traffic graphs represent various things like the number of transmitted (tx green) or received (rx blue) frames. The value next to avg ppb signifies the average of packets per bucket where each bucket corresponds to a single EpochStep. This means with 240 buckets represented in these charts and an EpochStep sample rate of 60Hz the graph below represents the last 4 seconds of tx and rx of the device. These are the statistics that give Device::Stats its name.
These major pieces: the 3-D objects, the visual effects and the windowed control represent the primary functionality of the wasm application.