Monitors & collects system resource utilization on Linux for Docker containers and Kubernetes pods with fine granularity and low overhead, emitting resource utilization logs in CSVY (csv + yaml) format. Originally, developed in Rust as a custom tool to help detect and analyze millibottlenecks in containerized online systems, rAdvisor runs by polling the target provider (either the local Docker daemon or the Kubernetes API server) every 1 second to get a list of active, running containers/pods. From this list, rAdvisor runs a collection thread every 50ms to get resource utilization data for each active target using Linux
cgroups, outputting the resultant logs in/var/log/radvisor/stats.
Note: filenames correspond to the ID/UID of the container/pod, with the collector initialization timestamp appended at the end.
---
Version: 1.4.0
Provider: docker
Metadata:
Created: "2022-01-09T13:25:04Z"
Command: 'bash -c ''sleep 2s; apt-get update; sleep 2s; DEBIAN_FRONTEND=noninteractive apt-get install -y stress wget; sleep 2s; dd if=/dev/zero of=/tmp/file1 bs=512M count=1 oflag=direct; sleep 2s; stress --cpu 8 --io 4 --vm 4 --vm-bytes 1024M --timeout 10s; sleep 2s; wget "http://ipv4.download.thinkbroadband.com/10MB.zip"; sleep 2s'''
Id: 7762ff15c99a2d238f4d26c22b5eda5b97ebc03bd0a711693104dcb6f71fe411
Image: ubuntu
Labels: {}
Names:
- /silly_elion
Ports: []
Status: Up Less than a second
SizeRw: ~
SizeRootFs: ~
PerfTable: # ...
System:
OsType: Linux
OsRelease: 4.15.0
Distribution: # ...
MemoryTotal: 65870408
SwapTotal: 3145724
Hostname: node-0.sandbox.infosphere.emulab.net
CpuCount: 32
CpuOnlineCount: 4
CpuSpeed: 1279
CollectorType: cgroup_v2
CollectorMetadata:
Cgroup: system.slice/docker-7762ff15c99a2d238f4d26c22b5eda5b97ebc03bd0a711693104dcb6f71fe411.scope
CgroupDriver: systemd
PolledAt: 1641734740142271945
InitializedAt: 1641734740157676566
---
read,pids.current,pids.max,cpu.stat/usage_usec,cpu.stat/system_usec,cpu.stat/user_usec,cpu.stat/nr_periods,cpu.stat/nr_throttled,cpu.stat/throttled_usec,memory.current,memory.high,memory.max,memory.stat/anon,memory.stat/file,memory.stat/kernel_stack,memory.stat/pagetables,memory.stat/percpu,memory.stat/sock,memory.stat/shmem,memory.stat/file_mapped,memory.stat/file_dirty,memory.stat/file_writeback,memory.stat/swapcached,memory.stat/inactive_anon,memory.stat/active_anon,memory.stat/inactive_file,memory.stat/active_file,memory.stat/unevictable,memory.stat/pgfault,memory.stat/pgmajfault,io.stat/rbytes,io.stat/wbytes,io.stat/rios,io.stat/wios,io.stat/dbytes,io.stat/dios
1641734705052508079,1,28989,58688,40630,18057,0,0,0,5558272,max,max,405504,3514368,49152,0,0,0,0,2838528,0,0,0,270336,0,1486848,2027520,0,1650,0,3891200,0,58,0,0,0
# ...More information about what each column represents can be found in the docs pages (for information about the columns ouputted when CollectorType: cgroup_v1, see this page instead).
---
Version: 1.4.0
Provider: kubernetes
Metadata:
Uid: 9f0b1893-15e7-442a-966a-b0d19a35fc1c
Name: kube-proxy-hsplg
CreatedAt: "2020-03-29T04:32:35Z"
Labels:
controller-revision-hash: c8bb659c5
k8s-app: kube-proxy
pod-template-generation: "1"
Namespace: kube-system
NodeName: node-0.sandbox.infosphere.emulab.net
HostIp: 130.127.133.26
Phase: Running
QosClass: BestEffort
StartedAt: "2020-03-29T04:32:36Z"
PerfTable: # ...
System:
OsType: Linux
OsRelease: 4.15.0
Distribution: # ...
MemoryTotal: 65870408
SwapTotal: 3145724
Hostname: node-0.sandbox.infosphere.emulab.net
CpuCount: 32
CpuOnlineCount: 32
CpuSpeed: 1198
PolledAt: 1585470948008442929
CollectorType: cgroup_v1
CollectorMetadata:
Cgroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f0b1893_15e7_442a_966a_b0d19a35fc1c.slice
CgroupDriver: systemd
InitializedAt: 1585470948030565581
---
read,pids.current,pids.max,cpu.usage.total,cpu.usage.system,cpu.usage.user,cpu.usage.percpu,cpu.stat.user,cpu.stat.system,cpu.throttling.periods,cpu.throttling.throttled.count,cpu.throttling.throttled.time,memory.usage.current,memory.usage.max,memory.limit.hard,memory.limit.soft,memory.failcnt,memory.hierarchical_limit.memory,memory.hierarchical_limit.memoryswap,memory.cache,memory.rss.all,memory.rss.huge,memory.mapped,memory.swap,memory.paged.in,memory.paged.out,memory.fault.total,memory.fault.major,memory.anon.inactive,memory.anon.active,memory.file.inactive,memory.file.active,memory.unevictable,blkio.time,blkio.sectors,blkio.service.bytes.read,blkio.service.bytes.write,blkio.service.bytes.sync,blkio.service.bytes.async,blkio.service.ios.read,blkio.service.ios.write,blkio.service.ios.sync,blkio.service.ios.async,blkio.service.time.read,blkio.service.time.write,blkio.service.time.sync,blkio.service.time.async,blkio.queued.read,blkio.queued.write,blkio.queued.sync,blkio.queued.async,blkio.wait.read,blkio.wait.write,blkio.wait.sync,blkio.wait.async,blkio.merged.read,blkio.merged.write,blkio.merged.sync,blkio.merged.async,blkio.throttle.service.bytes.read,blkio.throttle.service.bytes.write,blkio.throttle.service.bytes.sync,blkio.throttle.service.bytes.async,blkio.throttle.service.ios.read,blkio.throttle.service.ios.write,blkio.throttle.service.ios.sync,blkio.throttle.service.ios.async,blkio.bfq.service.bytes.read,blkio.bfq.service.bytes.write,blkio.bfq.service.bytes.sync,blkio.bfq.service.bytes.async,blkio.bfq.service.ios.read,blkio.bfq.service.ios.write,blkio.bfq.service.ios.sync,blkio.bfq.service.ios.async
1602390175053135973,18,4915,45675783181,0,45675783181,9719044209 12310201631 11027849186 12618688155 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0,2925,1668,0,0,0,2802823168,3667771392,9223372036854771712,9223372036854771712,0,9223372036854771712,,35323904,2754781184,0,28672,,6837817,6156636,6854722,0,0,2754711552,7380992,27942912,0,2273087370,1306336,0,668844032,662777856,6066176,0,331937,331753,184,0,68057100860,68011971780,45129080,0,0,0,0,0,222907407415,222860666999,46740416,0,32,0,32,0,668844032,662777856,6066176,0,331937,331753,184,,,,,,,,
# ...Many of the specific details of collection can be controlled via the command line interface. At the moment, this includes collection/polling intervals and output directory. To view information on the available CLI options, run radvisor help:
$ radvisor help
radvisor 1.4.0
Joseph Azevedo <[email protected]>, Bhanu Garg <[email protected]>
Monitors container resource utilization with high granularity and low overhead
USAGE:
radvisor [FLAGS] [OPTIONS] <SUBCOMMAND>
FLAGS:
-h, --help Prints help information
-q, --quiet Whether to run in quiet mode (minimal output)
-v, --verbose Whether to run in verbose mode (maximum output)
-V, --version Prints version information
OPTIONS:
-c, --color <color-mode> Color display mode for stdout/stderr output [default: auto]
SUBCOMMANDS:
help Prints this message or the help of the given subcommand(s)
run Runs a collection thread that writes resource statistics to output CSV files$ radvisor run <provider>The main subcommand of rAdvisor is run, which additionally requires the target provider (Docker or Kubernetes) to use to discover collection targets. For example, to run rAdvisor and collect resource utilization statistics on Docker containers each 40ms, the following command would be used:
$ radvisor run docker -i 40ms
Initializing Docker API provider
Beginning statistics collection
Identified cgroupfs as cgroup driverAt the moment, rAdvisor only supports Linux (due to its heavy reliance on cgroups), though there is a tracking issue for extending its functionality to work with Window's own first-party containerization API, HCS: radvisor/issues/#3.
To build rAdvisor using Docker, run the following command (needs to have docker installed and running, and likely needs to be run as root):
$ sudo make
For the Docker build method, the Rust stable image is used (rust) to run a Docker container with the necessary toolchains pre-installed.
To build rAdvisor from source, Rust stable is used. We recommend using rustup to install the Rust toolchain.
Now, in the cloned repository root, run make compile to generate a release-grade binary at ./target/release/radvisor. This build process may take up to ten minutes.
$ make compile
cargo build --release --bins \
--target x86_64-unknown-linux-gnu
Compiling libc v0.2.68
Compiling autocfg v1.0.0
Compiling cfg-if v0.1.10
...
Compiling shiplift v0.6.0
Compiling radvisor v1.4.0 (/home/jazev/dev/radvisor)
Finished release [optimized] target(s) in 4m 52s
$ ./radvisor --version
radvisor 1.4.0This project is licensed under the GNU General Public License v3.0.
Additionally, this software was developed as a part of research work at the Georgia Institute of Technology, and may be subject to license restrictions.