Sysmon for Linux and distribution/kernel support

[MarioHewardt]

Hello Sysmon community,

First off, my apologies for being slow to respond to issues filed. We now have 3 tools in the Sysinternals for Linux portfolio and I'm spending my time across all three tools. With that, I am shifting full time focus to Sysmon for the foreseeable future. In terms of the issues and short term plans, most of the issues can be placed into two categories:

1. Sysmon doesn't start/work on distro/kernel X
2. Functional issues (such as certain fields being incorrect)

I'm currently spending my time on (1) as I want to first and foremost unblock users so they can successfully run Sysmon on their installed distributions. With that priority in mind, I want to take some time to explain some of the nuances of getting Sysmon to run across distributions/kernels.

Kernel data structure offsets
Sysmon needs to traverse kernel mode data structures to get the data it reports to users. More specifically it needs the offsets within the kernel mode structures. Due to the large number of Linux kernels, we employ a two-pronged approach to get the offsets:

1. Use a file that contains the offsets. This file is generated by a tool (getOffsets) that we ship with Sysmon. It requires the kernel header package to be installed on the target machine. You can find the instructions here - https://github.com/Sysinternals/SysinternalsEBPF/blob/main/getOffsets/README.md
2. Auto-discovery. This is where we live interrogate memory and try to guess the offsets based on certain markers within the kernel mode data structure. Please note that we may not always successfully find the offsets via this mechanism.

What does this mean from a usage perspective?
Don't worry about generating a config file via getOffsets. Start by simply running sysmon (which will use auto discovery). If sysmon fails to start or events seem to have incorrect fields (such as empty fields) only then use the getOffsets program. Please do let us know that Sysmon failed with the distribution/kernel version and we may be able to fix the auto-discovery.
Please note that getOffsets only has to be run once for that particular kernel. The config file can then be copied to other machines. Also note that getOffsets builds a kernel module. However, we don't ever load the kernel module, rather extract sections from the binary that gives us the offsets.

Longer term, we plan on adding support for BTF which should eliminate most of these issues.

Couple of fixes coming up in the near future
There is a bug that results in Sysmon not being able to read the above config file. The fix for this will be pushed soon.
There is a bug where the eBPF verifier complains about unbounded access (there are a few issues with this error). The fix for this will be pushed soon.

[weslambert]

Thanks, Mario!

I'm sure it's been tough maintaining all of these tools, but I definitely appreciate you being able to work more on Sysmon for Linux, and look forward to being able to use it in a more widescale fashion once some of these issues are addressed!

One thing we've asked for before was native JSON and socket output. I think this would really make it easier for folks ingesting this data.

#49
#50

[MarioHewardt]

Hi @weslambert - It is definitely a good idea and at the top of the list once some of these distro/kernel issues are sorted out. Appreciate the feedback!

[GhostKeep]

Good instructions. Everything went smooth until I got to the build. >> error compiling "networkmiscCommon.cpp". Upon inspection I noticed a declaration was missing. Once added, I was able to compile. I know that's not you but wanted to share the feedback.

[MarioHewardt]

Hi,

Wanted to provide an update on the Sysmon distro work.

The two bugs above ("Couple of fixes coming up in the near future") have been fixed.

I'm also happy to announce that Sysmon now works with BTF enabled kernels and latest libbpf which means we get the CORE capability. If the system you are running Sysmon on has BTF enabled Sysmon will use it. This gains us the capability of not having to rely on auto discovery of offsets, rather have the kernel relocate based on the BTF information. Not all distros (particularly the older ones) have BTF enabled. In this case Sysmon will revert to auto-discovery. There is a really nice GH repo (https://github.com/aquasecurity/btfhub) that has offline BTF files that you can download. You can then use this BTF file by using the /btf and Sysmon will use the specified BTF file. Of course, you can always use the getOffsets approach as well. If you are interested in knowing which approach Sysmon is using on any given system check syslog upon Sysmon startup and look for - "Discovery process: XYZ"

We haven't published the packages for the above yet so you will have to build from source. I will update once the packages have been published.

Thanks!