Project Proposal: Audit Logging SIG

[mlenkeit]

This PR contains a project proposal for an Audit Logging SIG as discussed on Slack.

We are aware that the project proposal still has several tbd's especially with regard to staffing and timeline that need to be defined before the SIG can start working.

We will approach other vendors directly with this proposal to identify additional contributors. Of course, anyone who comes across this proposal here on GitHub is invited to contribute.

While we do have some ideas about a potential timeline for semantic conventions, OTEL SDK/API and collector adjustments respectively, we would like to align this with other contributors first before publishing.

Any feedback from the community on the proposed scope of the SIG is highly appreciated!

Open topics

The following items reference topics from the PR discussion that are still open:

• immutability, tamper-proof logs, signing - in scope?
  • Project Proposal: Audit Logging SIG #2409 (comment)
  • Project Proposal: Audit Logging SIG #2409 (comment)
  • Project Proposal: Audit Logging SIG #2409 (comment)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: mlenkeit / name: Maximilian Lenkeit (f81c2f4, 9337b7f, 5094fb1, 65ae32e, 405ddb5, 0adb8e5, 75f2c57, a5ef343, d7e265f, 776b821, 6dd519d, 3876a31, 2ec002d, 087865c, 711dc46, a6b34f1, 066501b, 70cbac4, 8b38626)

[mtwo]

Suggested change

	- no semantic conventions for audit logs in OTEL
	- There aren't currently any semantic conventions designed specifically for audit logs in OTEL

[mtwo]

Are there any requirements around signing logs / detecting tampering? I've heard that mentioned before in the context of audit logs, but I don't know how common of a requirement it is

[reyang]

Good points! In addition, these are something we might want to consider:

1. Audit logs might be considered as a critical part of the business, which could result in a different API design strategy - for example, audit logging might require a different API behavior, if the information provided by the caller is invalid, the API might throw exception instead of failing silently and move on.
2. Audit log might require some sensitive information without redaction due to the regulation requirements (e.g. user identity and client IP address).
3. The data path could require higher level of access control or privilege.

[mlenkeit]

@reyang thanks for mentioning these points.

Especially the API behavior is something that we had thought about initially. However, when we first pitched audit logging on Slack, we received the following comment from Ted Young:

As a rule, the OpenTelemetry API never throws an exception. I understand why you might want this, though it is not present in many audit logging systems, which use regular loggers. So a strong case would have to be made on this particular point.

Based on this initial feedback, we decided to file this SIG proposal without proposing such API changes.

[reyang]

I think we need prototype in two parts:

1. API/SDK - this is where we need three programming languages IIRC.
2. OTel Collector - higher guarantee on data delivery (completeness, integrity, latency, etc.), data path security.

[mlenkeit]

Thanks for pointing this out! It's clear to us, but I'll work on making this clearer in the doc...

[mlenkeit]

Are there any requirements around signing logs / detecting tampering? I've heard that mentioned before in the context of audit logs, but I don't know how common of a requirement it is

@mtwo for all I know, immutability of audit logs is a common requirement although not all audit logging systems/use cases that I've seen address this requirement with technical measures but sometimes also organizational measures. However, given the flexibility of OTel processing queues (i.e. different topologies of collectors), having a technical solution in OTel would be favorable.

@reyang what is your opinion on this?

[svrnm]

Suggested change

	- no semantic conventions for audit logs in OTel
	- no semantic conventions for audit logs in OTel

Can you provide some examples of what would be part of such semantic conventions? My knowledge on audit logs is very limited, so it would help to understand the problem much better.

[mlenkeit]

@svrnm our experience has shown that in order to analyze audit logs at scale, it is important to define an (extensible) event catalog. The event catalog standardizes audit log events across workloads/produces. For example, our internal event catalog currently consists of 50+ such events. Ideally, such a catalog would be part of semantic conventions.

To make this more tangible, I've added some examples to the appendix of the document:
https://github.com/open-telemetry/community/pull/2409/files#diff-736e6b0ae9ae655b78d9ba007d08592071abb6cc1ef64d7893ff81642c8ec734R115-R192

[renewelches]

another examples from the security world is https://github.com/ocsf.

[svrnm]

thanks @mlenkeit. Makes it much clearer

The metadata looks like attributes that would be covered by other semantic conventions (e.g. there is a log.record.id for the metadata.id, the timestamp of course and some of the other ones (e.g. for k8sCluster we have k8s.cluster.name. So I would assume here it is more about re-using and extending certain other domains that are not unique to "audit logs"

For the event and data examples you gave, I would argue that they are not "semantic conventions for audit logs" but "semantic conventions for log types that typically require the strict requirements of auditing". What do I mean by that: if we talk about "semantic conventions for audit logs" I think about a namespace called audit. that holds attributes that are specific to the business logic of audit logging, like a signature that helps to tamperproof the log line, or maybe even meta information under which regulation this log is required to be an "audit log"
In contrast "semantic conventions for log types that typically require the strict requirements of auditing" are their own namespaces like the "UserLoginFailure" example would fall into a "authentication" or "auth" namespace, with "auth.login.method" or "auth.login.failureReason" as potential attributes, event.name being set to auth.login.failure or something.

I am just making those things up to exemplify the difference, they will probably take a different form or shape eventually, so to make a long story short, here is a suggestion to rephrase:

Suggested change

	- no semantic conventions for audit logs in OTel
	- no semantic conventions for audit logs in OTel
	- no semantic conventions for log types that typically rquire the strict requirements of auditing, like authentication, authorization and data changes

@renewelches thanks for calling out OCSF, if I remember correctly there were conversations in the past between OTel and OCSF, cc @lmolkova

[mlenkeit]

Regarding metadata, I fully agree: Most of these attributes are already covered by semconv. We may identify additional attributes in SIG meetings though, depending on the experience/requirement of other contributors/companies.

I understand how "semantic conventions for audit logs" can be misleading. To me, the suggestion that you made has a notion of particularly describe logs that are "already there" (e.g. events emitted by a K8s cluster) and can be considered relevant for audit purposes. Especially in enterprise software, it's common that applications produce logs that are specifically mean to be audit logs (and nothing else). To me, it' s important that we find wording that covers these two types that we do have.

How about the following?

Suggested change

	- no semantic conventions for audit logs in OTel
	- no semantic conventions for representing and identifying audit trail-relevant events in OTel (like authentication, authorization or modification of

[renewelches]

As mentioned in another comment, this all depends on what attributes are changeable or must be immutable. As of my understanding an attribute could be altered by a processor in the collector. Which is something we would want to avoid or want to prevent in cases of audit logs. If we conclude that we can or should only guarantee immutability for the log itself then we must live with replication/doublication. Otherwise we might have to add the constrain that also certain attributes must be immutable.

[reyang]

Are there any requirements around signing logs / detecting tampering? I've heard that mentioned before in the context of audit logs, but I don't know how common of a requirement it is

@mtwo for all I know, immutability of audit logs is a common requirement although not all audit logging systems/use cases that I've seen address this requirement with technical measures but sometimes also organizational measures. However, given the flexibility of OTel processing queues (i.e. different topologies of collectors), having a technical solution in OTel would be favorable.

@reyang what is your opinion on this?

@mlenkeit I think this can be achieved as long as OpenTelemetry is designed to allow additive changes, doesn't have to be there in the first place. I personally haven't seen people signing logs, and I've seen lots of cases where immutable data path is used.

[reyang]

Does "OTel collectors" mean the "OpenTelemetry Collector" (https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification#project-naming) or "any collectors that can handle OpenTelemetry data (whether OTLP or something else)"?

[mlenkeit]

@reyang This is indeed supposed to refer to the OpenTelemetry Collector. I've rephrased this particular sentence to "OTel Collector instances may lose audit logs in transit" and adjusted other occurrences of OTel collectors as well. 066501b

[reyang]

What is the expectation if the collector instance disappeared (e.g., the machine running the collector exploded / was stolen)?

[svrnm]

I think this is the most tricky part, or to put it in a question: do we need guarantee of delivery between 2 components (workload->collector,collector->S3) or end-to-end (workload->S3)?

I would assume "end-to-end" except the collector can guarantee that data is persisted according the auditing requirements

[mlenkeit]

(e.g., the machine running the collector exploded / was stolen)

If the solution for audit logging with OTel meant that the OTel Collector had an own persistence, I would argue that theft/explosion/etc. are rather in the responsibility of Operations in terms of configuring said persistence such that it is resilient "enough".

Or to make this more concrete: if for example something such as the storage extension was used, Operations would need to make sure that the database/file/redis storage runs in an HA mode.

I'm stressing the if here, because I think is a detail that the SIG should work out. Or do you think that's something that should rather be clarified upfront?

[reyang]

I'm stressing the if here, because I think is a detail that the SIG should work out. Or do you think that's something that should rather be clarified upfront?

I suggest that we leave this for the SIG to figure out. In the OTEP, I suggest that we avoid "guaranteed delivery" and use something like "certain degree/level of data delivery guarantee". Not a blocker for this PR though (I'm good with the current version).

[svrnm]

+1 for what @reyang wrote. I think it is good to have this in the appendix and some wording around it, since there is many people (including myself) who have a superficial knowledge around audit logs, so it helps to contextualize and understand what this is all about. So no more details are needed in this doc, this would be for the SIG to figure out

[renewelches]

Are there any requirements around signing logs / detecting tampering? I've heard that mentioned before in the context of audit logs, but I don't know how common of a requirement it is

@mtwo for all I know, immutability of audit logs is a common requirement although not all audit logging systems/use cases that I've seen address this requirement with technical measures but sometimes also organizational measures. However, given the flexibility of OTel processing queues (i.e. different topologies of collectors), having a technical solution in OTel would be favorable.
@reyang what is your opinion on this?

@mlenkeit I think this can be achieved as long as OpenTelemetry is designed to allow additive changes, doesn't have to be there in the first place. I personally haven't seen people signing logs, and I've seen lots of cases where immutable data path is used.

I think the key parts are encryption at REST e.g. when a buffer writes to disk and encryption at TRANSPORT.
Where it is getting tricky is when we have to separate audit log data and meta/transport data. In some cases this might lead to duplication. E.g the K8s cluster name that triggered the audit event could be part of the audit log message. In this case it must be immutable. But it is also an OTLP attribute and as such could be changed. Or to phrase it differently, is immutability required for the whole signal or just the message?

[mlenkeit]

Releasing for review as per @reyang's (offline) suggestion. I'm aware that there's open tbd's that we still need to fill.