v2.12.17

About this patch release

2024-12-10 lttng-tools 2.12.17 (National Lager Day)

Full changelog: v2.12.16...v2.12.17

About LTTng-tools 2.12

This release is named after Ta Meilleure, a Northeast IPA beer brewed by Lagabière. Translating to "Your best one", this beer gives out strong aromas of passion fruit, lemon, and peaches. Tastewise, expect a lot of fruit, a creamy texture, and a smooth lingering hop bitterness.

The most notable features of this new release are:

• session clearing,
• uid and gid tracking,
• file descriptor pooling (relay daemon),
• per-session grouping (relay daemon),
• working directory override (relay daemon),
• new network reception entry/exit tracepoints (LTTng-modules),
• statedump of interrupt threads (LTTng-modules),
• statedump of x86 CPU topology (LTTng-modules),
• new product UUID environment field (LTTng-modules).

Read on for a short description of each of these features and the links to this release!

Session clearing

You can use the new lttng-clear command to clear the contents of one or more tracing sessions.

In essence, this new feature allows you to prune the content of long-running sessions without destroying and reconfiguring them. This is especially useful to clear a session's tracing data between attempts to reproduce a problem.

Clearing a tracing session deletes the contents of the tracing buffers and all local or streamed trace data on a remote peer. Note that an lttng-relayd daemon can be configured to disallow clear operations using the LTTNG_RELAYD_DISALLOW_CLEAR environment variable.

If a session is configured in snapshot mode, only the tracing buffers are cleared.

If a session is configured in live mode, any attached client that is lagging behind will finish the consumption of its current trace data packets and jump forward in time to events generated after the beginning of the clear command.

uid and gid tracking

The existing lttng-track command has been expanded to support uid and gid tracking.

By default, a tracing session tracks all applications and users, following LTTng's permission model.
However, this new options allows you to restrict which users and groups are tracked by both the user space and kernel tracers.

In previous versions of LTTng, it was effectively possible to filter on the basis of uids and gids using the --filter mechanism. However, this dedicated filtering mechanism is both more efficient in terms of tracing overhead, but also prevents the creation of tracing buffers for users and groups which are not tracked.

Overall, this results in far less memory consumption by the user space tracer on systems which have multiple active users.

File descriptor pooling (relay daemon)

A number of users have reported having encountered file descriptor exhaustion issues when using the relay daemon to serve a large number of consumers or live clients.

The current on-disk CTF representation used by LTTng (and expected by a number of viewers) uses one file per CPU, per channel, to organize traces. This causes the default RLIMIT_NOFILE value (1024 on many systems) to be reached easily, especially when tracing systems with a large number of cores.

In order to alleviate this problem, the new --fd-pool-size option allows you to specify a maximal number of simultaneously opened file descriptors (using the soft RLIMIT_NOFILE resource limit of the process by default). This is meant as a work-around for users who can't bump the system-limit because of permission restrictions.

As its name indicates, this option causes the relay daemon to maintain a pool (or cache) of open file descriptors which are re-purposed as needed. The most recently used files' file descriptors are kept open and only closed as the --fd-pool-size limit is reached, keeping the number of simultaneously opened file descriptors under the user-specified limit.

Note that setting this value too low can degrade the performance of the relay daemon.

Per-session grouping (relay daemon)

By default, the relay daemon writes the traces under a predefined directory hierarchy:
$LTTNG_HOME/lttng-traces/HOSTNAME/SESSION/DOMAIN where

• HOSTNAME is the remote hostname,
• SESSION is the full session name,
• DOMAIN is the tracing domain (ust or kernel),

Using the new relay daemon --group-output-by-session option, you can now change this hierarchy to group traces by sessions, rather than by hostname:
$LTTNG_HOME/lttng-traces/SESSION/HOST/DOMAIN.

This proves especially useful if you are tracing a number of hosts (with different hostnames) which share the same session name as part of their configuration. Hence, a descriptive session name (e.g. connection-hang) can be used across a fleet of machines streaming to a given relay daemon.

Note that the default behaviour can be explicitly specified using the --group-output-by-host option.

Working directory override (relay daemon)

This small quality of life feature allows you to override the working directory of the relay daemon using the daemon's launch options (-w PATH/--working-directory=PATH).

New network reception entry/exit tracepoints (LTTng-modules)

New instrumentation hooks were added to the kernel tracer in order to trace the entry and exit tracepoints of the network reception code paths of the Linux kernel.

You can use those tracepoints to identify the bounds of a network reception and link the events that happen in the interim (e.g. wakeups) to a specific network reception instance. Those tracepoints can also be used to analyse the network stack's latency.

Statedump of interrupt threads (LTTng-modules)

Threaded IRQs have an associated thread field in the irqaction structure which specifies the process to wake up when the IRQ happens. This field is now extracted as part of the lttng_statedump_interrupt statedump tracepoint.

You can use this information to know which processes handle the various IRQs. It is also possible to associate the events occurring in the context of those processes to their respective IRQ.

Statedump of x86 CPU topology (LTTng-modules)

A new lttng_statedump_cpu_topology tracepoint has been added to extract the active CPU/NUMA topology. You can use this information to know which CPUs are SMT siblings or part of the same socket. For the time being, only x86 is supported since all architectures describe their topologies differently.

The architecture field is statically defined and should be present for all architecture implementations. Hence, it is possible for analysis tools to anticipate the event's layout.

Example output:

lttng_statedump_cpu_topology: { cpu_id = 3 }, { architecture = "x86", cpu_id = 0, vendor = "GenuineIntel", family = 6, model = 142, model_name = "Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz", physical_id = 0, core_id = 0, cores = 2 }

New product UUID environment field (LTTng-modules)

The product UUID, taken from the DMI system information, is now saved as part of the kernel traces' environment fields as the product_uuid. You can use this field to uniquely identify a machine (virtual or physical) in order to correlate traces gathered on multiple virtual machines.

v2.13.13

About this patch release

2024-04-05 lttng-tools 2.13.13 (National Deep Dish Pizza Day)

• Fix: consumerd: leak of tracing buffers on relayd connectivity issue
• Fix: consumerd: wrong timer mentioned in error logging
• Fix: consumerd: type confusion in lttng_consumer_send_error

Full Changelog: v2.13.12...v2.13.13

About LTTng-tools 2.13

The most notable features of this new release are:

• Event-rule matches condition triggers and new actions, allowing internal
  actions or external monitoring applications to quickly react when kernel
  or user-space instrumentation is hit,
• Notification payload capture, allowing external monitoring applications
  to read elements of the instrumentation payload when instrumentation is
  hit.
• Instrumentation API: vtracef and vtracelog (LTTng-UST),
• User space time namespace context (LTTng-UST and LTTng-modules).

This release is named after "Nordicité", the product of a collaboration between
Champ Libre and Boréale. This farmhouse IPA is brewed with Kveik yeast and
Québec-grown barley, oats and juniper branches. The result is a remarkable
fruity hazy golden IPA that offers a balanced touch of resinous and woodsy
bitterness.

Based on the LTTng project's documented stable releases lifetime, this 2.13
release coincides with the end-of-life of the LTTng 2.11 release series.

Read on for a short description of each of the new features and the
links to this release.

Note on LTTng-UST backward compatibility

• soname major version change
  This release changes the LTTng-UST soname major from 0 to 1.

  The event notifier (triggers using an event-rule-matches condition)
  functionality required a significant rework of public data structures which
  should never have been made public in the first place.

  Bumping the soname major to 1, will require applications and tracepoint
  providers to be rebuilt against an updated LTTng-UST to use it.

  Old applications and tracepoint providers linked against libraries with
  major soname 0 should be able to co-exist on the same system.

• Building probe providers using a C++ compiler requires C++11

• API namespaceing
  The LTTng-UST API is now systematically namespaced under lttng_ust_* (e.g.
  tracepoint() becomes lttng_ust_tracepoint()).

  However, the non-namespaced names are still exposed to maintain API
  compatibility.

Event-rule matches condition and new actions

Expanding the trigger infrastructure and making it usable through the lttng
client was the core focus of this release.

A trigger is an association between a condition and one or more actions. When
the condition associated to a trigger is met, the actions associated to that
trigger are executed. The tracing does not have to be active for the conditions
to be met, and triggers are independent from tracing sessions.

Since their introduction as part of LTTng 2.10, new conditions and actions were
added to make this little-known mechanism more flexible.

For instance, before this release, triggers supported the following condition
types:

• Buffer usage exceeded a given threshold,
• Buffer usage went under a configurable threshold,
• A session rotation occurred,
• A session rotation completed.

A notify action could be used to send a notification to a third party
applications whenever those conditions were met.

This made it possible, for instance, to disable certain event rules if the
tracing buffers were almost full. It could also be used to wait for session
rotations to be completed to start processing the resulting trace chunk
archives as part of various post-processing trace analyses.

This release introduces a new powerful condition type: event-rule matches.

This type of condition is met when the tracer encounters an event matching the
given even rule. The arguments describing the event rule are the same as those
describing the event rules of the enable-event command.

While this is not intended as a general replacement for the existing
high-throughput tracing facilities, this makes it possible for an application
to wait for a very-specific event to occur and take action whenever it occurs.
The purpose of event-rule matches triggers is to react quickly to an event
without the delay introduced by buffering.

For example, the following command will create a trigger that emits a
notification whenever the 'openat' system call is invoked with the
'/etc/passwd' filename argument.

$ lttng add-trigger
    --condition event-rule-matches
      --type=kernel:syscall
      --name='openat'
    --action notify

New actions were also introduced as part of this release:

• Start session
  This action causes the LTTng session daemon to start tracing for the session
  with the given name. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Stop session
  This action causes the LTTng session daemon to stop tracing for the session
  with the given name. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Rotate session
  This action causes the LTTng session daemon to rotate the session with the
  given name. See lttng-rotate(1) for more information about the session
  rotation concept. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Snapshot session
  This action causes the LTTng session daemon to take a snapshot of the
  session with the given name. See lttng-snapshot(1) for more information
  about the session snapshot concept. If no session with the given name exist
  at the time the condition is met, nothing is done.

These new actions can also be combined together. For instance, the following
trigger will stop my_session, record a snapshot of my_session, and notify
any listening application when /etc/passwd is opened:

$ lttng add-trigger
    --condition event-rule-matches
      --type kernel:syscall
      --name 'openat'
      --filter 'filename == "/etc/passwd"'
    --action stop-session my_session
    --action snapshot-session my_session
    --action notify

For more information, see the following manual pages:

• lttng-add-trigger(1),
• lttng-remove-trigger(1),
• lttng-list-triggers(1).

Notification payload capture

The new event-rule matches condition type also allows 'captures'.
This allow event record and context fields to be captured when an event-rule
matches condition is satisfied.

The captured field values are made available in the evaluation object of the
notifications transmitted to listening applications.

Capture descriptors can be specified using a syntax reminiscent of the one used
by the filter expressions.

The following example will capture a process's name and the 'filename' argument
of all openat() system calls:

$ lttng add-trigger
    --condition event-rule-matches
      --type kernel:syscall
      --name 'openat'
      --capture 'filename'
      --capture '$ctx.procname'
    --action notify

See the lttng-add-trigger(1) manual page for more information.

vtracef and vtracelog (LTTng-UST)

New versions of the tracef() and tracelog() tracing helpers accepting
variable argument lists are introduced as vtracef() and vtracelog().

See the tracef(3) and tracelog(3) manual pages for more information.

Add time namespace context (LTTng-UST and LTTng-modules)

It is now possible to add the time namespace of a process as a context to
channels (time_ns) using the add-context command.

See the time_namespaces(7) manual page for more information.

v2.13.9

About this patch release

2023-01-05 lttng-tools 2.13.9 (National Whipped Cream Day)

• Fix: lttng: poptGetArg doesn't provide string ownership
• Fix: relayd: missing space in trace creation logging statement
• Fix: event field value: assertion fails on empty string
• Build fix: rpath of test libraries results in non-reproducible build
• Fix: never use 'no' in a command variable
• Fix: sessiond: abort called on undefined client command
• Fix: consumer: snapshot: assertion on subsequent snapshot
• Fix: sessiond: instance uuid is not sufficiently unique

Full Changelog: v2.13.8...v2.13.9

About LTTng-tools 2.13

The most notable features of this new release are:

• Event-rule matches condition triggers and new actions, allowing internal
  actions or external monitoring applications to quickly react when kernel
  or user-space instrumentation is hit,
• Notification payload capture, allowing external monitoring applications
  to read elements of the instrumentation payload when instrumentation is
  hit.
• Instrumentation API: vtracef and vtracelog (LTTng-UST),
• User space time namespace context (LTTng-UST and LTTng-modules).

This release is named after "Nordicité", the product of a collaboration between
Champ Libre and Boréale. This farmhouse IPA is brewed with Kveik yeast and
Québec-grown barley, oats and juniper branches. The result is a remarkable
fruity hazy golden IPA that offers a balanced touch of resinous and woodsy
bitterness.

Based on the LTTng project's documented stable releases lifetime, this 2.13
release coincides with the end-of-life of the LTTng 2.11 release series.

Read on for a short description of each of the new features and the
links to this release.

Note on LTTng-UST backward compatibility

• soname major version change
  This release changes the LTTng-UST soname major from 0 to 1.

  The event notifier (triggers using an event-rule-matches condition)
  functionality required a significant rework of public data structures which
  should never have been made public in the first place.

  Bumping the soname major to 1, will require applications and tracepoint
  providers to be rebuilt against an updated LTTng-UST to use it.

  Old applications and tracepoint providers linked against libraries with
  major soname 0 should be able to co-exist on the same system.

• Building probe providers using a C++ compiler requires C++11

• API namespaceing
  The LTTng-UST API is now systematically namespaced under lttng_ust_* (e.g.
  tracepoint() becomes lttng_ust_tracepoint()).

  However, the non-namespaced names are still exposed to maintain API
  compatibility.

Event-rule matches condition and new actions

Expanding the trigger infrastructure and making it usable through the lttng
client was the core focus of this release.

A trigger is an association between a condition and one or more actions. When
the condition associated to a trigger is met, the actions associated to that
trigger are executed. The tracing does not have to be active for the conditions
to be met, and triggers are independent from tracing sessions.

Since their introduction as part of LTTng 2.10, new conditions and actions were
added to make this little-known mechanism more flexible.

For instance, before this release, triggers supported the following condition
types:

• Buffer usage exceeded a given threshold,
• Buffer usage went under a configurable threshold,
• A session rotation occurred,
• A session rotation completed.

A notify action could be used to send a notification to a third party
applications whenever those conditions were met.

This made it possible, for instance, to disable certain event rules if the
tracing buffers were almost full. It could also be used to wait for session
rotations to be completed to start processing the resulting trace chunk
archives as part of various post-processing trace analyses.

This release introduces a new powerful condition type: event-rule matches.

This type of condition is met when the tracer encounters an event matching the
given even rule. The arguments describing the event rule are the same as those
describing the event rules of the enable-event command.

While this is not intended as a general replacement for the existing
high-throughput tracing facilities, this makes it possible for an application
to wait for a very-specific event to occur and take action whenever it occurs.
The purpose of event-rule matches triggers is to react quickly to an event
without the delay introduced by buffering.

For example, the following command will create a trigger that emits a
notification whenever the 'openat' system call is invoked with the
'/etc/passwd' filename argument.

$ lttng add-trigger
    --condition event-rule-matches
      --type=kernel:syscall
      --name='openat'
    --action notify

New actions were also introduced as part of this release:

• Start session
  This action causes the LTTng session daemon to start tracing for the session
  with the given name. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Stop session
  This action causes the LTTng session daemon to stop tracing for the session
  with the given name. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Rotate session
  This action causes the LTTng session daemon to rotate the session with the
  given name. See lttng-rotate(1) for more information about the session
  rotation concept. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Snapshot session
  This action causes the LTTng session daemon to take a snapshot of the
  session with the given name. See lttng-snapshot(1) for more information
  about the session snapshot concept. If no session with the given name exist
  at the time the condition is met, nothing is done.

These new actions can also be combined together. For instance, the following
trigger will stop my_session, record a snapshot of my_session, and notify
any listening application when /etc/passwd is opened:

$ lttng add-trigger
    --condition event-rule-matches
      --type kernel:syscall
      --name 'openat'
      --filter 'filename == "/etc/passwd"'
    --action stop-session my_session
    --action snapshot-session my_session
    --action notify

For more information, see the following manual pages:

• lttng-add-trigger(1),
• lttng-remove-trigger(1),
• lttng-list-triggers(1).

Notification payload capture

The new event-rule matches condition type also allows 'captures'.
This allow event record and context fields to be captured when an event-rule
matches condition is satisfied.

The captured field values are made available in the evaluation object of the
notifications transmitted to listening applications.

Capture descriptors can be specified using a syntax reminiscent of the one used
by the filter expressions.

The following example will capture a process's name and the 'filename' argument
of all openat() system calls:

$ lttng add-trigger
    --condition event-rule-matches
      --type kernel:syscall
      --name 'openat'
      --capture 'filename'
      --capture '$ctx.procname'
    --action notify

See the lttng-add-trigger(1) manual page for more information.

vtracef and vtracelog (LTTng-UST)

New versions of the tracef() and tracelog() tracing helpers accepting
variable argument lists are introduced as vtracef() and vtracelog().

See the tracef(3) and tracelog(3) manual pages for more information.

Add time namespace context (LTTng-UST and LTTng-modules)

It is now possible to add the time namespace of a process as a context to
channels (time_ns) using the add-context command.

See the time_namespaces(7) manual page for more information.

LTTng 2.13.0 - Nordicité

Hi everyone,

Today is the official release of LTTng 2.13 - Nordicité! It is the result of
one year of development from most of the EfficiOS team.

The most notable features of this new release are:

• Event-rule matches condition triggers and new actions, allowing internal
  actions or external monitoring applications to quickly react when kernel
  or user-space instrumentation is hit,
• Notification payload capture, allowing external monitoring applications
  to read elements of the instrumentation payload when instrumentation is
  hit.
• Instrumentation API: vtracef and vtracelog (LTTng-UST),
• User space time namespace context (LTTng-UST and LTTng-modules).

This release is named after "Nordicité", the product of a collaboration between
Champ Libre and Boréale. This farmhouse IPA is brewed with Kveik yeast and
Québec-grown barley, oats and juniper branches. The result is a remarkable
fruity hazy golden IPA that offers a balanced touch of resinous and woodsy
bitterness.

Based on the LTTng project's documented stable releases lifetime, this 2.13
release coincides with the end-of-life of the LTTng 2.11 release series.

Read on for a short description of each of the new features and the
links to this release.

Note on LTTng-UST backward compatibility

• soname major version change
  This release changes the LTTng-UST soname major from 0 to 1.

  The event notifier (triggers using an event-rule-matches condition)
  functionality required a significant rework of public data structures which
  should never have been made public in the first place.

  Bumping the soname major to 1, will require applications and tracepoint
  providers to be rebuilt against an updated LTTng-UST to use it.

  Old applications and tracepoint providers linked against libraries with
  major soname 0 should be able to co-exist on the same system.

• Building probe providers using a C++ compiler requires C++11

• API namespaceing
  The LTTng-UST API is now systematically namespaced under lttng_ust_* (e.g.
  tracepoint() becomes lttng_ust_tracepoint()).

  However, the non-namespaced names are still exposed to maintain API
  compatibility.

Event-rule matches condition and new actions

Expanding the trigger infrastructure and making it usable through the lttng
client was the core focus of this release.

A trigger is an association between a condition and one or more actions. When
the condition associated to a trigger is met, the actions associated to that
trigger are executed. The tracing does not have to be active for the conditions
to be met, and triggers are independent from tracing sessions.

Since their introduction as part of LTTng 2.10, new conditions and actions were
added to make this little-known mechanism more flexible.

For instance, before this release, triggers supported the following condition
types:

• Buffer usage exceeded a given threshold,
• Buffer usage went under a configurable threshold,
• A session rotation occurred,
• A session rotation completed.

A notify action could be used to send a notification to a third party
applications whenever those conditions were met.

This made it possible, for instance, to disable certain event rules if the
tracing buffers were almost full. It could also be used to wait for session
rotations to be completed to start processing the resulting trace chunk
archives as part of various post-processing trace analyses.

This release introduces a new powerful condition type: event-rule matches.

This type of condition is met when the tracer encounters an event matching the
given even rule. The arguments describing the event rule are the same as those
describing the event rules of the enable-event command.

While this is not intended as a general replacement for the existing
high-throughput tracing facilities, this makes it possible for an application
to wait for a very-specific event to occur and take action whenever it occurs.
The purpose of event-rule matches triggers is to react quickly to an event
without the delay introduced by buffering.

For example, the following command will create a trigger that emits a
notification whenever the 'openat' system call is invoked with the
'/etc/passwd' filename argument.

$ lttng add-trigger
    --condition event-rule-matches
      --type=kernel:syscall
      --name='openat'
    --action notify

New actions were also introduced as part of this release:

• Start session
  This action causes the LTTng session daemon to start tracing for the session
  with the given name. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Stop session
  This action causes the LTTng session daemon to stop tracing for the session
  with the given name. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Rotate session
  This action causes the LTTng session daemon to rotate the session with the
  given name. See lttng-rotate(1) for more information about the session
  rotation concept. If no session with the given name exist at the time the
  condition is met, nothing is done.

• Snapshot session
  This action causes the LTTng session daemon to take a snapshot of the
  session with the given name. See lttng-snapshot(1) for more information
  about the session snapshot concept. If no session with the given name exist
  at the time the condition is met, nothing is done.

These new actions can also be combined together. For instance, the following
trigger will stop my_session, record a snapshot of my_session, and notify
any listening application when /etc/passwd is opened:

$ lttng add-trigger
    --condition event-rule-matches
      --type kernel:syscall
      --name 'openat'
      --filter 'filename == "/etc/passwd"'
    --action stop-session my_session
    --action snapshot-session my_session
    --action notify

For more information, see the following manual pages:

• lttng-add-trigger(1),
• lttng-remove-trigger(1),
• lttng-list-triggers(1).

Notification payload capture

The new event-rule matches condition type also allows 'captures'.
This allow event record and context fields to be captured when an event-rule
matches condition is satisfied.

The captured field values are made available in the evaluation object of the
notifications transmitted to listening applications.

Capture descriptors can be specified using a syntax reminiscent of the one used
by the filter expressions.

The following example will capture a process's name and the 'filename' argument
of all openat() system calls:

$ lttng add-trigger
    --condition event-rule-matches
      --type kernel:syscall
      --name 'openat'
      --capture 'filename'
      --capture '$ctx.procname'
    --action notify

See the lttng-add-trigger(1) manual page for more information.

vtracef and vtracelog (LTTng-UST)

New versions of the tracef() and tracelog() tracing helpers accepting
variable argument lists are introduced as vtracef() and vtracelog().

See the tracef(3) and tracelog(3) manual pages for more information.

Add time namespace context (LTTng-UST and LTTng-modules)

It is now possible to add the time namespace of a process as a context to
channels (time_ns) using the add-context command.

See the time_namespaces(7) manual page for more information.

LTTng 2.13-rc3 - Nordicité

Hi everyone,

Today marks the release of the third LTTng 2.13 - Nordicité release candidate.

Have a look at the first release candidate's release notes for an overview of the features introduced in LTTng 2.13.

What's new in LTTng-UST (Linux application tracer) 2.13.0-rc3?

• Removed vtracelog and vtracef from the v0 compatibility API

• Various documentation updates.

See the ChangeLog for the full list of changes.

What's new in LTTng-modules (Linux kernel tracer) 2.13.0-rc3?

• Adapted the instrumentation for 5.14-rc upstream kernels and recent RHEL 8.4 kernels.

See the ChangeLog for the full list of changes.

What's new in LTTng-tools 2.13.0-rc3?

• Usage examples were added to all lttng command man pages.

• Some symbols introduced in liblttng-ctl as part of the 2.13 release were changed:

  • Error query actions are now specified using an lttng_action_path,
  • The event rule API has been namespaced to include the domain name.

• Bumped the minimal Userspace RCU dependency to 0.11.

• Added the Machine Interface (MI) for the CLI's newly introduced trigger commands
  (add-trigger, remove-trigger, list-triggers).

See the ChangeLog for the full list of changes.

LTTng 2.13-rc2 - Nordicité

Release Notes

Updating from Release Candidate 1

If you were already testing the first release candidate of this release, please note that LTTng-UST and LTTng-tools must be updated to this new release candidate (rc2) in locked-step due to breaking ABI changes. Also, note that probe providers must be rebuilt against LTTng-UST 2.13.0-rc2.

Have a look at the first release candidate's release notes for an overview of the features introduced in LTTng 2.13.

What's new in LTTng-UST (Linux application tracer) 2.13.0-rc2?

• Allow explicit tracepoint instance provider name. Allow re-using a tracepoint class from tracepoint instances in other providers.

  This is a localized API-breaking change introduced very early in the 2.13-rc cycle. The LTTNG_UST_TRACEPOINT_EVENT_INSTANCE macro newly introduced in LTTng-UST 2.13-rc1 now takes an extra "template_provider" argument.

  This changes the ABI exposed by the probe providers, so bump the probe provider major version, and make newer LTTng-UST reject old incompatible major.

  Probe providers built against LTTng-UST 2.13-rc1 must be rebuilt against LTTng-UST 2.13-rc2 to be traced.

• Add probe descriptor field to enumeration and event class

  Also breaks the ABI exposed by probe providers (provider major version already bumped for rc2). This enables additional validation of probe provider compatibility when using event class and enumerations across providers.

• Validate provider version for event class.

• Validate provider version for event enumeration field types.

• Introduce api0 and api1 compilation tests. This tests whether the API 0 compatibility API works. See ust-api-compat.h for details.

• Fix Java (JUL and log4j) application context segmentation fault and leaks.

• liblttng-ust-ctl: implement SIGBUS handling:

  This fixes a long-standing DoS where an unprivileged application could cause a SIGBUS in the consumer daemon by truncating the shared memory files it receives from the session daemon. This requires that LTTng-tools be upgraded to 2.13-rc2 in locked-step with LTTng-UST 2.13-rc2.

• Restore communication protocol backward compatibility with LTTng-UST 2.12

  • libLTTng-UST-ctl: keep using LTTng-UST-sock-8 and LTTng-UST-wait-8 filename

    LTTng-UST 2.13-rc1 erroneously changed the file names of those socket and pipe files from [...]-8 to [...]-9, even though we intended to keep protocol compatibility between LTTng-UST 2.12 and LTTng-tools/ust 2.13. This changes reverts back to the prior filenames to make sure LTTng-UST 2.12 applications can be traced by a 2.13 LTTng-tools using 2.13 LTTng-UST.

  • Fix: add missing fields in struct lttng_ust_abi_channel_config

    Three fields were removed by mistake from this structure which is ABI between the consumer daemon and applications traced with LTTng-UST. Add them back to allow tracing 2.12 LTTng-UST applications with a 2.13 LTTng-tools using 2.13 LTTng-UST.

• Fix LTTng-UST communication protocol variable-length data handling

  When mixing older (e.g. 2.12) and newer versions of LTTng-UST talking to each other between application and LTTng-tools, situations where commands are unknown lead to out-of-sync communication protocol.

  Change the protocol for commands new to LTTng-UST 2.13 so they acknowledge the fixed-size commands before the variable-sized data is sent, so the communication protocol is kept in sync even when unknown command error happens. This also happens for more ancient commands released in prior versions for which we cannot change the protocol. Shutdown the socket from the session daemon side when we detect such errors, because there is no point in trying to communicate further on an out-of-sync socket.

• Detect mixed liblttng-ust and liblttng-ust-tracepoint .so.0 and .so.1 libraries

  Detect and report with a new "critical" logging message whether .so.0 and .so.1 tracer libraries are loaded within a single process, which is an unsupported use-case. Note that the LTTNG_UST_DEBUG environment variable must be set to allow users to observe those critical messages.

  We also introduce a new LTTNG_UST_ABORT_ON_CRITICAL environment variable to allow the tracer to abort() the process whenever a critical logging statement is encountered. This is meant to facilitate discovering problematic deployments where an application indirectly loads different soname major numbers of LTTng-UST through its instrumented shared libraries.

See the ChangeLog for the full list of changes.

What's new in LTTng-modules (Linux kernel tracer) 2.13.0-rc2?

• There are refactoring commits that were introduced after 2.13-rc1 to align the code of LTTng-modules with the refactoring that was done in LTTng-UST prior to 2.13-rc1. Given that LTTng-modules does not expose its kernel APIs to external users, it was possible to do this refactoring even after rc1.

• Compiling the system call tracing code has typically been a resource-heavy endeavour both in terms of memory usage (~1G memory required) and CPU time. It has been split into smaller compile units to allow parallel compilation and lessen the memory requirements on the compiler.

• Bitwise enumerations are now disabled by default. Only compiled in with make CONFIG_LTTNG_EXPERIMENTAL_BITWISE_ENUM=y.

• Fix: increment buffer offset when failing to copy from user-space.

• Implement ring buffer Pascal string copy: allows taking C-strings as inputs and record them to ctf_array_text and ctf_sequence_text fields (which are Pascal strings) without causing out-of-bound reads of the C-strings of those are smaller than the size reserved for the pascal strings. This mirrors the behaviour introduced in LTTng-UST 2.13-rc1.

• Support for 5.12 Linux kernels,

• Support recent stable kernel branches 4.14, 4.19, 5.4,

• Support for newer Ubuntu 4.15, 5.4, and RHEL 8.2/8.3 kernels,

See the ChangeLog for the full list of changes:

What's new in LTTng-tools 2.13.0-rc2?

• The new features introduced in the 2.13 release are now fully documented as part of the installed man pages. A number of improvements were also made to the existing documentation such as adding examples for the lttng-add-context command.

• lttng client messages referring to "event rule hit" conditions have been changed to use the correct "event-rule matches" condition name.

• An example was added under doc/examples/trigger-condition-event-matches. It contains an application instrumented with LTTng-UST, a script that adds triggers to notify external clients whenever the application's events occur, and a notification client which consumes the resulting notification along with any captured payload.

• A bug resulting in the production of erroneous path hierarchies when a snapshot is streamed to a relay daemon was fixed. This bug resulted in duplicated domain names being present in the final path of a snapshot (ust/ust or kernel/kernel) which could cause some automation scripts to fail.

• A bug that caused some trace data to be invisible to live viewers has been fixed.

  See this thread for more details.

• A number of bugs that could result, under rare circumstances, in a hang when using the lttng-clear command against a session that contains kernel channels was fixed.

• Miscellaneous stability improvements.

See the ChangeLog for the full list of changes.

LTTng 2.13-rc1 - Nordicité

Release Notes

This release is named after Nordicité, the product of a collaboration between Champ Libre and Boréale. This farmhouse IPA is brewed with Kveik yeast and Québec-grown barley, oats and juniper branches. The result is a remarkable fruity hazy golden IPA that offers a balanced touch of resinous and woodsy bitterness.

The most notable features of this new release are:

• Event-rule matches condition triggers and new actions,
• Notification payload capture,
• vtracef and vtracelog (LTTng-UST),
• Add user space time namespace context (LTTng-UST and LTTng-modules).

Read on for a short description of each of the new features and the links to this release.

Note on LTTng-UST backward compatibility

• soname major version change

  This release changes the LTTng-UST soname major from 0 to 1.

  The event notifier (triggers using an event-rule-matches condition) functionality required a significant rework of public data structures which should never have been made public in the first place.

  Bumping the soname major to 1 will require applications and tracepoint providers to be rebuilt against an updated LTTng-UST.

  Old applications and tracepoint providers linked against libraries with the 0 major soname can co-exist on the same system.

• Building probe providers using a C++ compiler requires C++11

• API namespace-ing

  The LTTng-UST API is now systematically namespaced under lttng_ust_* (e.g tracepoint() becomes lttng_ust_tracepoint()).

  However, the un-namespaced names are still exposed to maintain API compatibility.

What's New?

Event-rule matches condition trigger and new actions

Expanding the trigger infrastructure and making it usable through the lttng client was the core focus of this release.

A trigger is an association between a condition and one or more actions. When the condition associated to a trigger is met, the actions associated to that trigger are executed. The tracing does not have to be active for the conditions to be met, and triggers are independent from tracing sessions.

Since their introduction as part of LTTng 2.10, new conditions and actions were added to make this little-known mechanism more flexible.

For instance, before this release, triggers supported the following condition
types:

• Buffer usage exceeded a given threshold,
• Buffer usage went under a configurable threshold,
• A session rotation occurred,
• A session rotation completed.

A notify action could be used to send a notification to a third party applications whenever those conditions were met.

This made it possible, for instance, to disable certain event rules if the tracing buffers were almost full. It could also be used to wait for session
rotations to be completed to start processing the resulting trace chunk archives as part of various post-processing trace analyses.

This release introduces a new powerful condition type: event-rule matches.

This type of condition is met when the tracer encounters an event matching the given even rule. The arguments describing the event rule are the same as those describing the event rules of the enable-event command.

While this is not intended as a general replacement for the existing high-throughput tracing facilities, this makes it possible for an application
to wait for a very-specific event to occur and take action whenever it occurs.
The purpose of event-rule matches triggers is to react quickly to an event without the delay introduced by buffering.

For example, the following command will create a trigger that emits a notification whenever the openat system call is invoked with the
/etc/passwd filename argument.

$ lttng add-trigger
    --condition event-rule-matches
      --domain=kernel
      --type=syscall
      --name="openat"
    --action notify

New actions were also introduced as part of this release:

• Start session

  This action causes the LTTng session daemon to start tracing for the session with the given name. If no session with the given name exist at the time the condition is met, nothing is done.

• Stop session

  This action causes the LTTng session daemon to stop tracing for the session with the given name. If no session with the given name exist at the time the condition is met, nothing is done.

• Rotate session

  This action causes the LTTng session daemon to rotate the session with the given name. See lttng-rotate(1) for more information about the session rotation concept. If no session with the given name exist at the time the condition is met, nothing is done.

• Snapshot session

  This action causes the LTTng session daemon to take a snapshot of the
  session with the given name. See lttng-snapshot(1) for more information
  about the session snapshot concept. If no session with the given name exist
  at the time the condition is met, nothing is done.

These new actions can also be combined together. For instance, the following
trigger will stop my_session, record a snapshot of my_session, and notify
any listening application when /etc/passwd is opened:

$ lttng add-trigger
    --condition event-rule-matches
      --domain kernel
      --type syscall
      --name "openat"
      --filter 'filename == "/etc/passwd"'
    --action stop-session my_session
    --action snapshot-session my_session
    --action notify

For more information, see the following manual pages:

• lttng-add-trigger(1),
• lttng-remove-trigger(1),
• lttng-list-triggers(1).

Notification payload capture

The new event-rule matches condition type also supports captures. Captures allow event record and context fields to be captured when an event-rule matches condition is satisfied.

The captured field values are made available in the evaluation object of the notifications transmitted to listening applications.

Capture descriptors can be specified using a syntax reminiscent of the one used by the filter expressions.

The following example will capture a process's name and the filename argument of all openat() system calls:

$ lttng add-trigger
    --condition event-rule-matches
      --domain kernel
      --type syscall
      --name "openat"
      --capture "filename"
      --capture "$ctx.procname"
    --action notify

See the lttng-add-trigger(1) manual page for more information.

vtracef and vtracelog (LTTng-UST)

New versions of the tracef() and tracelog() tracing helpers accepting variable argument lists are introduced as vtracef() and vtracelog().

See the tracef(3) and tracelog(3) manual pages for more information.

Add time namespace context (LTTng-UST and LTTng-modules)

It is now possible to add the time namespace of a process as a context to channels (time_ns) using the add-context command.

See the time_namespaces(7) manual page for more information.

LTTng 2.12.0 - (Ta) Meilleure

Release Notes

This release is named after Ta Meilleure, a Northeast IPA beer brewed by Lagabière. Translating to "Your best one", this beer gives out strong aromas of passion fruit, lemon, and peaches. Tastewise, expect a lot of fruit, a creamy texture, and a smooth lingering hop bitterness.

The most notable features of this new release are:

• session clearing,
• uid and gid tracking,
• file descriptor pooling (relay daemon),
• per-session grouping (relay daemon),
• working directory override (relay daemon),
• new network reception entry/exit tracepoints (LTTng-modules),
• statedump of interrupt threads (LTTng-modules),
• statedump of x86 CPU topology (LTTng-modules),
• new product UUID environment field (LTTng-modules).

Read on for a short description of each of these features and the links to this release!

Session clearing

You can use the new lttng-clear command to clear the contents of one or more tracing sessions.

In essence, this new feature allows you to prune the content of long-running sessions without destroying and reconfiguring them. This is especially useful to clear a session's tracing data between attempts to reproduce a problem.

Clearing a tracing session deletes the contents of the tracing buffers and all local or streamed trace data on a remote peer. Note that an lttng-relayd daemon can be configured to disallow clear operations using the LTTNG_RELAYD_DISALLOW_CLEAR environment variable.

If a session is configured in snapshot mode, only the tracing buffers are cleared.

If a session is configured in live mode, any attached client that is lagging behind will finish the consumption of its current trace data packets and jump forward in time to events generated after the beginning of the clear command.

uid and gid tracking

The existing lttng-track command has been expanded to support uid and gid tracking.

By default, a tracing session tracks all applications and users, following LTTng's permission model.
However, this new options allows you to restrict which users and groups are tracked by both the user space and kernel tracers.

In previous versions of LTTng, it was effectively possible to filter on the basis of uids and gids using the --filter mechanism. However, this dedicated filtering mechanism is both more efficient in terms of tracing overhead, but also prevents the creation of tracing buffers for users and groups which are not tracked.

Overall, this results in far less memory consumption by the user space tracer on systems which have multiple active users.

File descriptor pooling (relay daemon)

A number of users have reported having encountered file descriptor exhaustion issues when using the relay daemon to serve a large number of consumers or live clients.

The current on-disk CTF representation used by LTTng (and expected by a number of viewers) uses one file per CPU, per channel, to organize traces. This causes the default RLIMIT_NOFILE value (1024 on many systems) to be reached easily, especially when tracing systems with a large number of cores.

In order to alleviate this problem, the new --fd-pool-size option allows you to specify a maximal number of simultaneously opened file descriptors (using the soft RLIMIT_NOFILE resource limit of the process by default). This is meant as a work-around for users who can't bump the system-limit because of permission restrictions.

As its name indicates, this option causes the relay daemon to maintain a pool (or cache) of open file descriptors which are re-purposed as needed. The most recently used files' file descriptors are kept open and only closed as the --fd-pool-size limit is reached, keeping the number of simultaneously opened file descriptors under the user-specified limit.

Note that setting this value too low can degrade the performance of the relay daemon.

Per-session grouping (relay daemon)

By default, the relay daemon writes the traces under a predefined directory hierarchy:
$LTTNG_HOME/lttng-traces/HOSTNAME/SESSION/DOMAIN where

• HOSTNAME is the remote hostname,
• SESSION is the full session name,
• DOMAIN is the tracing domain (ust or kernel),

Using the new relay daemon --group-output-by-session option, you can now change this hierarchy to group traces by sessions, rather than by hostname:
$LTTNG_HOME/lttng-traces/SESSION/HOST/DOMAIN.

This proves especially useful if you are tracing a number of hosts (with different hostnames) which share the same session name as part of their configuration. Hence, a descriptive session name (e.g. connection-hang) can be used across a fleet of machines streaming to a given relay daemon.

Note that the default behaviour can be explicitly specified using the --group-output-by-host option.

Working directory override (relay daemon)

This small quality of life feature allows you to override the working directory of the relay daemon using the daemon's launch options (-w PATH/--working-directory=PATH).

New network reception entry/exit tracepoints (LTTng-modules)

New instrumentation hooks were added to the kernel tracer in order to trace the entry and exit tracepoints of the network reception code paths of the Linux kernel.

You can use those tracepoints to identify the bounds of a network reception and link the events that happen in the interim (e.g. wakeups) to a specific network reception instance. Those tracepoints can also be used to analyse the network stack's latency.

Statedump of interrupt threads (LTTng-modules)

Threaded IRQs have an associated thread field in the irqaction structure which specifies the process to wake up when the IRQ happens. This field is now extracted as part of the lttng_statedump_interrupt statedump tracepoint.

You can use this information to know which processes handle the various IRQs. It is also possible to associate the events occurring in the context of those processes to their respective IRQ.

Statedump of x86 CPU topology (LTTng-modules)

A new lttng_statedump_cpu_topology tracepoint has been added to extract the active CPU/NUMA topology. You can use this information to know which CPUs are SMT siblings or part of the same socket. For the time being, only x86 is supported since all architectures describe their topologies differently.

The architecture field is statically defined and should be present for all architecture implementations. Hence, it is possible for analysis tools to anticipate the event's layout.

Example output:

lttng_statedump_cpu_topology: { cpu_id = 3 }, { architecture = "x86", cpu_id = 0, vendor = "GenuineIntel", family = 6, model = 142, model_name = "Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz", physical_id = 0, core_id = 0, cores = 2 }

New product UUID environment field (LTTng-modules)

The product UUID, taken from the DMI system information, is now saved as part of the kernel traces' environment fields as the product_uuid. You can use this field to uniquely identify a machine (virtual or physical) in order to correlate traces gathered on multiple virtual machines.

LTTng 2.12-rc1 - (Ta) Meilleure

Release Notes

This release is named after Ta Meilleure, a Northeast IPA beer brewed by Lagabière. Translating to "Your best one", this beer gives out strong aromas of passion fruit, lemon, and peaches. Tastewise, expect a lot of fruit, a creamy texture, and a smooth lingering hop bitterness.

The most notable features of this new release are:

• session clearing,
• uid and gid tracking,
• file descriptor pooling (relay daemon),
• per-session grouping (relay daemon),
• working directory override (relay daemon),
• new network reception entry/exit tracepoints (LTTng-modules),
• statedump of interrupt threads (LTTng-modules),
• statedump of x86 CPU topology (LTTng-modules),
• new product UUID environment field (LTTng-modules).

Read on for a short description of each of these features and
the links to this release!

Session clearing

You can use the new lttng-clear command to clear the contents of one or more tracing sessions.

In essence, this new feature allows you to prune the content of long-running sessions without destroying and reconfiguring them. This is especially useful to clear a session's tracing data between attempts to reproduce a problem.

Clearing a tracing session deletes the contents of the tracing buffers and all local or streamed trace data on a remote peer. Note that an lttng-relayd daemon can be configured to disallow clear operations using the LTTNG_RELAYD_DISALLOW_CLEAR environment variable.

If a session is configured in snapshot mode, only the tracing buffers are cleared.

If a session is configured in live mode, any attached client that is lagging behind will finish the consumption of its current trace data packets and jump forward in time to events generated after the beginning of the clear command.

uid and gid tracking

The existing lttng-track command has been expanded to support uid and gid tracking.

By default, a tracing session tracks all applications and users, following LTTng's permission model.
However, this new options allows you to restrict which users and groups are tracked by both the user space and kernel tracers.

In previous versions of LTTng, it was effectively possible to filter on the basis of uids and gids using the --filter mechanism. However, this dedicated filtering mechanism is both more efficient in terms of tracing overhead, but also prevents the creation of tracing buffers for users and groups which are not tracked.

Overall, this results in far less memory consumption by the user space tracer on systems which have multiple active users.

File descriptor pooling (relay daemon)

A number of users have reported having encountered file descriptor exhaustion issues when using the relay daemon to serve a large number of consumers or live clients.

The current on-disk CTF representation used by LTTng (and expected by a number of viewers) uses one file per CPU, per channel, to organize traces. This causes the default RLIMIT_NOFILE value (1024 on many systems) to be reached easily, especially when tracing systems with a large number of cores.

In order to alleviate this problem, the new --fd-pool-size option allows you to specify a maximal number of simultaneously opened file descriptors (using the soft RLIMIT_NOFILE resource limit of the process by default). This is meant as a work-around for users who can't bump the system-limit because of permission restrictions.

As its name indicates, this option causes the relay daemon to maintain a pool (or cache) of open file descriptors which are re-purposed as needed. The most recently used files' file descriptors are kept open and only closed as the --fd-pool-size limit is reached, keeping the number of simultaneously opened file descriptors under the user-specified limit.

Note that setting this value too low can degrade the performance of the relay daemon.

Per-session grouping (relay daemon)

By default, the relay daemon writes the traces under a predefined directory hierarchy:
$LTTNG_HOME/lttng-traces/HOSTNAME/SESSION/DOMAIN where

• HOSTNAME is the remote hostname,
• SESSION is the full session name,
• DOMAIN is the tracing domain (ust or kernel),

Using the new relay daemon --group-output-by-session option, you can now change this hierarchy to group traces by sessions, rather than by hostname:
$LTTNG_HOME/lttng-traces/SESSION/HOST/DOMAIN.

This proves especially useful if you are tracing a number of hosts (with different hostnames) which share the same session name as part of their configuration. Hence, a descriptive session name (e.g. connection-hang) can be used across a fleet of machines streaming to a given relay daemon.

Note that the default behaviour can be explicitly specified using the --group-output-by-host option.

Working directory override (relay daemon)

This small quality of life feature allows you to override the working directory of the relay daemon using the daemon's launch options (-w PATH/--working-directory=PATH).

New network reception entry/exit tracepoints (LTTng-modules)

New instrumentation hooks were added to the kernel tracer in order to trace the entry and exit tracepoints of the network reception code paths of the Linux kernel.

You can use those tracepoints to identify the bounds of a network reception and link the events that happen in the interim (e.g. wakeups) to a specific network reception instance. Those tracepoints can also be used to analyse the network stack's latency.

Statedump of interrupt threads (LTTng-modules)

Threaded IRQs have an associated thread field in the irqaction structure which specifies the process to wake up when the IRQ happens. This field is now extracted as part of the lttng_statedump_interrupt statedump tracepoint.

You can use this information to know which processes handle the various IRQs. It is also possible to associate the events occurring in the context of those processes to their respective IRQ.

Statedump of x86 CPU topology (LTTng-modules)

A new lttng_statedump_cpu_topology tracepoint has been added to extract the active CPU/NUMA topology. You can use this information to know which CPUs are SMT siblings or part of the same socket. For the time being, only x86 is supported since all architectures describe their topologies differently.

The architecture field is statically defined and should be present for all architecture implementations. Hence, it is possible for analysis tools to anticipate the event's layout.

Example output:

lttng_statedump_cpu_topology: { cpu_id = 3 }, { architecture = "x86", cpu_id = 0, vendor = "GenuineIntel", family = 6, model = 142, model_name = "Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz", physical_id = 0, core_id = 0, cores = 2 }

New product UUID environment field (LTTng-modules)

The product UUID, taken from the DMI system information, is now saved as part of the kernel traces' environment fields as the product_uuid. You can use this field to uniquely identify a machine (virtual or physical) in order to correlate traces gathered on multiple virtual machines.

LTTng 2.11.0 - Lafontaine

We're happy to announce the release of LTTng 2.11 "Lafontaine".

This release is named after a modern Saison beer from Montréal's Oshlag microbrewery. It is a refreshing, zesty, rice beer with hints of fruit and spices. Some even say it makes for a great Somaek when mixed with Chamisul Soju, not that we've tried!

Lafontaine is also a tongue-in-cheek reference to a water leak that affected EfficiOS's offices during the development of this release.

The most notable features of this new release are:

• Session rotation,
• Dynamic tracing of user-space (LTTng-modules),
• Support for arrays and bitwise binary operators in filters,
• User and kernel space call stack capture from the LTTng-modules kernel tracer (LTTng-modules),
• Improved networking performance of the relay daemon,
• Take NUMA configuration into account for UST buffer allocation (LTTng-UST),
• Support unloading of probe providers (dlclose) (LTTng-UST).

See the LTTng Blog for the complete announcement.