Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flow metrics: provide lazy-initiazed implementation #3

Draft
wants to merge 4 commits into
base: flow-metrics-extended
Choose a base branch
from
Draft

Flow metrics: provide lazy-initiazed implementation #3

wants to merge 4 commits into from

Commits on Sep 29, 2022

  1. flow metrics: extract to interface, sharable-comon base, and implemen… 

    …tation

In preparation of landing an additional implementation of FlowMetric, we
shuffle the current parts net-unchanged to provide interfaces for `FlowMetric`
and `FlowCapture`, along with a sharable-common `BaseFlowMetric`, and move
our initial implementation to a new `SimpleFlowMetric`, accessible only
through a static factory method on our new `FlowMetric` interface.
    @yaauie
    yaauie committed Sep 29, 2022
    Configuration menu
    Copy the full SHA
    620dcfc View commit details
    Browse the repository at this point in the history

  2. flow rates: limit precision for readability

    @yaauie
    yaauie committed Sep 29, 2022
    Configuration menu
    Copy the full SHA
    f1d3ebe View commit details
    Browse the repository at this point in the history

  3. flow metrics: introduce policy-driven extended windows implementation 

    The new ExtendedFlowMetric is an alternate implementation of the FlowMetric
introduced in Logstash 8.5.0 that is capable of producing windoes for a set of
policies, which dictate the desired retention for the rate along with a
desired resolution.

 - `current`: 10s retention, 1s resolution [*]
 - `last_1_minute`: one minute retention, at 3s resolution [*]
 - `last_5_minutes`: five minutes retention, at 15s resolution
 - `last_15_minutes`: fifteen minutes retention, at 30s resolution
 - `last_1_hour`: one hour retention, at 60s resolution
 - `last_24_hours`: one day retention at 15 minute resolution

A given series may report a range for slightly longer than its configured
retention period, up to the either the series' configured resolution or
our capture rate (currently ~5s), whichever is greater. This approach
allows us to retain sufficient data-points to present meaningful rolling
averages while ensuring that our memory footprint is bounded.

When recording these captures, we first stage the newest capture, and then
promote the previously-staged caputure to the tail of a linked list IFF
the gap between our new capture and the newest promoted capture is larger
than our desired resolution.

When _reading_ these rates, we compact the head of that linked list forward
in time as far as possible without crossing the desired retention barrier,
at which point the head points to the youngest record that is old enough
to satisfy the period for the series.

We also occesionally compact the head during writes, but only if the head
is significantly out-of-date relative to the allowed retention.

As implemented here, this extended flow rates are on by default, but can be
disabled by setting the JVM system property `-Dlogstash.flowMetric=simple`
    @yaauie
    yaauie committed Sep 29, 2022
    Configuration menu
    Copy the full SHA
    29bd71b View commit details
    Browse the repository at this point in the history

  4. flow metrics: provide lazy-initiazed implementation

    @yaauie
    yaauie committed Sep 29, 2022
    Configuration menu
    Copy the full SHA
    8faf033 View commit details
    Browse the repository at this point in the history