Changes for new metrics endpoint

[reshmabidikar]

No description provided.

[sbrossie]

We need spend more time on this doc, so I did a pass. I suggest you start incorporating those comments, and then we will likely need at least another pass.

It's important to document it well not just for API users but also down the line for when this is integrated into the Aviate UI to provide the necessary context to make good use of it.

[sbrossie]

Let's remove the SECOND, we build these metrics once per minute on average so SECOND is not available. I am not sure why we have have it in this enum.

Also default is MINUTE.

[reshmabidikar]

Addressed in 823e174

[sbrossie]

Provide some context on what this endpoint is about. Example:

This endpoint provides a list of metrics that can be used to assess the health of the system and gain visibility into it. The metrics are computed by the aviate plugin when the com.killbill.billing.plugin.aviate.enableHealthReporter is set to true. Each metric is a timeseries that can be analyzed and/or displayed (Aviate UI will eventually provide the screens to render such metrics).

Some metrics are global to the deployment (i.e independent on the number of Kill Bill nodes/containers deployed) and some others are per-node - in which case the host parameter can be used to return metrics for a specific node.

@pierre Should we change api to be consistent and either talk about host (current query param) or nodeName in the json response?

[pierre]

Consistency is good: maybe changing the query to nodeName is easier?

[reshmabidikar]

Just a thought: we also have a parameter called name for the metric name. To avoid confusion should we also change name to metricName?

[sbrossie]

Yes see https://github.com/killbill/killbill-aviate-plugin/pull/325

Although the form is longer, we already use node_name in several places like here and here

[reshmabidikar]

Addressed in 823e174

[sbrossie]

Provide details on the format. Explain that if omitted this is since beginning of time - since aviate plugin started to compute metrics.

[reshmabidikar]

Addressed in 823e174

[sbrossie]

Same remark that to

[reshmabidikar]

Addressed in 823e174

[sbrossie]

1. Provide example request with host
2. Explain if not provided all nodes are returned for "per node" metrics
3. Provide similar explains as in name: "Multiple metrics can be specified..."
4. Lets wait and agree os resolution to change align on either host or nodeName - see question above

[reshmabidikar]

Addressed in 823e174

[sbrossie]

I can't see in this PR the 'model' being defined, but regardless we need to explain what these fields are, and in particular since we expose eventGroup and sampleKind explain what those are and their values (e.g. sampleKind={gauge, meter, timer}). This can happen in a subsequent PR.

[reshmabidikar]

Models are already documented in the doc. See this section. We have a SamplesForSampleKindAndHost model. But yes, the fields in this model need to be explained in detail (which is why I had filed #185 in the first place). We should also rename this to MetricResult as discussed on Slack.

[reshmabidikar]

I will address this is a separate PR.

[reshmabidikar]

Opened #329

[sbrossie]

Can we create some sections for the different types of metrics and provide a small overview and descriptions. Any formatting/indenting to make it easier to read is best - just providing text below for input:

1. Let's not talk about 'published' name and instead provide the list of metrics
2. Provide intro
   Below is the list of published metric names. Any of these can be specified as the value for the name parameter.

Queue Metrics

Kill Bill has its own internal queues used to dispatch events. Events that are dispatched right away as a result of some internal state being created or updated are called bus events - e.g. a subscription created event is generated as a result of creating a new subscritpion. Events that are scheduled to be dispatched in the future are called notifications - e.g. invoice scheduled on a periodic basis matching account settings and plan billing periods.

The health of these internal queues is critical to maintaining correct functionning of the system and the following metrics can be used to assess the health of theses queues. Note that such metrics are global so the nodeName will be empty.

• queue.bus.late: This is a counter (gauge) shows how many unprocessed bus events we have at time 't'. This number should be close to 0.
• queue.bus.incoming: The is rate of incoming requests at time t. The default granularity is the minute but result can be aggregated based on the granularity parameter from the call - e.g hourly incoming rate.
• queue.bus.processing: This is an estimation of the time in mSec that was used to process the bus event. These values may become incorrect once we have late bus entries.

Similar metrics exist for the notifications with similar definitions:

• queue.notifications.late:
• queue.notifications.incoming
• queue.notifications.processing

Logs

Kill Bill is configured to output its internal logs as specified by the logback.xml deployed configuration. The aviate plugin running on each node will extract important information from the logs and compute some metrics to highlight potential issues with warn and error logs that have happened through time. These metrics are computed per host and because they are meter they will each provide different data points:

• count: Monotonic increasing value since last reboot.
• {one_minute|five_minute|fifteen_minute}_rate: Rate of logs through a window of time
• mean_rate: Mean rate since last reboot.

The metrics are:

• logs.rates.warn
• logs.rates.error

Servlet Responses

Those metrics provide visibility into any of the endpoints exposed by the system, either from Kill Bill core (/1.0/kb or any plugins exposing endpoints). These metrics are computed per host and because they are meter they will each provide different data points, count, {one_minute|five_minute|fifteen_minute}_rate, mean_rate.

• servlets.responses.ok
• servlets.responses.created
• servlets.responses.badRequest
• servlets.responses.noContent
• servlets.responses.notFound
• servlets.responses.serverError
• servlets.responses.other

Database Connection Pools

Kill Bill uses 3 different database connection pool, both main and shiro are internal connection pool from Kill Bill core. The osgi connection pool is used by the plugins running on top of Kill Bill platform for any database calls. These metrics are computed per host`.

The following list are counters (gauge)

• main.pool.TotalConnections: Total (created) connections at time 't'
• main.pool.ActiveConnections: Active (in use) connections at time 't'
• main.pool.IdleConnections: Idel connections at time 't'

The same metrics exist for the other pools:

• osgi.pool.TotalConnections
• osgi.pool.ActiveConnections
• osgi.pool.IdleConnections
• shiro.pool.TotalConnections
• shiro.pool.ActiveConnections
• shiro.pool.IdleConnections

We also private the wait time as a distibution metrics (timer):

• main.pool.Wait
• osgi.pool.Wait
• shiro.pool.Wait

[reshmabidikar]

I'm wondering if we should include this information in the health feature doc? I'm in the process of including some examples of using the diagnostic endpoint for https://github.com/killbill/technical-support/issues/189.

[sbrossie]

Ok, that 's fine. We just need a place where our metrics are documented and folks understand what hey are looking at - all the info above is required somewhere.

Besides I moved your technical-support task under aviate plugin - I'd rather aviate all tasks related in the plugin (code, doc, ...) at the same place. Transferred task is: https://github.com/killbill/killbill-aviate-plugin/issues/329

Could you do a first PR just for the metrics related to this end point (the task talks about /v1/health/diagnostic ... which is useful but not directly relevant for our upcoming Capricorn milestones)?

[reshmabidikar]

Opened killbill/killbill-docs#591