Review feedback

- Make a queue per tracked kstat target, so that noisy producers don't
  impact samples from the other targets.
- Add overview doc to `oximeter_instruments::kstat`, and make most
  docstrings public for visibility.
- Delegate future impl to `Sleep` in `YieldIdAfter`, rather than
  spawning a new task.
- Don't fork to get hostname on every call, only when needed.
- Fixup to naming for test Etherstubs, to avoid invalid names.

oximeter/instruments/src/kstat/link.rs

@@ -231,7 +231,7 @@ mod tests {
     use crate::kstat::TargetStatus;
     use kstat_rs::Ctl;
     use oximeter::Producer;
-    use rand::distributions::Alphanumeric;
+    use rand::distributions::Uniform;
     use rand::Rng;
     use slog::info;
     use slog::Drain;
@@ -269,8 +269,8 @@ mod tests {
             let name = format!(
                 "kstest{}0",
                 rand::thread_rng()
-                    .sample_iter(&Alphanumeric)
-                    .take(3)
+                    .sample_iter(Uniform::new('a', 'z'))
+                    .take(5)
                     .map(char::from)
                     .collect::<String>(),
             );

oximeter/instruments/src/kstat/mod.rs

@@ -2,7 +2,74 @@
 // License, v. 2.0. If a copy of the MPL was not distributed with this
 // file, You can obtain one at https://mozilla.org/MPL/2.0/.
 
+// Copyright 2023 Oxide Computer Company
+
 //! Types for publishing kernel statistics via oximeter.
+//!
+//! # illumos kernel statistics
+//!
+//! illumos defines a generic framework for tracking statistics throughout the
+//! OS, called `kstat`. These statistics are generally defined by the kernel or
+//! device drivers, and made available to userspace for reading through
+//! `libkstat`.
+//!
+//! Kernel statistics are defined by a 4-tuple of names, canonically separated
+//! with a ":". For example, the `cpu:0:vm:pgin` kstat tracks the number of
+//! memory pages paged in for CPU 0. In this case, the kstat is tracked by a
+//! u64, though several data types are supported.
+//!
+//! This module uses the `kstat-rs` crate, which is a fairly low-level wrapper
+//! around `libkstat`. For the purposes of this module, most folks will be
+//! interested in the types [`kstat_rs::Kstat`], [`kstat_rs::Data`], and
+//! [`kstat_rs::NamedData`].
+//!
+//! # Oximeter
+//!
+//! Oximeter is the Oxide control plane component which collects telemetry from
+//! other parts of the system, called _producers_. Statistics are defined using
+//! a _target_, which is an object about which statistics are collected, and
+//! _metrics_, which are the actual measurements about a target. As an example,
+//! a target might be an NVMe drive on the rack, and a metric might be its
+//! temperature, or the estimated remaining drive lifetime.
+//!
+//! Targets and metrics are encapsulated by traits of the same name. Using
+//! these, producers can generate timestamped [`Sample`]s, which the `oximeter`
+//! collector program pulls at regular intervals for storage in the timeseries
+//! database.
+//!
+//! # What does this mod do?
+//!
+//! This module is intended to connect illumos kstats with oximeter. Developers
+//! can use this to define a mapping betweeen one or more kstats, and an
+//! oximeter `Target` and `Metric`. This mapping is encapsulated in the
+//! [`KstatTarget`] trait, which extends the `Target` trait itself.
+//!
+//! To implement the trait, developers register their interest in a particular
+//! [`Kstat`] through the [`KstatTarget::interested()`] method. They then
+//! describe how to generate any number of [`Sample`]s from that set of kstats,
+//! through the [`KstatTarget::to_samples()`] method.
+//!
+//! # The [`KstatSampler`]
+//!
+//! Most folks will instantiate a [`KstatSampler`], which manages any number of
+//! tracked `KstatTarget`s. Users can register their implementation of
+//! `KstatTarget` with the sampler, and it will periodically generate samples
+//! from it, converting the "interesting" kstats into `Sample`s.
+//!
+//! # Intervals and expiration
+//!
+//! When users register a target for sampling, they are required to include
+//! details about how often their target should be sampled, and what to do if we
+//! cannot produce samples due to an error. These two are captured in the
+//! [`CollectionDetails`] type.
+//!
+//! After a configurable period of errors (expressed in either consecutive error
+//! counts or a duration of concecutive errors), a target is _expired_, and will
+//! no longer be collected from. A target's status may be queried with the
+//! [`KstatSampler::target_status()`] method, which will inform the caller if
+//! the target has expired. In this case, users can re-register a target, which
+//! will replace the expired one, generating new samples (assuming the error
+//! condition has been resolved).
 
 use chrono::DateTime;
 use chrono::Utc;