merge v1.26.0 into master branch

F5Networks · Jan 24, 2022 · 3bc1439 · 3bc1439
1 parent af15fb1
commit 3bc1439
Show file tree

Hide file tree

Showing 274 changed files with 6,461 additions and 1,070 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,14 @@
 # Changelog
 Changes to this project are documented in this file. More detail and links can be found in the Telemetry Streaming [Document Revision History](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html).
 
+## 1.26.0
+### Added
+- AUTOTOOL-2801: [GitHub #171](https://github.com/F5Networks/f5-telemetry-streaming/issues/171): Added fqdn property to pool members that use FQDNs
+### Fixed
+### Changed
+- AUTOTOOL-2890: Update npm packages (@grpc/proto-loader from ~0.3.0 to ^0.6.4)
+### Removed
+
 ## 1.25.0
 ### Added
 - AUTOTOOL-2798: [GitHub #154](https://github.com/F5Networks/f5-telemetry-streaming/issues/154): Allow keyless (service account based) authorization for GCP Metrics and Logging consumers.

diff --git a/README.md b/README.md
@@ -33,7 +33,7 @@ We encourage you to use our [Slack channel](https://f5cloudsolutions.herokuapp.c
 
 ## Copyright
 
-Copyright 2014-2021 F5 Networks Inc.
+Copyright 2014-2022 F5 Networks Inc.
 
 ### F5 Networks Contributor License Agreement
 

diff --git a/SUPPORT.md b/SUPPORT.md
@@ -18,9 +18,9 @@ Currently supported versions:
 | Software Version | Release Type  | First Customer Ship | End of Support  |
 |------------------|---------------|---------------------|-----------------|
 | TS 1.20.1        | LTS           |  30-Jun-2021        | 30-Jun-2022     |
-| TS 1.23.0        | Feature       |  21-Sep-2021        | 21-Dec-2021     |
 | TS 1.24.0        | Feature       |  02-Nov-2021        | 02-Feb-2022     |
 | TS 1.25.0        | Feature       |  14-Dec-2021        | 14-Mar-2022     |
+| TS 1.26.0        | Feature       |  25-Jan-2022        | 25-Apr-2022     |
 
 Versions no longer supported:
 
@@ -48,5 +48,6 @@ Versions no longer supported:
 | TS 1.20.0        | Feature       |  18-May-2021        | 18-Aug-2021     |
 | TS 1.21.0        | Feature       |  28-Jun-2021        | 28-Sep-2021     |
 | TS 1.22.0        | Feature       |  09-Aug-2021        | 09-Nov-2021     |
+| TS 1.23.0        | Feature       |  21-Sep-2021        | 21-Dec-2021     |
 
 See the [Release notes](https://github.com/F5Networks/f5-telemetry-streaming/releases) and [Telemetry Streaming documentation](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html) for new features and issues resolved for each release.
diff --git a/contributing/README.md b/contributing/README.md
@@ -108,7 +108,7 @@ How does the project handle a typical `POST` request?
             "trace": false,
             "format": "default"
         },
-        "schemaVersion": "1.25.0"
+        "schemaVersion": "1.26.0"
     }
 }
 ```

diff --git a/contributing/process_release.md b/contributing/process_release.md
@@ -24,6 +24,7 @@
   * [package.json](package.json)
   * [package-lock.json](package-lock.json)
   * [project.spec](project.spec) (not required starting from 1.5)
+  * [versions.json](versions.json)
   * [src/lib/constants.js](src/lib/constants.js) (not required starting from 1.10)
   * [src/schema/latest/base_schema.json](src/schema/latest/base_schema.json)
   * [contributing/README.md](contributing/README.md) (example of response, optional)
@@ -60,6 +61,7 @@
     * 1.23.0 - 17.8 MB (NOTE: inclusion of OpenTelemetry libraries)
     * 1.24.0 - 19.2 MB
     * 1.25.0 - 17.7 MB
+    * 1.26.0 - 17.8 MB
   * Install build to BIG-IP, navigate to folder `/var/config/rest/iapps/f5-telemetry/` and check following:
     * Run `du -sh` and check that folder's size (shouldn't be much greater than previous versions):
       * 1.4.0 - 65 MB
@@ -84,6 +86,7 @@
       * 1.23.0 - 132 MB (NOTE: inclusion of OpenTelemetry libraries)
       * 1.24.0 - 134 MB
       * 1.25.0 - 130 MB
+      * 1.26.0 - 127 MB
     * Check `node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
 * Ensure that all tests (unit tests and functional tests passed)
 * Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:

diff --git a/docs/conf.py b/docs/conf.py
@@ -79,7 +79,7 @@
 # The short X.Y version.
 version = u''
 # The full version, including alpha/beta/rc tags.
-release = u'1.25.0'
+release = u'1.26.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/docs/output-example.rst b/docs/output-example.rst
@@ -7,21 +7,47 @@ Use this page to see the type of information that Telemetry Streaming collects.
 
 .. IMPORTANT::  The following are just examples; the actual fields will vary depending on factors such as how the log/event source is configured, BIG-IP versions, and so on. 
 
+.. NOTE:: For some of the output to appear, you must have the applicable BIG-IP module licensed and provisioned (for example, you must have BIG-IP DNS provisioned to get GSLB wide IP and Pool information).
+
 .. _sysinfo:
 
 System Information
 ------------------
 The following shows the system information that Telemetry Streaming collects. 
 
-.. NOTE:: For some of the output to appear, you must have the applicable BIG-IP module licensed and provisioned (for example, you must have BIG-IP DNS provisioned to get GSLB wide IP and Pool information).
+.. NOTE:: In Telemetry Streaming 1.24 and later, properties that require the bash endpoint are skipped if bash is not available on the target BIG-IP: system.diskStorage, system.diskLatency, system.apmState, and all tmstats properties (used with “Splunk legacy” format).
+
+**Additional properties** |br|
+The following properties were introduced in recent Telemetry Streaming versions.  These properties do not appear in the output in previous versions.  The following table shows the TS version the property was introduced, the type of output, and property (with description if applicable).
+
+.. list-table::
+      :widths: 20 20 200 
+      :header-rows: 1
+
+      * - TS Version
+        - Type
+        - Property
+
+      * - 1.22
+        - Virtual Server
+        - **clientside.slowKilled**, **clientside.evictedConns**
 
-Telemetry Streaming 1.22 introduced additional virtual server output: **clientside.slowKilled** and **clientside.evictedConns**.  These do not appear in earlier versions.
+      * - 1.23
+        - Virtual Server
+        - **isAvailable**, **isEnabled** (both boolean), which monitor virtual server availability, particularly for the Prometheus consumer. 
 
-Telemetry Streaming 1.23 introduced the following, which do not appear in earlier versions:
-- Virtual server output: **isAvailable** and **isEnabled** (both boolean), which monitor virtual server availability, particularly for the Prometheus consumer.  
-- System output: **throughtputPerformance**, which provides throughput performance information.
+      * - 1.23
+        - System
+        - **throughtputPerformance**, which shows throughput performance information. 
+
+      * - 1.25
+        - Pool Member
+        - **poolName**, which shows the associated pool name.
+
+      * - 1.26
+        - Pool Member
+        - **fqdn**, which shows FQDN information for LTM pool member nodes that are FQDN nodes.
 
-In Telemetry Streaming 1.24 and later, properties that require the bash endpoint are now skipped if bash is not available on the target BIG-IP: system.diskStorage, system.diskLatency, system.apmState, and all tmstats properties (used with “Splunk legacy” format).
 
 
 .. literalinclude:: ../examples/output/system_poller/output.json
@@ -94,4 +120,9 @@ AVR Request Log
     :language: json
 
 
-:ref:`Back to top<outputexample-ref>`
+:ref:`Back to top<outputexample-ref>`
+
+
+.. |br| raw:: html
+
+   <br />
diff --git a/docs/revision-history.rst b/docs/revision-history.rst
@@ -11,8 +11,12 @@ Document Revision History
         - Description
         - Date
 
+      * - 1.26.0
+        - Updated the documentation for Telemetry Streaming v1.26.0. This release contains the following changes: |br| * Added the **fqdn** property to the output for pool members that use FQDNs (see :ref:`System Information<sysinfo>`) |br| * Added two new sections to :doc:`troubleshooting` for logging and tracing 
+        - 1-25-22
+
       * - 1.25.0
-        - Updated the documentation for Telemetry Streaming v1.25.0. This release contains the following changes: |br| * Added support for IAM roles for Google Cloud Platform (see Google :ref:`Cloud Monitoring<gcmiam>` and :ref:`Cloud Logging<gcliam>`) |br| * Added the **metricPrefix** and **customTags** properties to the DataDog consumer (see :ref:`DataDog<datadog>`) |br| * Added the **convertBooleansToMetrics** property to the DataDog, StatsD, and OpenTelemetry consumers (see :ref:`DataDog<datadog>`, :ref:`StatsD<statsd-ref>`, and :ref:`OpenTelemetry<opent>`) |br| * Added the **endpointUrl** property to AWS S3 and CloudWatch consumers (see :ref:`AWS S3<awss3-ref>` and :ref:`AWS CloudWatch<awscloud-ref>`) |br| * Removed the experimental label from the autoTag property for StatsD (see :ref:`StatsD addTags<addtags>`) |br| * Removed the experimental label from Splunk multi-metric format (see :ref:`Splunk multi-metric<multi-metric>`) |br| |br| Issues Resolved: |br| * 
+        - Updated the documentation for Telemetry Streaming v1.25.0. This release contains the following changes: |br| * Added support for IAM roles for Google Cloud Platform (see Google :ref:`Cloud Monitoring<gcmiam>` and :ref:`Cloud Logging<gcliam>`), `GitHub #154 <https://github.com/F5Networks/f5-telemetry-streaming/issues/154>`_ |br| * Added the **metricPrefix** (`GitHub #152 <https://github.com/F5Networks/f5-telemetry-streaming/issues/152>`_) and **customTags** (`GitHub #160 <https://github.com/F5Networks/f5-telemetry-streaming/issues/160>`_) properties to the DataDog consumer (see :ref:`DataDog<datadog>`) |br| * Added the **convertBooleansToMetrics** property to the DataDog, StatsD, and OpenTelemetry consumers (see :ref:`DataDog<datadog>`, :ref:`StatsD<statsd-ref>`, and :ref:`OpenTelemetry<opent>`) |br| * Added the **endpointUrl** property to AWS S3 and CloudWatch consumers (see :ref:`AWS S3<awss3-ref>` and :ref:`AWS CloudWatch<awscloud-ref>`), `GitHub #173 <https://github.com/F5Networks/f5-telemetry-streaming/issues/173>`_ |br| * Added **poolName** to pool member output (see :ref:`System Information<sysinfo>`), `GitHub #170 <https://github.com/F5Networks/f5-telemetry-streaming/issues/170>`_ |br| * Removed the experimental label from the autoTag property for StatsD (see :ref:`StatsD addTags<addtags>`) |br| * Removed the experimental label from Splunk multi-metric format (see :ref:`Splunk multi-metric<multi-metric>`)
         - 12-14-21
 
       * - 1.24.0

diff --git a/docs/setting-up-consumer.rst b/docs/setting-up-consumer.rst
@@ -7,6 +7,10 @@ Use this section to find example declarations and notes for supported push-based
 
 .. IMPORTANT:: Each of the following examples shows only the **Consumer** class of a declaration and must be included with the rest of the base declaration (see :ref:`components`).
 
+Use the index on the right to locate a specific consumer.
+
+|
+
 .. _splunk-ref:
 
 Splunk

diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
@@ -5,18 +5,103 @@ Use this section to read about known issues and for common troubleshooting steps
 Telemetry Streaming general troubleshooting tips
 ------------------------------------------------
 
-- Examine the restnoded failure log at /var/log/restnoded/restnoded.log (this is where Telemetry Streaming records error messages)
+- Examine the restnoded failure log at **/var/log/restnoded/restnoded.log** (this is where Telemetry Streaming records error messages).
 
 - Examine the REST response:
 
-  - A 400-level response will carry an error message with it
-  - If this message is missing, incorrect, or misleading, please let us know by filing an issue on Github.
+  - A 400-level response carries an error message with it
+  - If this message is missing, incorrect, or misleading, let us know by filing an |issues|.
 
 - Use Telemetry's trace option to create a detailed trace of the configuration process for subsequent analysis. Telemetry's trace option can be a powerful tool to learn about its working details and to review Telemetry's operations in detail.
 
+Logging
+-------
+Telemetry Streaming writes log output to the file **/var/log/restnoded/restnoded.log** on the BIG-IP.
+The verbosity of the log output can be adjusted by submitting a Telemetry Streaming declaration with a Controls class.
+The allowed log levels (in increasing order of verbosity) are **error**, **info**, and **debug**.
+The following is an example declaration containing a Controls class that sets the logging level to debug.
 
-Troubleshooting
----------------
+.. code-block:: json
+   :emphasize-lines: 3-5
+
+    {
+        "class": "Telemetry",
+        "Controls": {
+            "logLevel": "debug"
+        },
+        "My_System": {
+            "class": "Telemetry_System",
+            "systemPoller": {
+                "interval": 60
+            }
+        },
+        "My_Listener": {
+            "class": "Telemetry_Listener",
+            "port": 6514
+        },
+        "My_Consumer": {
+            "class": "Telemetry_Consumer",
+            "type": "Splunk",
+            "host": "192.0.2.1",
+            "protocol": "https",
+            "port": 8088,
+            "passphrase": {
+                "cipherText": "apikey"
+            }
+        }
+    }
+
+
+Tracing
+-------
+While Telemetry Streaming is processing data, you can configure it to write the current state of the data at key points to files for further inspection.
+These files are known as trace files, and can be enabled for the Telemetry_System, Telemetry_System_Poller, Telemetry_Listener, and Telemetry_Consumer classes.
+The trace files are written to the BIG-IP file system in the **/var/tmp/telemetry** directory using a filename based on the class and the name it was given.
+
+To enable tracing, set the trace property to **true**. To override the default file output location, set the trace property to a string value containing the new path.
+
+The **Telemetry_Listener** trace setting is a little more complex, see :ref:`trace` for more information.
+
+The most common use for trace files is to determine how far data progresses through the path from the BIG-IP to the third party tool.
+**Telemetry_System**, **Telemetry_System_Poller**, and **Telemetry_Listener** trace files are useful for determining if data is making it from the BIG-IP to Telemetry Streaming.
+If these trace files are not being generated, or are empty, check the BIG-IP configuration that sets up the sending of data to Telemetry streaming.
+
+**Telemetry_Consumer** trace files are useful for determining if the data is being sent from Telemetry Streaming to the desired third party tool.
+If these trace files are not being generated, or are empty, then check your **Telemetry_Consumer** settings in the Telemetry Streaming declaration.
+Also, check the logs for any issues sending the data to the third party tool.
+If all trace files look correct, then check any settings, queries, and logs in the third party tool.
+
+The following is an example that enables tracing on the Telemetry_System and Telemetry_Consumer classes.
+
+.. code-block:: json
+
+    {
+        "class": "Telemetry",
+        "My_System": {
+            "class": "Telemetry_System",
+            "trace": true,
+            "systemPoller": {
+                "interval": 60
+            }
+        },
+        "My_Consumer": {
+            "class": "Telemetry_Consumer",
+            "trace": true,
+            "type": "Splunk",
+            "host": "192.0.2.1",
+            "protocol": "https",
+            "port": 8088,
+            "passphrase": {
+                "cipherText": "apikey"
+            }
+        }
+    }
+
+|
+
+
+Specific troubleshooting entries
+--------------------------------
 
 I'm receiving a path not registered error when I try to post a declaration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
@@ -25,10 +110,10 @@ If you are receiving this error, it means either you did not install Telemetry S
 
 .. code-block:: shell
 
-    {
-        "code":404,
-        "message": "Public URI path no registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details.".
-        ...
+   {
+       "code":404,
+       "message": "Public URI path no registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details.".
+       ...
     }
 
 
@@ -350,4 +435,9 @@ Telemetry Streaming 1.19 and later includes the **compressionType** property in
 
 .. |telemetryconsumer| raw:: html
 
-   <a href="https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/schema-reference.html#telemetry-consumer" target="_blank">Telemetry_Consumer</a>
+   <a href="https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/schema-reference.html#telemetry-consumer" target="_blank">Telemetry_Consumer</a>
+
+
+.. |issues| raw:: html
+
+   <a href="https://github.com/F5Networks/f5-telemetry-streaming/issues" target="_blank">Issue on GitHub</a>
diff --git a/examples/output/system_poller/output.json b/examples/output/system_poller/output.json
@@ -634,6 +634,42 @@
                     "availabilityState": "offline",
                     "enabledState": "enabled",
                     "status.statusReason": "Pool member is available"
+                },
+                "/Example_Tenant/_auto_192.0.2.14:80": {
+                    "addr": "192.0.2.14",
+                    "monitorStatus": "unchecked",
+                    "poolName": "/Example_Tenant/A1/web_pool",
+                    "port": 80,
+                    "serverside.bitsIn": 0,
+                    "serverside.bitsOut": 0,
+                    "serverside.curConns": 0,
+                    "serverside.maxConns": 0,
+                    "serverside.pktsIn": 0,
+                    "serverside.pktsOut": 0,
+                    "serverside.totConns": 0,
+                    "availabilityState": "unknown",
+                    "enabledState": "enabled",
+                    "status.statusReason": "Pool member does not have service checking enabled",
+                    "totRequests": 0,
+                    "fqdn": "bestwebsite.com"
+                },
+                "/Example_Tenant/bestwebsite.com:80": {
+                    "addr": "::",
+                    "monitorStatus": "fqdn-up",
+                    "poolName": "/Example_Tenant/A1/web_pool",
+                    "port": 80,
+                    "serverside.bitsIn": 0,
+                    "serverside.bitsOut": 0,
+                    "serverside.curConns": 0,
+                    "serverside.maxConns": 0,
+                    "serverside.pktsIn": 0,
+                    "serverside.pktsOut": 0,
+                    "serverside.totConns": 0,
+                    "availabilityState": "available",
+                    "enabledState": "enabled",
+                    "status.statusReason": "The DNS server(s) are available",
+                    "totRequests": 0,
+                    "fqdn": "bestwebsite.com"
                 }
             },
             "tenant": "Example_Tenant",
-Original file line number
+Diff line change
@@ Expand Up @@
     .. IMPORTANT:: Each of the following examples shows only the **Consumer** class of a declaration and must be included with the rest of the base declaration (see :ref:`components`).
+    Use the index on the right to locate a specific consumer.
+    |
     .. _splunk-ref:
     Splunk
@@ Expand Down @@