Telemetry Streaming v1.22.0

F5Networks · Aug 6, 2021 · ea57aa4 · ea57aa4
1 parent 4db2b55
commit ea57aa4
Show file tree

Hide file tree

Showing 81 changed files with 7,728 additions and 2,215 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -461,7 +461,7 @@ pages:
         # only update on designated, stable branch
         - develop
         - doc-release-branch
-        - joes-ts-wip
+        - joes-new-ts-wip
 
 # Publish docs to clouddocs.f5.com
 publish_docs_to_production:
@@ -478,6 +478,6 @@ publish_docs_to_production:
     script:
         # Uncomment and set to create desired version format
         - aws s3 sync docs/_build/html s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest
-        # - aws s3 cp versions.json s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/versions.json
+        - aws s3 cp versions.json s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/versions.json
         # create invalidation to clear cloudfront cache
         - aws cloudfront create-invalidation --distribution-id $AWS_DIST --paths /products/extensions/f5-telemetry-streaming/latest
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,22 @@
 # Changelog
 Changes to this project are documented in this file. More detail and links can be found in the Telemetry Streaming [Document Revision History](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html).
 
+## 1.22.0
+### Added
+- AUTOTOOL-622: [GitHub #94](https://github.com/F5Networks/f5-telemetry-streaming/issues/94): Add Data Dog consumer (experimental feature)
+- AUTOTOOL-2600: [GitHub #125](https://github.com/F5Networks/f5-telemetry-streaming/issues/125): Added "slowKilled" and "evictedConns" properties to "clientside" of virtual servers
+- AUTOTOOL-2605: [GitHub #141](https://github.com/F5Networks/f5-telemetry-streaming/issues/141): Add "eventSchemaVersion" property to F5_Cloud consumer
+- AUTOTOOL-2453: [GitHub #143](https://github.com/F5Networks/f5-telemetry-streaming/issues/143): Add new Google Cloud Logging consumer
+- AUTOTOOL-2623: Added new property "reportInstanceMetadata" to "Google_Cloud_Monitoring" consumer to control metadata reporting
+### Fixed
+- AUTOTOOL-2543: [GitHub #134](https://github.com/F5Networks/f5-telemetry-streaming/issues/134): Fixed issue where Prometheus consumer encounters an error when a metric name is registered more than once
+- AUTOTOOL-2591: Fixed issue where the Kafka Consumer would create a new Kafka connection on each Telemetry Streaming payload
+### Changed
+- AUTOTOOL-2411: The default namespace (f5telemetry_default) is now stated explicitly in traceName and paths of the to the trace files. 
+- AUTOTOOL-2573: Google Cloud Monitoring Consumer now uses instance_id and zone as time-series resource labels for BIG-IPs that run in Google Cloud
+- AUTOTOOL-2613: Update npm packages (@grpc/grpc-js from v1.3.0 to v1.3.4, aws-sdk from v2.914.0 to v2.943.0, statsd-client from v0.4.6 to v0.4.7)
+### Removed
+
 ## 1.21.0
 ### Added
 - AUTOTOOL-2555: Add new property "addTags" to StatsD consumer that makes Consumer to send tags along with metrics (experimental feature)
@@ -10,6 +26,7 @@ Changes to this project are documented in this file. More detail and links can b
 - AUTOTOOL-2510: Update npm packages (@f5devcentral/f5-teem from v1.4.6 to v1.5.0, aws-sdk from v2.885.0 to v2.914.0, mustache from v4.1.0 to v4.2.0, statsd-client from v0.4.5 to v0.4.6)
 - Replace grpc-js-0.2-modified with @grpc/grpc-js
 - AUTOTOOL-2535: Refactor Google Cloud (used by the Google Cloud Monitoring Consumer) authentication, and add support for caching short-lived authentication tokens
+### Removed
 
 ## 1.20.1
 ### Added

diff --git a/SUPPORT.md b/SUPPORT.md
@@ -17,10 +17,10 @@ Currently supported versions:
 
 | Software Version | Release Type  | First Customer Ship | End of Support  |
 |------------------|---------------|---------------------|-----------------|
-| TS 1.19.0        | Feature       |  06-Apr-2021        | 06-Jul-2021     |
 | TS 1.20.0        | Feature       |  18-May-2021        | 18-Aug-2021     |
 | TS 1.20.1        | LTS           |  30-Jun-2021        | 30-Jun-2022     |
 | TS 1.21.0        | Feature       |  28-Jun-2021        | 28-Sep-2021     |
+| TS 1.22.0        | Feature       |  09-Aug-2021        | 09-Nov-2021     |
 
 Versions no longer supported:
 
@@ -44,5 +44,6 @@ Versions no longer supported:
 | TS 1.16.0        | Feature       |  20-Nov-2020        | 20-Feb-2021     |
 | TS 1.17.0        | Feature       |  12-Jan-2021        | 12-Apr-2021     |
 | TS 1.18.0        | Feature       |  23-Feb-2021        | 23-May-2021     |
+| TS 1.19.0        | Feature       |  06-Apr-2021        | 06-Jul-2021     |
 
 See the [Release notes](https://github.com/F5Networks/f5-telemetry-streaming/releases) and [Telemetry Streaming documentation](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html) for new features and issues resolved for each release.
diff --git a/contributing/README.md b/contributing/README.md
@@ -108,7 +108,7 @@ How does the project handle a typical `POST` request?
             "trace": false,
             "format": "default"
         },
-        "schemaVersion": "1.21.0"
+        "schemaVersion": "1.22.0"
     }
 }
 ```

diff --git a/contributing/process_release.md b/contributing/process_release.md
@@ -56,6 +56,7 @@
     * 1.19.0 - 14.1 MB
     * 1.20.0 - 14.4 MB
     * 1.21.0 - 15.5 MB
+    * 1.22.0 - 15.6 MB
   * Install build to BIG-IP, navigate to folder `/var/config/rest/iapps/f5-telemetry/` and check following:
     * Run `du -sh` and check that folder's size (shouldn't be much greater than previous versions):
       * 1.4.0 - 65 MB
@@ -74,9 +75,10 @@
       * 1.17.0 - 95 MB (NOTE: grpc module deps increase)
       * 1.18.0 - 100 MB
       * 1.19.0 - 101 MB
-      * 1.20.0 - 103MB
-      * 1.21.0 - 111MB
-    * Check `nodejs/node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
+      * 1.20.0 - 103 MB
+      * 1.21.0 - 111 MB
+      * 1.22.0 - 112 MB
+    * Check `node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
 * Ensure that all tests (unit tests and functional tests passed)
 * Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:
   * git tag -l -n
@@ -92,6 +94,17 @@
   * push the new build artifacts to the 'f5-automation-toolchain-generic/f5-telemetry' Artifactory repository
   * send the release candidate email with features, bugs, artifactory URL
 
+## LTS Release process (developers only)
+
+Following list of changes should be done for both master/develop and LTS branches:
+
+* Update [SUPPORT.md](SUPPORT.md) - set release type "LTS" and update support dates for LTS release
+* Update [CHANGELOG.md](CHANGELOG.md) - add *Promoted to LTS* to **Changed** section
+* Schema changes:
+  * Create new folder for LTS version (with name X.Y.Z) in [src/schema](src/schema)
+  * All schemas that are "newer" than LTS release should be updated to support LTS version. Update *base_schema.json* file only for each applicable version
+  * **LTS branch only** - update [src/schema/latest](src/schema/latest) to match LTS schema
+
 ## Release process
 
 * Create new branch from `master`, e.g. `rc-master-branch`. It will be easier to merge branches and resolve conflicts without any following up issues.

diff --git a/docs/conf.py b/docs/conf.py
@@ -21,6 +21,7 @@
 sys.path.insert(0, os.path.abspath('.'))
 
 import f5_sphinx_theme
+import re
 
 
 # -- General configuration ------------------------------------------------
@@ -78,7 +79,7 @@
 # The short X.Y version.
 version = u''
 # The full version, including alpha/beta/rc tags.
-release = u'1.21.0'
+release = u'1.22.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -154,13 +155,14 @@
 html_theme = 'f5_sphinx_theme'
 html_theme_path = f5_sphinx_theme.get_html_theme_path()
 html_theme_options = {
-    'next_prev_link': True
+    'next_prev_link': True,
+    'version_selector': True
 }
 
-# html_context = {
-# 'version_meta_path': '/products/extensions/f5-telemetry-streaming/versions.json',
-# 'project_safe': re.sub('[^A-Za-z0-9]+', '', project)
-# }
+html_context = {
+'version_meta_path': '/products/extensions/f5-telemetry-streaming/versions.json',
+'project_safe': re.sub('[^A-Za-z0-9]+', '', project)
+}
 
 
 # Theme options are theme-specific and customize the look and feel of a theme

diff --git a/docs/custom-endpoints.rst b/docs/custom-endpoints.rst
@@ -93,7 +93,7 @@ In order to have hostname information available in the data sent to your consume
 
 Creating System Pollers specific to the custom endpoint
 -------------------------------------------------------
-Because you might want to specify different polling intervals for the custom endpoints, v1.10.0 also enables the ability to create an system poller specific to an endpoint or array of endpoints.  To do this, you use the new **endpointList** property in your system poller definition.
+Because you may want to specify different polling intervals for the custom endpoints, v1.10.0 also enables the ability to create an system poller specific to an endpoint or array of endpoints.  To do this, you use the new **endpointList** property in your system poller definition.
 
 EndpointList is simply a list of endpoints to use in data collection, and can include the following types:
 

diff --git a/docs/declarations.rst b/docs/declarations.rst
@@ -233,8 +233,118 @@ Customizing the Generic HTTP consumer payload
 
 |
 
+.. _addtagex:
 
+Using the addTags property for the StatsD consumer
+--------------------------------------------------
+.. IMPORTANT:: The addTags property is available in TS v1.21.0 and later. See :ref:`StatsD addTags<addtags>` for more information on this feature.
 
+The following example shows a declaration with a systemPoller, a Listener, and a StatsD consumer with **addTags** using the **sibling** method.
+
+.. code-block:: json
+
+    {
+        "class": "Telemetry",
+        "My_System": {
+            "class": "Telemetry_System",
+            "systemPoller": {
+                "interval": 60
+            }
+        },
+        "My_Consumer_with_AutoTagging": {
+            "class": "Telemetry_Consumer",
+            "type": "Statsd",
+            "host": "192.0.2.1",
+            "protocol": "udp",
+            "port": 8125,
+            "addTags": {
+                "method": "sibling"
+            }
+        }
+    }
+
+|
+
+Next, we show the output from the System poller:
+
+.. code-block:: json
+
+    {
+        "pools": {
+            "/Common/app.app/app_pool": {
+                "activeMemberCnt": 0,
+                "serverside.bitsIn": 0,
+                "serverside.bitsOut": 0,
+                "serverside.curConns": 0,
+                "serverside.maxConns": 0,
+                "serverside.pktsIn": 0,
+                "serverside.pktsOut": 0,
+                "serverside.totConns": 0,
+                "availabilityState": "available",
+                "enabledState": "enabled",
+                "status.statusReason": "The pool is available",
+                "name": "/Common/app.app/app_pool",
+                "members": {
+                    "/Common/10.0.3.5:80": {
+                        "addr": "10.0.3.5",
+                        "monitorStatus": "up",
+                        "port": 0,
+                        "serverside.bitsIn": 0,
+                        "serverside.bitsOut": 0,
+                        "serverside.curConns": 0,
+                        "serverside.maxConns": 0,
+                        "serverside.pktsIn": 0,
+                        "serverside.pktsOut": 0,
+                        "serverside.totConns": 0,
+                        "availabilityState": "available",
+                        "enabledState": "enabled",
+                        "status.statusReason": "Pool member is available"
+                    }
+                },
+                "tenant": "Common",
+                "application": "app.app"
+            }
+        }
+    }
+
+|
+
+Without the **addTags** property, the StatsD consumer sends only numeric metrics like **serverside.bitsIn** and so on.
+
+With the **addTags** property, the StatsD consumer also sends numeric metrics but also sends the following tags along with each metric
+
+- For nested metrics under **pools.-Common-app.app-app_pool**, it sends following tags:
+
+  .. code-block:: json
+
+      {
+         "availabilityState": "available",
+         "enabledState": "enabled",
+         "status.statusReason": "The pool is available",
+         "name": "/Common/app.app/app_pool",
+         "tenant": "Common",
+         "application": "app.app"
+      }
+
+
+- For nested metrics under **pools.-Common-app.app-app_pool.memebers.-Common-10-0-3-5-80**, it sends following tags:
+
+  .. code-block:: json
+     
+      {                   
+         "addr": "10.0.3.5",
+         "monitorStatus": "up",
+         "availabilityState": "available",
+         "enabledState": "enabled",
+         "status.statusReason": "Pool member is available"
+      }
+    
+
+|
+
+:ref:`Back to top<examples>`
+
+|
 
 
 

diff --git a/docs/index.rst b/docs/index.rst
@@ -3,7 +3,7 @@ F5 Telemetry Streaming
 
 Welcome to the F5 Telemetry Streaming User Guide. To provide feedback on this documentation, you can file a GitHub Issue, or email us at [email protected]. |br|
 
-For information on supported versions of Telemetry Streaming, see |supportmd|.
+.. IMPORTANT:: This the documentation for the **latest** version of Telemetry Streaming, if you want to see the documentation for a long term support (LTS) version, use the version selector on the top left (for details, see |supportmd|). 
 
 Introduction
 ------------

diff --git a/docs/output-example.rst b/docs/output-example.rst
@@ -15,7 +15,7 @@ The following shows the system information that Telemetry Streaming collects.
 
 .. NOTE:: For some of the output to appear, you must have the applicable BIG-IP module licensed and provisioned (for example, you must have BIG-IP DNS provisioned to get GSLB wide IP and Pool information).
 
-Telemetry Streaming 1.17 introduced additional pool and virtual server output, such as virtual server **clientside.pktsIn** and pools **serverside.totConns**.  These do not appear in earlier versions.
+Telemetry Streaming 1.22 introduced additional virtual server output: **clientside.slowKilled** and **clientside.evictedConns**.  These do not appear in earlier versions.
 
 .. literalinclude:: ../examples/output/system_poller/output.json
     :language: json

diff --git a/docs/rest-api-endpoints.rst b/docs/rest-api-endpoints.rst
@@ -24,7 +24,7 @@ Response
 ````````
 Response is valid JSON data. When the response is HTTP 200
 
-When the response code is other than 2xx, then the response body in general will look like the following object:
+When the response code is other than 2xx, the response body typically looks like the following object:
 
 .. code-block:: bash
 

diff --git a/docs/revision-history.rst b/docs/revision-history.rst
@@ -11,13 +11,16 @@ Document Revision History
         - Description
         - Date
 
-      * - 1.21.0
-        - Updated the documentation for Telemetry Streaming v1.21.0. This release contains the following changes: |br| * An **experimental** property for the StatsD consumer (see :ref:`StatsD addTags<addtags>`) |br| * A number of minor bug fixes, stability enhancements, and foundational elements for some upcoming features. 
-        - 6-28-21
-
+      * - 1.22.0
+        - Updated the documentation for Telemetry Streaming v1.22.0. This release contains the following changes: |br| * Added  **clientside.slowKilled** and **clientside.evictedConns** to virtual server output (see :ref:`System information output<sysinfo>`) |br| |br| Issues Resolved: |br| * Fixed issue where Prometheus consumer encounters an error when a metric name is registered more than once, `GitHub #134 <https://github.com/F5Networks/f5-telemetry-streaming/issues/134>`_ |br| * Fixed issue where the Kafka Consumer would create a new Kafka connection on each Telemetry Streaming payload |br| * Google Cloud Monitoring Consumer now uses instance_id and zone as time-series resource labels for BIG-IPs that run in Google Cloud
+        - 8-9-21
+
+      * - 1.20.1
+        - Released Telemetry Streaming v1.20.1 as a LTS (Long Term Support) version. See the Telemetry Streaming |supportmd| for information about the Telemetry Streaming support policy.
+        - 6-30-21
 
       * - 1.21.0
-        - Updated the documentation for Telemetry Streaming v1.21.0. This release contains a number of minor bug fixes, stability enhancements, and foundational elements for some upcoming features. 
+        - Updated the documentation for Telemetry Streaming v1.21.0. This release contains the following changes: |br| * An **experimental** property for the StatsD consumer (see :ref:`StatsD addTags<addtags>`) |br| * A number of minor bug fixes, stability enhancements, and foundational elements for some upcoming features. 
         - 6-28-21
 
       * - 1.20.0
@@ -131,4 +134,8 @@ Document Revision History
 
 .. |schemalink| raw:: html
 
-   <a href="https://github.com/F5Networks/f5-telemetry-streaming/tree/master/src/schema" target="_blank">schema files</a>
+   <a href="https://github.com/F5Networks/f5-telemetry-streaming/tree/master/src/schema" target="_blank">schema files</a>
+
+.. |supportmd| raw:: html
+
+   <a href="https://github.com/F5Networks/f5-telemetry-streaming/blob/master/SUPPORT.md" target="_blank">Support information on GitHub</a>
diff --git a/docs/setting-up-consumer.rst b/docs/setting-up-consumer.rst
@@ -456,13 +456,15 @@ addTags (experimental)
 ``````````````````````
 .. sidebar:: :fonticon:`fa fa-info-circle fa-lg` Version Notice:
 
-    The EXPERIMENTAL feature addTags for StatsD is available in Telemetry Streaming 1.20 and later.
+    The EXPERIMENTAL feature addTags for StatsD is available in Telemetry Streaming 1.21 and later.
 
-Telemetry Streaming 1.20 adds a new property for the StatsD consumer: **addTags**.  This feature causes TS to parse the incoming payload for values to automatically add as tags. Currently only the **sibling** method is supported.  
+Telemetry Streaming 1.21 adds a new property for the StatsD consumer: **addTags**.  This feature causes TS to parse the incoming payload for values to automatically add as tags. Currently only the **sibling** method is supported.  
+
+To see an example and the output from **addTags**, see :ref:`addTags example<addtagex>`.
 
 |
 
-Example Declaration (updated with the **experimental** feature *addTags* introduced in TS 1.20 (see below). If using a prior version, **remove** the highlighted lines, and the comma in line 9):
+Example Declaration (updated with the **experimental** feature *addTags* introduced in TS 1.21. If using a prior version, **remove** the highlighted lines, and the comma in line 9):
 
 .. literalinclude:: ../examples/declarations/statsd.json
     :language: json

diff --git a/examples/declarations/data_dog.json b/examples/declarations/data_dog.json
@@ -0,0 +1,8 @@
+{
+    "class": "Telemetry",
+    "My_Consumer": {
+        "class": "Telemetry_Consumer",
+        "type": "DataDog",
+        "apiKey": "api_key"
+    }
+}
diff --git a/examples/declarations/f5_cloud.json b/examples/declarations/f5_cloud.json
@@ -90,7 +90,8 @@
       "authProviderX509CertUrl": "https://www.googleapis.com/oauth2/v1/certs",
       "clientX509CertUrl": "https://www.googleapis.com/robot/v1/metadata/x509/test%40deos-dev.iam.gserviceaccount.com"
     },
-    "targetAudience": "deos-ingest"
+    "targetAudience": "deos-ingest",
+    "eventSchemaVersion": "5"
   },
   "schemaVersion": "1.15.0"
 }
diff --git a/examples/declarations/google_cloud_logging.json b/examples/declarations/google_cloud_logging.json
@@ -0,0 +1,15 @@
+{
+    "class": "Telemetry",
+    "My_Consumer": {
+        "class": "Telemetry_Consumer",
+        "type": "Google_Cloud_Logging",
+        "logScope": "projects",
+        "logScopeId": "yourProjectId",
+        "logId": "yourLogId",
+        "privateKey": {
+            "cipherText": "yourPrivateKey"
+    	},
+        "privateKeyId": "yourPrivateKeyId",
+        "serviceEmail": "yourServiceEmail"
+    }
+}