Merge branch 'docs-1.20.1' into LTS-1.20.1

.gitlab-ci.yml

@@ -469,15 +469,15 @@ publish_docs_to_production:
     stage: doc
     environment:
         name: production
-        url: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest
+        url: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/1.20
     only:
         # fill in desired release branch name to add deployment from a branch: currently *docs-latest*
-        - docs-latest@automation-toolchain/f5-telemetry
+        - docs-1.20.1@automation-toolchain/f5-telemetry
     tags:
         - cm-official-docker-executor
     script:
         # Uncomment and set to create desired version format
-        - aws s3 sync docs/_build/html s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest
-        # - aws s3 cp versions.json s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/versions.json
+        - aws s3 sync docs/_build/html s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/1.20
+        - aws s3 cp versions.json s3://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/versions.json
         # create invalidation to clear cloudfront cache
-        - aws cloudfront create-invalidation --distribution-id $AWS_DIST --paths /products/extensions/f5-telemetry-streaming/latest
+        - aws cloudfront create-invalidation --distribution-id $AWS_DIST --paths /products/extensions/f5-telemetry-streaming/1.20

docs/conf.py

@@ -21,6 +21,7 @@
 sys.path.insert(0, os.path.abspath('.'))
 
 import f5_sphinx_theme
+import re
 
 
 # -- General configuration ------------------------------------------------
@@ -153,14 +154,16 @@
 #
 html_theme = 'f5_sphinx_theme'
 html_theme_path = f5_sphinx_theme.get_html_theme_path()
+
 html_theme_options = {
-    'next_prev_link': True
+    'next_prev_link': True,
+    'version_selector': True}
+
+html_context = {
+ 'version_meta_path': '/products/extensions/f5-telemetry-streaming/versions.json',
+ 'project_safe': re.sub('[^A-Za-z0-9]+', '', project)
 }
 
-# html_context = {
-# 'version_meta_path': '/products/extensions/f5-telemetry-streaming/versions.json',
-# 'project_safe': re.sub('[^A-Za-z0-9]+', '', project)
-# }
 
 
 # Theme options are theme-specific and customize the look and feel of a theme

docs/event-listener.rst

@@ -12,7 +12,7 @@ To use the Event Listener, you must:
 
 1. Configure the sources of log/event data. You can do this by either POSTing a single AS3 declaration or you can use TMSH or the GUI to configure individual modules.
 
-2. Post a telemetry declaration with the Telemetry_Listener class, as shown in the following minimal example of an Event Listner:
+2. Post a telemetry declaration with the Telemetry_Listener class, as shown in the following minimal example of an Event Listener:
 
 
 .. code-block:: bash
@@ -23,10 +23,15 @@ To use the Event Listener, you must:
    }
 
 
+Tracing can also can be optionally enabled for each Telemetry Streaming Listener, using the **trace** property. By default, tracing is disabled (the **trace** property defaults to **false**), but can be enabled by setting the property to **true**, or to a string value representing a valid operating system path (for example: ``/var/telemetry/myTraceFile.json``). When tracing is enabled, the Listener will write the data it receives (after Telemetry Streaming has processed and normalized the data) to disk. Telemetry Streaming 1.20.0 and later also includes the ability to trace the raw input data (before Telemetry Streaming has processed the data). For more information and instructions, see :ref:`Trace<trace>`.
+
 IMPORTANT:
 
 - The following configuration examples assume that TS is running on the same BIG-IP that is being monitored, and that the listener is using default port 6514.
 - When TS is not a local listener, the corresponding configurations should be adjusted to reflect remote addresses.
+
+
+.. NOTE:: See :ref:`this troubleshooting entry<trace>` for information on how to write an Event Listener's incoming raw data to a trace file to assist with troubleshooting.
 
 
 
@@ -467,6 +472,7 @@ The following shows the input sent as different buffers, and the resulting outpu
 
 
 
+
 .. |logging-png| image:: /images/high-speed-logging.png
    :alt: High Speed Logging
 

docs/index.rst

@@ -123,7 +123,6 @@ You can click the following links to go directly to a specific :doc:`setting-up-
    revision-history
 
 
-
 .. |video| raw:: html
 
     <iframe width="560" height="315" src="https://www.youtube.com/embed/YeEAovBvUkY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

docs/revision-history.rst

@@ -20,7 +20,7 @@ Document Revision History
         - 6-28-21
 
       * - 1.20.0
-        - Updated the documentation for Telemetry Streaming v1.20.0. This release contains the following changes: |br| * Added data tracing and 'success' message to Google Cloud Monitoring consumer |br| * Added system.configSyncSucceeded to default output and f5_system_configSyncSucceeded to Prometheus output, `GitHub #74 <https://github.com/F5Networks/f5-telemetry-streaming/issues/74>`_  |br| * Added Capacity_Float to system.diskStorage, `GitHub #119 <https://github.com/F5Networks/f5-telemetry-streaming/issues/119>`_ |br| |br| Issues Resolved: |br| * 
+        - Updated the documentation for Telemetry Streaming v1.20.0. This release contains the following changes: |br| * Added support for customizing the Telemetry Streaming payload (see :doc:`customizing-data` and the related :ref:`example<custompl>`) |br| * Added a new troubleshooting entry for writing an Event Listener's incoming raw data to a trace file (see :ref:`Troubleshooting<trace>`), also added a new note to the description of :doc:`event-listener` about tracing. |br| * Added data tracing and 'success' message to Google Cloud Monitoring consumer  |br| * Added system.configSyncSucceeded to default output and f5_system_configSyncSucceeded to Prometheus output, `GitHub #74 <https://github.com/F5Networks/f5-telemetry-streaming/issues/74>`_ |br| * Added Capacity_Float to system.diskStorage, `GitHub #119 <https://github.com/F5Networks/f5-telemetry-streaming/issues/119>`_ |br| |br| Issues Resolved: |br| * Fixed issue where 'long' fields in ASM events were causing ASM events to become malformed. Individual event fields are now permitted to be <= MAX_BUFFER_SIZE (16k), `GitHub #127 <https://github.com/F5Networks/f5-telemetry-streaming/issues/127>`_ |br| * AWS S3 date path was incorrect |br| * Should not create multiple Tracer instances that point to the same file
         - 5-18-21
 
       * - 1.19.0

docs/troubleshooting.rst

@@ -192,6 +192,58 @@ Telemetry Streaming sends this JSON payload to the Event Listener you specified,
 
 |
 
+
+.. _trace:
+
+How can I write an Event Listener's incoming raw data to a trace file?
+----------------------------------------------------------------------
+.. sidebar:: :fonticon:`fa fa-info-circle fa-lg` Version Notice:
+
+   Support for writing an Event Listener's incoming raw data to a trace file is available in TS v1.20 and later
+
+In Telemetry Streaming 1.20 and later you can configure TS to write an Event Listener's incoming raw data to a trace file. This is useful when troubleshooting, as it allows you to reproduce the exact issue instead of relying on the BIG-IP configuration, profiles, and traffic generation.
+
+This feature is enabled using the **trace** property with values of **input** and/or **output**. All data is written to the ``/var/tmp/telemetry`` directory (or check logs for the exact file path).
+
+.. IMPORTANT:: **Input** tracing data is written in HEX format. If you want to remove sensitive data, you need to decode HEX data, clean or remove the sensitive data, and re-encode it back to HEX format. But this operation does not guarantee 100% reproduction of issue (in the case of input tracing data will be sent to F5 Support for further investigation). Instead of cleaning the data (or complete removal of sensitive data), we recommend replacing it with non-sensitive data (i.e. the exact same size and original encoding).
+
+The following is an example of configuring the Event Listener to trace incoming data:
+
+.. code-block:: json
+
+    {
+        "class": "Telemetry",
+        "Listener": {
+            "class": "Telemetry_Listener",
+            "trace": {
+                "type": "input"
+            }
+        }
+    }
+
+|
+
+If you want to enable both input and output tracing, use the following syntax in your Event Listener:
+
+.. code-block:: json
+
+    {
+        "class": "Telemetry",
+        "Listener": {
+            "class": "Telemetry_Listener",
+            "trace": [
+                {
+                    "type": "input"
+                },
+                {
+                    "type": "output"
+                }
+            ]
+        }
+    }
+
+|
+
 .. _restjavad:
 
 Why is my BIG-IP experiencing occasional high CPU usage and slower performance?

versions.json

@@ -2,9 +2,9 @@
   "versionMetaTimestamp": 1540928503,
   "latestVersion": {
     "name": "1.21 (non-LTS)",
-    "url": "/products/extensions/f5-telemetry/latest/"
+    "url": "/products/extensions/f5-telemetry-streaming/latest/"
   },
   "otherVersions": [
-    { "name": "1.20.1 (LTS)", "url": "/products/extensions/f5-telemetry/1.20/"}
+    { "name": "1.20.1 (LTS)", "url": "/products/extensions/f5-telemetry-streaming/1.20/"}
   ]
 }