merge v1.18.0 in

F5Networks · Feb 22, 2021 · 52fadd4 · 52fadd4
1 parent 2eedf62
commit 52fadd4
Show file tree

Hide file tree

Showing 101 changed files with 9,129 additions and 2,530 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,18 @@
 # Changelog
 Changes to this project are documented in this file. More detail and links can be found in the Telemetry Streaming [Document Revision History](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html).
 
+## 1.18.0
+### Added
+- AUTOTOOL-1987: Added new Namespace declare endpoint (/namespace/$namespace/declare) which supports POST and GET
+- AUTOTOOL-2148 [GitHub #104](https://github.com/F5Networks/f5-telemetry-streaming/issues/104): Add support for TLS Client Authentication to Generic HTTP Consumer
+### Fixed
+- AUTOTOOL-1710: Fix Event Listener startup errors that might cause restnoded to crash
+- AUTOTOOL-2227: Splunk multiEvent format should ignore 'References'
+### Changed
+- AUTOTOOL-2111: Update npm packages (applicationinsights from 1.8.7 to 1.8.9, aws-sdk from 2.775.0 to 2.830.0, google-auth-library from 6.1.1 to 6.1.4, mustache from 4.0.0 to 4.1.0)
+- AUTOTOOL-2212: Add AWS specific certificates to AWS Consumers
+### Removed
+
 ## 1.17.0
 ### Added
 - AUTOTOOL-2027 [GitHub #91](https://github.com/F5Networks/f5-telemetry-streaming/issues/91): Add custom timestamp for APM Events

diff --git a/SUPPORT.md b/SUPPORT.md
@@ -17,9 +17,8 @@ Currently supported versions:
 
 | Software Version | Release Type  | First Customer Ship | End of Support  |
 |------------------|---------------|---------------------|-----------------|
-| TS 1.15.0        | Feature       |  13-Oct-2020        | 13-Jan-2021     |
-| TS 1.16.0        | Feature       |  20-Nov-2020        | 20-Feb-2021     |
 | TS 1.17.0        | Feature       |  12-Jan-2021        | 12-Apr-2021     |
+| TS 1.18.0        | Feature       |  23-Feb-2021        | 23-May-2021     |
 
 Versions no longer supported:
 
@@ -39,5 +38,7 @@ Versions no longer supported:
 | TS 1.12.0        | Feature       |  02-Jun-2020        | 02-Sep-2020     |
 | TS 1.13.0        | Feature       |  21-Jul-2020        | 21-Oct-2020     |
 | TS 1.14.0        | Feature       |  01-Sep-2020        | 01-Dec-2020     |
+| TS 1.15.0        | Feature       |  13-Oct-2020        | 13-Jan-2021     |
+| TS 1.16.0        | Feature       |  20-Nov-2020        | 20-Feb-2021     |
 
 See the [Release notes](https://github.com/F5Networks/f5-telemetry-streaming/releases) and [Telemetry Streaming documentation](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html) for new features and issues resolved for each release.
diff --git a/contributing/README.md b/contributing/README.md
@@ -108,7 +108,7 @@ How does the project handle a typical `POST` request?
             "trace": false,
             "format": "default"
         },
-        "schemaVersion": "1.17.0"
+        "schemaVersion": "1.18.0"
     }
 }
 ```

diff --git a/contributing/process_release.md b/contributing/process_release.md
@@ -52,6 +52,7 @@
     * 1.15.0 - 10.9 MB
     * 1.16.0 - 11.3 MB
     * 1.17.0 - 13.1 MB (NOTE: grpc module deps increase)
+    * 1.18.0 - 13.3 MB
   * Install build to BIG-IP, navigate to folder `/var/config/rest/iapps/f5-telemetry/` and check following:
     * Run `du -sh` and check that folder's size (shouldn't be much greater than previous versions):
       * 1.4.0 - 65 MB
@@ -68,6 +69,7 @@
       * 1.15.0 - 79 MB
       * 1.16.0 - 82 MB
       * 1.17.0 - 95 MB (NOTE: grpc module deps increase)
+      * 1.18.0 - 100 MB
     * Check `nodejs/node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
 * Ensure that all tests (unit tests and functional tests passed)
 * Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:

diff --git a/docs/conf.py b/docs/conf.py
@@ -78,7 +78,7 @@
 # The short X.Y version.
 version = u''
 # The full version, including alpha/beta/rc tags.
-release = u'1.17.0'
+release = u'1.18.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/docs/declarations.rst b/docs/declarations.rst
@@ -2,7 +2,7 @@
 
 Example Declarations
 ====================
-This section contains example Telemetry Streaming declarations.  Use the index on the left to go directly to a specific declaration.
+This section contains example Telemetry Streaming declarations.  Use the index on the right to go directly to a specific declaration.
 
 
 Base Declaration

diff --git a/docs/event-listener.rst b/docs/event-listener.rst
@@ -5,6 +5,9 @@ Event Listener class
 
 The Telemetry Streaming Event Listener collects event logs it receives on the specified port from configured BIG-IP sources, including LTM, ASM, AFM, APM, and AVR.
 
+.. NOTE:: Each **Telemetry_Event_Listener** opens 3 ports: TCP (dual stack - IPv4 and IPv6), UDPv4, and UDPv6 |br| If two or more Event Listeners use same port, all of them receive same events, but you can still use filters for each listener individually.
+
+
 To use the Event Listener, you must:
 
 1. Configure the sources of log/event data. You can do this by either POSTing a single AS3 declaration or you can use TMSH or the GUI to configure individual modules.

diff --git a/docs/namespaces.rst b/docs/namespaces.rst
@@ -12,9 +12,9 @@ The following are important notes about namespaces.
 - Each namespace works separately from another and cannot share configuration or object references. 
 - While each namespace must have a unique name, the components in a namespace can share the same name as the components in another namespace. 
 - Namespaces are not tied in any way to RBAC. 
-- Currently, Telemetry Streaming only supports a full declaration sent to **/telemetry/declare**. If there are multiple namespaces, they must all be declared in the POST body. 
+- You must send a full declaration to **/telemetry/declare**. If there are multiple namespaces, you must declare them all in the POST body, otherwise, they are omitted. To configure a single namespace, see :ref:`namespaceEP`.
 - All namespaces inherit the top level **controls** object. 
-- For pull consumers: If a pull consumer is declared under a namespace, the URI to get the data should specify the namespace in path, for example **/mgmt/shared/telemetry/namespace/${namespaceName} pullconsumer/${pullConsumerName}**
+- For pull consumers: If you declare a pull consumer under a namespace, the URI to get the data should specify the namespace in path, for example **/mgmt/shared/telemetry/namespace/${namespaceName}/pullconsumer/${pullConsumerName}**
 
 
 The following examples show how you can use namespaces in your Telemetry Streaming declarations.
@@ -23,6 +23,8 @@ Basic declaration with namespace only
 -------------------------------------
 In this example, all objects are in the namespace named **My_Namespace**.  Because there is only one namespace, except for the name, this is essentially the same as if there were no namespace specified.
 
+This example uses the **/telemetry/declare** endpoint.
+
 .. literalinclude:: ../examples/declarations/basic_namespace.json
     :language: json
 
@@ -32,7 +34,7 @@ Multiple namespaces in a declaration
 ------------------------------------
 In this example, we show how you can use multiple namespaces in a declaration.  This shows how namespaces can be used to group components by function.
 
-Note that the Consumers in each namespace are using the same name (highlighted in the example).
+Note that the Consumers in each namespace are using the same name (highlighted in the example).  This example also uses the **/telemetry/declare** endpoint.
 
 
 .. literalinclude:: ../examples/declarations/multiple_namespaces.json
@@ -50,4 +52,99 @@ The lines that are not highlighted in the example are all part of the default na
 
 .. literalinclude:: ../examples/declarations/default_and_custom_namespace.json
     :language: json
-    :emphasize-lines: 24-37
+    :emphasize-lines: 24-37
+
+|
+
+.. _namespaceEP:
+
+Namespace-specific endpoints
+----------------------------
+Telemetry Streaming 1.18 and later introduced new endpoints specific to individual namespaces. Using this endpoint allows you to configure a specific namespace without needing to know about other namespaces. 
+
+The following table describes the endpoint and request types you can use.
+
++------------------------------------------------------------+--------------+---------------------------------------------------------------------------------------------------+
+| URI                                                        | Request Type | Description                                                                                       |
++============================================================+==============+===================================================================================================+
+| /mgmt/shared/telemetry/namespace/${namespace_name}/declare | GET          | - Returns the single Telemetry Namespace object (configuration data), referenced by name          |
++                                                            +--------------+---------------------------------------------------------------------------------------------------+
+|                                                            | POST         | - Configures a single Telemetry Namespace class - accepts just a single Telemetry_Namespace class |
++                                                            |              | - Assumes defaults/existing configuration for Controls and Telemetry classes                      |
+|                                                            |              |                                                                                                   |
++------------------------------------------------------------+--------------+---------------------------------------------------------------------------------------------------+
+
+|
+
+For example, we use the new endpoint, and POST the following declaration to ``https://{{host}}/mgmt/shared/telemetry/namespace/NamespaceForEvents/declare``
+
+.. code-block:: json
+
+   {
+        "class": "Telemetry_Namespace",
+        "My_Listener": {
+            "class": "Telemetry_Listener",
+            "port": 6514,
+            "trace": true
+        },
+        "Elastic": {
+            "class": "Telemetry_Consumer",
+            "type": "ElasticSearch",
+            "host": "192.168.10.10",
+            "protocol": "http",
+            "port": "9200",
+            "apiVersion": "6.5",
+            "index": "eventdata",
+            "enable": true,
+            "trace": true
+        }
+    }
+
+|
+
+And we receive the following response:
+
+.. code-block:: json
+
+   {
+        "message": "success",
+        "declaration": {
+            "class": "Telemetry_Namespace",
+            "My_Listener": {
+                "class": "Telemetry_Listener",
+                "port": 6514,
+                "trace": true,
+                "enable": true,
+                "match": "",
+                "actions": [
+                    {
+                        "setTag": {
+                            "tenant": "`T`",
+                            "application": "`A`"
+                        },
+                        "enable": true
+                    }
+                ]
+            },
+            "Elastic": {
+                "class": "Telemetry_Consumer",
+                "type": "ElasticSearch",
+                "host": "192.168.10.10",
+                "protocol": "http",
+                "port": 9200,
+                "apiVersion": "6.5",
+                "index": "eventdata",
+                "enable": true,
+                "trace": true,
+                "allowSelfSignedCert": false,
+                "dataType": "f5.telemetry"
+            }
+        }
+    }
+
+|
+
+You receive the same output as response above when you send a GET request to ``https://{{host}}/mgmt/shared/telemetry/namespace/NamespaceForEvents/declare``.
+
+
+
diff --git a/docs/revision-history.rst b/docs/revision-history.rst
@@ -11,6 +11,10 @@ Document Revision History
         - Description
         - Date
 
+      * - 1.18.0
+        - Updated the documentation for Telemetry Streaming v1.18.0. This release contains the following changes: |br| * Added new endpoints for individual namespaces (see :ref:`Namespace endpoints<namespaceEP>`)  |br| |br| Issues Resolved: |br| * Fix Event Listener startup errors that might cause restnoded to crash |br| * Splunk multiEvent format should ignore 'References'
+        - 2-23-21
+
       * - 1.17.0
         - Updated the documentation for Telemetry Streaming v1.17.0. This release contains the following changes: |br| * Added support for configuring proxy settings on Generic HTTP consumers, `GitHub #92 <https://github.com/F5Networks/f5-telemetry-streaming/issues/92>`_ (see :ref:`proxy`) |br| * Added support for configuring proxy settings on Splunk consumers, `GitHub #85 <https://github.com/F5Networks/f5-telemetry-streaming/issues/85>`_ (see :ref:`splunkproxy`) |br| * Added a timestamp for APM Request Log output, `GitHub #91 <https://github.com/F5Networks/f5-telemetry-streaming/issues/91>`_  (see :ref:`APM Request Log<apm-rl>`) |br| * Added support for TLS client authentication to the Kafka consumer, `GitHub #90 <https://github.com/F5Networks/f5-telemetry-streaming/issues/90>`_ (see :ref:`kafka-ref`) |br| * Added an F5 Internal Only push consumer for F5 Cloud (see :ref:`F5 Cloud<f5cloud>`) |br| * Added the ability to use the Splunk multi-metric format, currently EXPERIMENTAL (see :ref:`multi-metric`) |br| * Added a new reference for the Telemetry Streaming Default Output (see :ref:`Default Output Appendix<poller-default-output-reference>`) |br| * Tracefile now stores up to 10 items |br| * Added a note to the System Information output page stating there is new pool and virtual server information collected (see :ref:`System Information<sysinfo>`) |br| * Deprecated TS support for the :ref:`Splunk Legacy Format<splunk-legacy>` |br| * Posting a declaration while a previous declaration is still processing now returns an HTTP 503 status code |br| |br| Issues Resolved: |br| * Fixed error where unavailable Custom Endpoint would return HTTP 500
         - 1-12-20

diff --git a/examples/declarations/generic_http_tls_client_auth.json b/examples/declarations/generic_http_tls_client_auth.json
@@ -0,0 +1,27 @@
+{
+    "class": "Telemetry",
+    "My_Consumer": {
+        "class": "Telemetry_Consumer",
+        "type": "Generic_HTTP",
+        "host": "192.0.2.1",
+        "protocol": "https",
+        "port": 443,
+        "path": "/",
+        "method": "POST",
+        "headers": [
+            {
+                "name": "content-type",
+                "value": "application/json"
+            }
+        ],
+        "privateKey": {
+            "cipherText": "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----"
+        },
+        "clientCertificate": {
+            "cipherText": "-----BEGIN CERTIFICATE-----\nMIID...\n-----END CERTIFICATE-----"
+        },
+        "rootCertificate": {
+            "cipherText": "-----BEGIN CERTIFICATE-----\nMIID...\n-----END CERTIFICATE-----"
+        }
+    }
+}