chore: v1.37.0

CHANGELOG.md

@@ -7,6 +7,13 @@ Changes to this project are documented in this file. More detail and links can b
 ### Changed
 ### Removed
 
+## 1.37.0
+### Added
+- MBIPMP-41258: Update Azure Log Analytics with new `propertyBasedV2` format that converts `system.asmAttackSignatures` into an array
+### Fixed
+### Changed
+### Removed
+
 ## 1.36.0
 **Important**: Starting from BIG-IP Telemetry Streaming version 1.36.0, BIG-IP Telemetry Streaming no longer supports BIG-IP 13.1 to 15.0.x. However, if you are still using the BIG-IP 13.1 to 15.0.x versions, you can use BIG-IP Telemetry Streaming 1.35.0 or earlier.
 ### Added

SUPPORT.md

@@ -25,6 +25,7 @@ Currently supported versions:
 |------------------|------------------------|---------------------|------------------|
 | TS 1.33.0        | LTS                    |  22-Mar-2023        | Maintenance mode |
 | TS 1.36.0        | Feature                |  14-Aug-2024        | 14-Nov-2024      |
+| TS 1.37.0        | Feature                |  28-Oct-2024        | 28-Jan-2024      |
 
 Versions no longer supported:
 

contributing/README.md

@@ -108,7 +108,7 @@ How does the project handle a typical `POST` request?
             "trace": false,
             "format": "default"
         },
-        "schemaVersion": "1.36.0"
+        "schemaVersion": "1.37.0"
     }
 }
 ```

contributing/process_release.md

@@ -68,7 +68,7 @@
     * 1.34.0 - 18.4 MB
     * 1.35.0 - 18.4 MB
     * 1.36.0 - 20.6 MB (NOTE: inclusion of new version of Kafka library)
-    * 1.37.0 - ???? MB
+    * 1.37.0 - 20.6 MB
   * Install build to BIG-IP, navigate to folder `/var/config/rest/iapps/f5-telemetry/` and check following:
     * Run `du -sh` and check that folder's size (shouldn't be much greater than previous versions):
       * 1.4.0 - 65 MB
@@ -103,7 +103,7 @@
       * 1.33.0 - 164 MB
       * 1.35.0 - 164 MB
       * 1.36.0 - 154 MB
-      * 1.37.0 - ??? MB
+      * 1.37.0 - 154 MB
     * Check `node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
 * Ensure that all tests (unit tests and functional tests passed)
 * Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:

docs/memory-monitor.rst

@@ -234,6 +234,8 @@ The "runtime" property of Controls class is where you define your runtime config
 
 .. IMPORTANT:: **RUNTIME CONFIGURATION OPTIONS SHOULD BE USED ONLY WHEN YOU ARE OBSERVING/EXPERIENCING MEMORY USAGE ISSUES**
 
+.. IMPORTANT:: **F5 BIG-IP Telemetry Streaming not meant to be used on F5 BIG-IP with FIPS option enabled. F5 BIG-IP Telemetry Streaming may change scripts on the host system that lead to System Integrity check to fail**
+
 .. list-table::
       :widths: 25 25 200
       :header-rows: 1

docs/setting-up-consumer.rst

@@ -167,6 +167,11 @@ The following items have been added to the Azure Log Analytics consumer since it
         - **region**
         - The **region** property for Azure Log Analytics and Application Insights was added in part to support the Azure Government regions. |br| - This optional property is used to determine cloud type (public/commercial, govcloud) so that the correct API URLs can be used (example values: westeurope, japanwest, centralus, usgovvirginia, and so on). |br| - If you do not provide a region, BIG-IP Telemetry Streaming attempts to look it up from the instance metadata. |br| - If it is unable to extract metadata, BIG-IP TS defaults to public/commercial |br| - Check the |azregion| for product/region compatibility for Azure Government. |br| - See the Azure documentation for a valid list of regions (resource location), and :ref:`Region list<azreg>` for example values from the Azure CLI.
 
+      * - 1.37
+        - **format**
+        - **propertyBasedV2** - provides the same output as **propertyBased** with an addition of **system.asmAttackSignatures** moved to its own category **asmAttackSignatures**
+
+
 
 
 To see more information about sending data to Log Analytics, see |HTTP Data Collector API|.
@@ -451,6 +456,28 @@ Required Information:
  - Username: The username to use for authentication process.
  - Password: The password to use for authentication process.
 
+Optional parameters:
+
+.. NOTE:: Available on F5 BIG-IP Telemetry Streaming 1.36.0 and later
+
+- Format: Toggles formatting of data. Options: ``default`` and ``split`` - splits system information into multiple smaller messages.
+- Partitioner Type: Allows the message to be sent using a chosen partitioning strategy. Options:
+    - **default** - uses the default or partition at index 0
+    - **random** - pick from available partitions randomly
+    - **cyclic** - will cycle through the available partitions
+    - **keyed** - use a specific partition with key, a value for ``partitionKey`` must be provided.
+- Partition Key: Key used to lookup a partition. Required when *Partitioner Type* is ``keyed``. Must not be specified if using other partitioner types.
+- CustomOpts: Custom settings to pass to Kafka client. These are a subset of what the https://github.com/SOHU-Co/kafka-node supports. See the example declaration for how to use those options. Options:
+    - **connectRetryOptions.retries** - the maximum amount of times to retry the operation
+    - **connectRetryOptions.factor** - the exponential factor to use
+    - **connectRetryOptions.minTimeout** - the number of milliseconds before starting the first retry
+    - **connectRetryOptions.maxTimeout** - the maximum number of milliseconds between two retries
+    - **connectRetryOptions.randomize** - randomizes the timeouts by multiplying with a **connectRetryOptions.factor**
+    - **connectTimeout** - how much time in milliseconds it takes to wait for a successful connection before moving to the next host
+    - **idleConnection** - allows the broker to disconnect an idle connection from a client. The value is elapsed time in milliseconds without any data written to the TCP socket
+    - **maxAsyncRequests** - maximum async operations at a time toward the Kafka cluster
+    - **requestTimeout** - how much time in milliseconds for a Kafka request to timeout
+
 .. NOTE:: To see more information about installing Kafka, see |Installing Kafka|.
 
 Additions to the Kafka consumer
@@ -468,7 +495,7 @@ The following items have been added to the Kafka consumer since it was introduce
 
       * - 1.17
         - **privateKey**
-        - This and the following properties provide the ability to add TLS client authentication to the Kafka consumer using the **TLS** authentication protocol.  This protocol configures BIG-IP Telemetry Streaming to provide the required private key and certificate(s) when the Kafka broker is configured to use SSL/TLS Client authentication.  You can find more information on Kafka's client authentication on the Confluent pages: https://docs.confluent.io/5.5.0/kafka/authentication_ssl.html. |br| |br| **privateKey** is the Private Key for the SSL certificate. Must be formatted as a 1-line string, with literal new line characters. 
+        - This and the following properties provide the ability to add TLS client authentication to the Kafka consumer using the **TLS** authentication protocol.  This protocol configures BIG-IP Telemetry Streaming to provide the required private key and certificate(s) when the Kafka broker is configured to use SSL/TLS Client authentication.  You can find more information on Kafka's client authentication on the Confluent pages:  https://docs.confluent.io/platform/current/kafka/authentication_ssl.html |br| |br| **privateKey** is the Private Key for the SSL certificate. Must be formatted as a 1-line string, with literal new line characters. 
 
       * - 
         - **clientCertificate**
@@ -478,16 +505,31 @@ The following items have been added to the Kafka consumer since it was introduce
         - **rootCertificate**
         - The Certificate Authority root certificate, used to validate the client certificate. Certificate verification can be disabled by setting allowSelfSignedCert=true. Must be formatted as a 1-line string, with literal new line characters.
 
+      * - 1.36
+        - **customOpts**
+        - Various options to configure Kafka client.
 
+      * - 
+        - **format**
+        - Toggles formatting of data.
 
-**IMPORTANT**: The following declaration includes the additional properties shown in the table. If you attempt to use this declaration on a previous version, it will fail. On previous versions, remove the highlighted line(s), and the comma from the previous line. 
+      * - 
+        - **partitionerType**
+        - Allows the message to be sent using a chosen partitioning strategy.
+
+      * - 
+        - **partitionKey**
+        - Key used to lookup a partition.
+
+
+
+**IMPORTANT**: The following declaration includes the additional properties shown in the table. If you attempt to use this declaration on a previous version, it will fail.
 
 Example Declaration:
 
 .. literalinclude:: ../examples/declarations/consumers/Kafka/kafka.json
     :language: json
     :linenos:
-    :emphasize-lines: 24-41
 
 |
 
@@ -1293,4 +1335,4 @@ In the following table, we list the Azure Government regions.
 
 .. |gcldocs| raw:: html
 
-   <a href="https://cloud.google.com/logging" target="_blank">Google Cloud Logging documentation</a>
+   <a href="https://cloud.google.com/logging" target="_blank">Google Cloud Logging documentation</a>

docs/troubleshooting.rst

@@ -565,6 +565,30 @@ For more information see:
 
 - `Bug ID 1602033 <https://cdn.f5.com/product/bugtracker/ID1602033.html>`_
 
+|
+
+.. _bigipplatformsupport:
+
+Why does not F5 BIG-IP Telemetry Streaming work on platforms with F5OS?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+F5 BIG-IP Telemetry Streaming has never been validated on F5 VELOS platform so F5 BIG-IP Telemetry Streaming is not supported on F5 VELOS.
+
+Some testing are done on new BIG-IP version but platform validation has not been done on new hardware like F5 VELOS and F5 rSeries.
+
+F5 BIG-IP Telemetry Streaming is in maintenance mode, focusing on the next generation of Telemetry innovations. We will continue to update BIG-IP TS with critical security updates.
+
+
+|
+
+.. _telemetrystreamingnofips:
+
+Why the FIPS self-check to fail once F5 BIG-IP Telemetry Streaming installed or modified runtime options?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+F5 BIG-IP Telemetry Streaming not meant to be used on F5 BIG-IP with FIPS option enabled. F5 BIG-IP Telemetry Streaming may change scripts on the host system that lead to
+System Integrity check to fail.
+
 
 .. |br| raw:: html
 

package.json

@@ -1,6 +1,6 @@
 {
     "name": "f5-telemetry",
-    "version": "1.36.0-1",
+    "version": "1.37.0-1",
     "author": "F5, Inc.",
     "license": "Apache-2.0",
     "repository": {

scripts/build/buildRpm.sh

@@ -5,8 +5,8 @@ set -evx
 
 # RPM template: <app-name>-<version>-<release>.<arch>.rpm
 # For DEV <release> === <release>.<buildtimestamp>.<commit-sha>.<sanitized-branch>
-# DEV RPM: f5-telemetry-1.36.0-0.20240107071243.28507f40.dev_build_info.noarch.rpm
-# Release RPM: f5-telemetry-1.36.0-0.noarch.rpm
+# DEV RPM: f5-telemetry-1.37.0-0.20240107071243.28507f40.dev_build_info.noarch.rpm
+# Release RPM: f5-telemetry-1.37.0-0.noarch.rpm
 
 is_release_tag () {(
     node -e "process.exit(+!(/^(v[0-9]+\.[0-9]+\.[0-9]+|latest)$/.test('$1')));"

src/lib/consumers/Azure_Log_Analytics/index.js

@@ -28,7 +28,7 @@ const EVENT_TYPES = require('../../constants').EVENT_TYPES;
 module.exports = function (context) {
     const fullURI = azureUtil.getApiUrl(context, 'opinsights');
     const workspaceId = context.config.workspaceId;
-    const format = context.config.format;
+    const isPropertyBased = (context.config.format || '').startsWith('propertyBased');
     const sharedKey = context.config.passphrase;
     const logType = context.config.logType || 'F5Telemetry';
 
@@ -38,6 +38,14 @@ module.exports = function (context) {
         const copyData = JSON.parse(JSON.stringify(context.event.data));
         context.event.data = {};
         context.event.data[context.event.type] = copyData;
+    } else if (isPropertyBased
+            && context.config.format.endsWith('V2')
+            && context.event.data.system && context.event.data.system.asmAttackSignatures
+    ) {
+        // move 'asmAttackSignatures' to the top level to allow `azureUtil.isConfigItems`
+        // to detect it and `azureUtil.transformConfigItems` to transform it
+        context.event.data.asmAttackSignatures = context.event.data.system.asmAttackSignatures;
+        delete context.event.data.system.asmAttackSignatures;
     }
 
     return Promise.resolve()
@@ -70,7 +78,7 @@ module.exports = function (context) {
                     data = { value: data }; // make data an object
                 }
 
-                if ((format === 'propertyBased')
+                if (isPropertyBased
                         && azureUtil.isConfigItems(data, type, poolMemberMapping.isPoolMembersType(type))) {
                     data = azureUtil.transformConfigItems(data);
                     // If it is a pool, transfer its pool members to the pool members table of the corresponding type.

src/lib/consumers/shared/azureUtil.js

@@ -373,7 +373,7 @@ function isConfigItems(data, type, isPoolMembersType) {
     }
 
     // is it of type sslCerts or keys are of format of format /.../...
-    if (type === 'sslCerts' || Object.keys(data).every((key) => /\/[^/]*\/.*/.test(key))) {
+    if (type === 'sslCerts' || type === 'asmAttackSignatures' || Object.keys(data).every((key) => /\/[^/]*\/.*/.test(key))) {
         // check that the key is the same as property 'name'
         return Object.keys(data)
             .every((key) => typeof data[key] === 'object' && key === data[key].name);