merge v1.15.0 rc branch to master

F5Networks · Oct 12, 2020 · a03933a · a03933a
1 parent 4430606
commit a03933a
Show file tree

Hide file tree

Showing 43 changed files with 2,962 additions and 440 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -404,19 +404,6 @@ teardown_env_aws:
 # END CLOUD                                                  #
 ##############################################################
 
-# Publish to internal artifactory
-# Note: Will publish when new tags are pushed and use the current build in dist directory
-# with the assumption being the rpm has been updated. However even if it hasn't the upload
-# will simply update the existing rpm version/release in artifactory
-publish_rpm_to_artifactory:
-    stage: publish
-    only:
-        - tags
-    tags:
-        - docker-executor
-    script:
-        - source ./scripts/build/publishRpm.sh
-
 # Publish docs to internal pages - note: this job name MUST be 'pages'
 pages:
     image: ${CONTAINTHEDOCS_IMAGE}
@@ -460,22 +447,6 @@ pages:
         - develop
         - doc-release-branch
 
-# Publish docs to clouddocs.f5networks.net
-publish_docs_to_staging:
-    image: ${CONTAINTHEDOCS_IMAGE}
-    stage: doc
-    environment:
-        name: staging
-        url: https://clouddocs.f5networks.net/products/extensions/f5-telemetry-streaming/latest
-    tags:
-        - cm-official-docker-executor
-    only:
-        - docs-staging@automation-toolchain/f5-telemetry
-    script:
-        - aws s3 sync docs/_build/html s3://clouddocs.f5networks.net/products/extensions/f5-telemetry-streaming/latest
-        # - aws s3 cp versions.json s3://clouddocs.f5networks.net/products/extensions/f5-telemetry-streaming/versions.json
-        # create invalidation to clear cloudfront cache
-        - aws cloudfront create-invalidation --distribution-id $AWS_DIST --paths /products/extensions/f5-telemetry-streaming/latest
 
 # Publish docs to clouddocs.f5.com
 publish_docs_to_production:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,17 @@
 # Changelog
 Changes to this project are documented in this file. More detail and links can be found in the Telemetry Streaming [Document Revision History](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html).
 
+## 1.15.0
+### Added
+### Fixed
+- AUTOTOOL-1790: Fixed syslog event hostname parsing for VCMP hosts
+- AUTOTOOL-1768: Resolve memory leak in ElasticSearch consumer, by replacing 'elasticsearch' library with 'request' library
+### Changed
+- AUTOTOOL-1747: Update npm packages (ajv from v6.12.3 to v6.12.4, applicationinsights from v1.8.5 to v1.8.6, aws-sdk from v2.728.0 to 2.749.0, lodash from v4.7.19 to 4.7.20)
+- AUTOTOOL-1894: Update default hostname for StatsD consumer
+
+### Removed
+
 ## 1.14.0
 ### Added
 - AUTOTOOL-1648: Added ability to specify fallback hosts for Generic HTTP consumer (experimental feature)
@@ -9,7 +20,7 @@ Changes to this project are documented in this file. More detail and links can b
 ### Fixed
 - AUTOTOOL-810: Fixed Event Listener parsing when received multiple events
 ### Changed
-- AUTOTOOL-1680: Update npm packages (ajv from v6.12.2 to v6.12.3, aplicationinsights from v1.8.0 to v1.8.5, aws-sdk from v.2.704.0 to 2.728.0, statsd-client from v0.4.4 to v0.4.5)
+- AUTOTOOL-1680: Update npm packages (ajv from v6.12.2 to v6.12.3, applicationinsights from v1.8.0 to v1.8.5, aws-sdk from v.2.704.0 to 2.728.0, statsd-client from v0.4.4 to v0.4.5)
 ### Removed
 
 

diff --git a/README.md b/README.md
@@ -28,6 +28,9 @@ Because Telemetry Streaming has been created and fully tested by F5 Networks, it
 
 Be sure to see the [Support page](SUPPORT.md) in this repo for more details and supported versions of Telemetry Streaming.
 
+### Community Help
+We encourage you to use our [Slack channel](https://f5cloudsolutions.herokuapp.com) for discussion and assistance on Telemetry Streaming (click the **telemetry-streaming** channel). There are F5 employees who are members of this community who typically monitor the channel Monday-Friday 9-5 PST and will offer best-effort assistance. See the [Slack Channel Statement](slack-channel-statement.md) for guidelines on using this channel.
+
 ## Copyright
 
 Copyright 2014-2020 F5 Networks Inc.

diff --git a/SUPPORT.md b/SUPPORT.md
@@ -19,6 +19,7 @@ Currently supported versions:
 |------------------|---------------|---------------------|-----------------|
 | TS 1.13.0        | Feature       |  21-Jul-2020        | 21-Oct-2020     |
 | TS 1.14.0        | Feature       |  01-Sep-2020        | 01-Dec-2020     |
+| TS 1.15.0        | Feature       |  13-Oct-2020        | 13-Jan-2021     |
 
 Versions no longer supported:
 

diff --git a/contributing/README.md b/contributing/README.md
@@ -108,7 +108,7 @@ How does the project handle a typical `POST` request?
             "trace": false,
             "format": "default"
         },
-        "schemaVersion": "1.14.0"
+        "schemaVersion": "1.15.0"
     }
 }
 ```

diff --git a/contributing/process_release.md b/contributing/process_release.md
@@ -7,6 +7,7 @@
   * RPM sha256 checksum
 * RPM is built in every pipeline run, and is kept in GitLab for one week
 * On a Git tag, RPM is published to Artifactory (f5-telemetry-streaming-rpm)
+* The atg-build project pushes TS RPM builds to Artifactory (f5-automation-toolchain-generic/f5-telemetry)
 * On a release, artifacts are copied from Artifactory to GitHub
 
 ## Release Notes
@@ -49,6 +50,7 @@
     * 1.12.0 - 10.3 MB
     * 1.13.0 - 11 MB
     * 1.14.0 - 11.2 MB
+    * 1.15.0 - 10.9 MB
   * Install build to BIG-IP, navigate to folder `/var/config/rest/iapps/f5-telemetry/` and check following:
     * Run `du -sh` and check that folder's size (shouldn't be much greater than previous versions):
       * 1.4.0 - 65 MB
@@ -62,14 +64,22 @@
       * 1.12.0 - 80 MB
       * 1.13.0 - 81 MB
       * 1.14.0 - 82 MB
+      * 1.15.0 - 79 MB
     * Check `nodejs/node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
 * Ensure that all tests (unit tests and functional tests passed)
-* Create pre-release tag and push it to GitLab:
-  * git tag -m 'Release candidate X.Y.Z-#' vX.Y.Z-#
-  * git push origin
-  * git push origin --tags
-* Check pipeline for artifactory URL to package (or browse in artifactory)
-* Send release candidate email with features, bugs, artifactory URL
+* Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:
+  * git tag -l -n
+  * git tag | xargs -n1 git tag -d
+  * git fetch --tags
+
+**In the atg-build repository:**
+* Update the `TS` CI Schedule, updating the `gitBranch` CI variable to point to the release candidate branch in the `TS` repo (ex: "vX.Y.Z")
+* Run the `TS` schedule. The `TS` schedule will run a pipeline that will programmatically:
+  * run unit tests in the `TS` repository
+  * git tag the next version
+  * update the git tags in the `TS` repository
+  * push the new build artifacts to the 'f5-automation-toolchain-generic/f5-telemetry' Artifactory repository
+  * send the release candidate email with features, bugs, artifactory URL
 
 ## Release process
 
@@ -84,25 +94,33 @@
   * git checkout master
   * git merge --squash rc-master-branch
   * git push origin
+* Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:
+  * git tag -l -n
+  * git tag | xargs -n1 git tag -d
+  * git fetch --tags
+* Check master history
+  * commits should be squashed
+  * check for any sensitive info in remaining commit messages
 * Create tag in master branch:
   * git checkout master
   * git tag -m 'Release X.Y.Z' vX.Y.Z
-  * git push origin --tags
-* Push to GitHub master:
-  * Create the GitHub remote (as needed):
-    * git remote add github https://github.com/f5networks/f5-telemetry-streaming.git
-  * git push github master
-  * git push github --tags
+  * git push origin tag vX.Y.Z
 * Merge GitLab master back into develop:
   * git checkout develop
   * git merge master
   * git push origin
   * Now you can remove RC branch
-* Do not forget to clean up stale branches, e.g. RC branches
+* Remove all RC tags and branches and other stale branches that were used for release or RC process
+
+### GitHub Publishing
+* Push to GitHub master:
+  * Create the GitHub remote (as needed):
+    * git remote add github https://github.com/f5networks/f5-telemetry-streaming.git
+  * git push github master
+  * git push github tag vX.Y.Z
 * Create GitHub release - [GitHub Releases](https://github.com/f5networks/f5-telemetry-streaming/releases)
   * Navigate to the latest release, select `edit` and upload artifacts:
     * `.rpm` file
     * `.sha256` file
-* Remove all RC tags and branches and other stale branches that were used for release or RC process
 
 # ATTENTION: DO NOT FORGET TO MERGE 'MASTER' BRANCH INTO 'DEVELOP' WHEN YOU ARE DONE WITH RELEASE PROCESS
diff --git a/docs/avr.rst b/docs/avr.rst
@@ -10,8 +10,13 @@ As of TS version 1.3.0, you can now export AVR data. The TS declaration will be
 - AVR should be configured to send data to TS
 - The Analytics profile for HTTP or TCP should be configured and assigned to the virtual server
 
-.. NOTE:: To see more information on AVR, see the |analytics|.
+To see more information on AVR, see the |analytics|.
 
+.. _avr-note:
+
+.. NOTE:: It is currently not possible to configure AVR to publish data to both Telemetry Streaming and BIG-IQ concurrently. Additionally, if a BIG-IP is managed by a BIG-IQ, and has Statistics Collection enabled, the entire AVR configuration of the BIG-IP will be overwritten to publish only to the BIG-IQ.
+
+|
 
 Modify system logging configuration to update what gets logged by running the following commands in TMSH:
 

diff --git a/docs/conf.py b/docs/conf.py
@@ -78,7 +78,7 @@
 # The short X.Y version.
 version = u''
 # The full version, including alpha/beta/rc tags.
-release = u'1.14.0'
+release = u'1.15.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/docs/faq.rst b/docs/faq.rst
@@ -124,6 +124,14 @@ As of January 2020, no breaking changes have been made to AS3, Declarative Onboa
 
 Beginning with TS v1.10.0, you can use Telemetry Streaming on F5 devices with multiple hosts, such as the Viprion platform and vCMP systems.  In versions prior to v1.10, devices with multiple hosts were not supported.
 
+|
+
+.. _poolstats:
+
+**Why am I seeing a decrease in some pool level metrics?**
+
+The BIG-IP system tracks pool level metrics by aggregating node metrics. If a node is deleted, you will see a decrease in some pool metrics, such as the bits in and out.
+
 
 .. |intro| raw:: html
 

diff --git a/docs/images/beacon-logo.png b/docs/images/beacon-logo.png
diff --git a/docs/images/statsd_logo.png b/docs/images/statsd_logo.png
diff --git a/docs/index.rst b/docs/index.rst
@@ -2,7 +2,9 @@ F5 Telemetry Streaming
 ======================
 
 Welcome to the F5 Telemetry Streaming User Guide.
-To provide feedback on this documentation, you can file a GitHub Issue, or email us at [email protected].
+To provide feedback on this documentation, you can file a GitHub Issue, or email us at [email protected].  
+
+For information on supported versions of Telemetry Streaming, see |supportmd|.
 
 Introduction
 ------------
@@ -59,6 +61,9 @@ You can click the following links to go directly to a specific :doc:`setting-up-
       * - :ref:`http-ref`
         - 
 
+      * - :ref:`beacon-ref`
+        - 
+
       * - :ref:`fluentd-ref`
         - 
 
@@ -85,14 +90,17 @@ You can click the following links to go directly to a specific :doc:`setting-up-
    output-example
    data-modification
    troubleshooting
-   revision-history
    schema-reference
    custom-endpoints
+   revision-history
 
 
 
 .. |video| raw:: html
 
+    <iframe width="560" height="315" src="https://www.youtube.com/embed/YeEAovBvUkY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+
 
+.. |supportmd| raw:: html
 
-    <iframe width="560" height="315" src="https://www.youtube.com/embed/YeEAovBvUkY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+   <a href="https://github.com/F5Networks/f5-telemetry-streaming/blob/master/SUPPORT.md" target="_blank">Support information on GitHub</a>
diff --git a/docs/pull-consumers.rst b/docs/pull-consumers.rst
@@ -90,6 +90,21 @@ This example shows how to use the Prometheus pull consumer. For this pull consum
 
 The Prometheus Pull Consumer outputs the telemetry data according to the Prometheus data model specification. For more information about the Prometheus data model, see https://prometheus.io/docs/concepts/data_model/.
 
+**NOTE**: To access the consumer endpoint with a user other than **admin**, refer to |dcarticle|. When you create the resource group while following this guide, you will need a resource group similar to the following:
+
+.. code-block:: json
+
+   {
+      "name": "prometheusResourceGrup",
+      "resources": [
+         {
+               "restMethod": "GET",
+               "resourceMask": "/mgmt/shared/telemetry/pullconsumer/My_Pull_Consumer"
+         }
+      ]
+   }
+
+|
 
 Example declaration:
 
@@ -111,3 +126,8 @@ Example declaration:
    :target: https://prometheus.io/
    :alt: Prometheus
 
+.. |dcarticle| raw:: html
+
+   <a href="https://devcentral.f5.com/s/articles/icontrol-rest-fine-grained-role-based-access-control-30773" target="_blank">this article on DevCentral</a>
+
+
diff --git a/docs/revision-history.rst b/docs/revision-history.rst
@@ -11,8 +11,12 @@ Document Revision History
         - Description
         - Date
 
+      * - 1.15.0
+        - Updated the documentation for Telemetry Streaming v1.15.0. This release contains the following changes: |br| * Updated the default hostname for the StatsD consumer (see the Important note in :ref:`statsd-ref`) |br| * Added a note to the :ref:`prometheus` consumer on how to access the consumer endpoint with a user other than **admin** |br| * Added a new FAQ entry on why you may see a decrease in some pool statistics (see :ref:`poolstats`)  |br| |br| Issues Resolved: |br| * Fixed syslog event hostname parsing for VCMP hosts |br| * Resolve memory leak in ElasticSearch consumer, by replacing 'elasticsearch' library with 'request' library
+        - 10-13-20
+
       * - 1.14.0
-        - Updated the documentation for Telemetry Streaming v1.14.0. This release contains the following changes: |br| * Added support for AWS CloudWatch Metrics (see :ref:`cw-metrics`) |br| * Added an EXPERIMENTAL feature to specify fallback hosts for generic HTTP consumers (see :ref:`fallback`) |br| * Added F5 Beacon as a Generic HTTP consumer (see :ref:`beacon-ref`) |br| *  |br| * Added **cipherUses** stats for Client and Server SSL profiles (see the cipherUses lines in :ref:`sysinfo`)  |br| |br| Issues Resolved: |br| * Fixed Event Listener parsing when receiving multiple events
+        - Updated the documentation for Telemetry Streaming v1.14.0. This release contains the following changes: |br| * Added support for AWS CloudWatch Metrics (see :ref:`cw-metrics`) |br| * Added an EXPERIMENTAL feature to specify fallback hosts for generic HTTP consumers (see :ref:`fallback`) |br| * Added F5 Beacon as a Generic HTTP consumer (see :ref:`beacon-ref`)  |br| * Added **cipherUses** stats for Client and Server SSL profiles (see the cipherUses lines in :ref:`sysinfo`) |br| * Added a troubleshooting entry for a restjavad issue (see :ref:`restjavad`) |br| * Added a note to the :doc:`avr` page about AVR caveats with TS and BIG-IQ (see the :ref:`AVR note<avr-note>`) |br| * Updated the introduction for the experimental :doc:`custom-endpoints` |br| * Added support for BIG-IP 16.0  |br| |br| Issues Resolved: |br| * Fixed Event Listener parsing when receiving multiple events
         - 09-01-20
 
       * - 1.13.0

diff --git a/docs/setting-up-consumer.rst b/docs/setting-up-consumer.rst
@@ -138,7 +138,7 @@ Optional Properties:
 - **App Insights Resource Name**: Name filter used to determine to which App Insights resource to send metrics. If not provided, TS will send metrics to App Insights in the subscription in which the managed identity has permissions. Note: To be used only when useManagedIdentity is true.
 - **customOpts**: Additional options for use by consumer client library. These are passthrough options (key value pair) to send to the Microsoft node client. 
 
-   .. WARNING:: The customOpts options are not guaranteed to work and may change according to the client library API; you must use these options with caution. Refer to corresponding consumer library documentation for acceptable keys and values.
+.. WARNING:: The **customOpts** options are not guaranteed to work and may change according to the client library API; you must use these options with caution. Refer to corresponding consumer library documentation for acceptable keys and values.
 
 To see more information about Azure Application Insights, see |appinsight|.
 
@@ -316,6 +316,8 @@ ElasticSearch
 -------------
 |ElasticSearch|
 
+.. IMPORTANT:: TS currently does not support sending data to ElasticSearch 7.
+
 Required Information:
  - Host: The address of the ElasticSearch system.
  - Index: The index where data should go within the ElasticSearch system.
@@ -372,7 +374,10 @@ Required Information:
  - Protocol: The protocol of the StatsD instance. Options: TCP (TS 1.12+) or UDP. The default is UDP.
  - Port: The port of the Statsd instance
 
-.. NOTE:: To see more information about installing StatsD, see |StatsDWiki|.
+.. IMPORTANT:: In TS v1.15 and later, if Telemetry Streaming is unable to locate the hostname in the systemPoller data, it sends the metric with hostname value **host.unknown**. This gets transformed to **hostname-unknown** as required by StatsD. This is because StatsD uses a **.** as a delimiter, and TS automatically replaces it with a **-**. For example: |br| |bold| { |br| |sp| |sp| |sp| metricName: 'f5telemetry.hostname-unknown.customStats.clientSideTraffic-bitsIn', |br| |sp| |sp| |sp| metricValue: 111111030 |br| } |boldclose|
+
+
+To see more information about installing StatsD, see |StatsDWiki|.
 
 .. literalinclude:: ../examples/declarations/statsd.json
     :language: json
@@ -402,6 +407,8 @@ To see an example of the EXPERIMENTAL feature where you can specify fallback IP
 .. literalinclude:: ../examples/declarations/generic_http.json
     :language: json
 
+.. _beacon-ref:
+
 F5 Beacon
 ^^^^^^^^^
 F5 Beacon, a SaaS offering, provides visibility and actionable insights into the health and performance of applications. 
@@ -648,6 +655,10 @@ In the following table, we list the Azure Government regions.
    :target: https://cloud.google.com/products/operations
    :alt: Google Cloud
 
+.. |beaconlogo| image:: /images/beacon-logo.png
+   :target: https://www.f5.com/products/beacon-visibility-and-analytics
+   :alt: F5 Beacon
+
 
 .. |Azure documentation| raw:: html
 
@@ -717,6 +728,18 @@ In the following table, we list the Azure Government regions.
 
    <br />
 
+.. |sp| raw:: html
+
+   &nbsp;
+
+.. |bold| raw:: html
+
+   <strong>
+
+.. |boldclose| raw:: html
+
+   </strong>
+
 .. |managedid| raw:: html
 
    <a href="https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview" target="_blank">Microsoft documentation</a>