System Poller stops sending logs to Telemetry when a new image is created into a new volume on bigip

[thepowercoders]

Environment

• Telemetry Streaming Version: 1.33.0
• BIG-IP Version: 16.1.4.1 Build 0.117.5
• Telemetry Destination: Microsoft Azure Log Analytics

Summary

After a bigip device has a image or hotfix uploaded to it and this image is used to make a new volume, the System Poller in Telemetry Streaming stops sending any logs to the F5Telemetry_system_CL custom log in Azure Log Analytics.

Steps To Reproduce

Steps to reproduce the behavior:

1. Setup TS to Azure LAW with the following declaration:

{
    "class": "Telemetry",
    "controls": {
        "class": "Controls",
        "logLevel": "info",
        "debug": false
    },
    "tel-system-azuresentinel": {
        "class": "Telemetry_System",
        "trace": false,
        "allowSelfSignedCert": true,
        "host": "localhost",
        "port": 8100,
        "protocol": "http",
        "systemPoller": "tel-syspoll-azuresentinel"
    },
    "tel-syspoll-azuresentinel": {
        "class": "Telemetry_System_Poller",
        "interval": 60
    },
    "tel-listener-azuresentinel": {
        "class": "Telemetry_Listener",
        "port": 6514,
        "enable": true,
        "trace": false,
        "match": "",
        "actions": [
            {
                "setTag": {
                    "tenant": "`T`",
                    "application": "`A`"
                },
                "enable": true
            }
        ]
    },
    "tel-pushcon-azuresentinel": {
        "class": "Telemetry_Consumer",
        "type": "Azure_Log_Analytics",
        "format": "propertyBased",
        "trace": false,
        "workspaceId": "<WORKSPACE-ID>",
        "passphrase": {
            "cipherText": "<WORKSPACE-PRIMARY-KEY>"
        },
        "useManagedIdentity": false,
        "region": "uksouth"
    }
}

2. Observe Telemetry Streaming is working correctly and the system-poller is sending telemetry into Log Analytics to the custom log F5Telemetry_system_CL
3. Upload a new image or hotfix on the bigip to System ›› Software Management : Image List or System ›› Software Management : Hotfix List
4. Click on 'Install' button and select a new volume to install the image onto
5. Observe after this process completes, Telemetry streaming for the system-poller has stopped and no further logs are sent to Azure for system (other logging continues fine).

Expected Behavior

Background loading of new images and hotfixes should have no impact to telemetry streaming (or any system impact at all).

Actual Behavior

When a new volume is created, the following logs are observed in /var/log/ltm :

Nov 28 17:03:12 bigip001 warning tmsh[29691]: 01420013:4: [api-status-warning] security/log/profile, properties : deprecated : application/local-storage
Nov 28 17:03:12 bigip001 warning tmsh[29691]: 01420013:4: [api-status-warning] sys/datastor is deprecated
Nov 28 17:03:13 bigip001 warning tmsh[29691]: 01420013:4: [api-status-warning] sys/ecm/cloud-provider is deprecated
Nov 28 17:03:13 bigip001 warning tmsh[29691]: 01420013:4: [api-status-warning] wom/deduplication is deprecated
Nov 28 17:03:13 bigip001 warning tmsh[29691]: 01420013:4: [api-status-warning] wom/endpoint-discovery is deprecated
Nov 28 17:03:13 bigip001 warning tmsh[29691]: 01420013:4: Per-invocation log rate exceeded; throttling.
Nov 28 17:03:20 bigip001 info iAppsLX_save_pre[29839]: Exporting: f5-service-discovery - /var/config/rest/iapps/f5-service-discovery
Nov 28 17:05:27 bigip001 info iAppsLX_save_pre[29839]: Exporting: f5-declarative-onboarding - /var/config/rest/iapps/f5-declarative-onbo
arding
Nov 28 17:05:42 bigip001 info iAppsLX_save_pre[29839]: Exporting: f5-appsvcs - /var/config/rest/iapps/f5-appsvcs
Nov 28 17:05:58 bigip001 info iAppsLX_save_pre[29839]: Exporting: f5-telemetry - /var/config/rest/iapps/f5-telemetry
Nov 28 17:08:45 bigip001 notice tmsh[29687]: 01420012:5: private key export: All keys are being exported by user "root" via UCS saving at file "/var/tmp/gdUMm7fzTq.ucs".

There is no other indication why logging has stopped.