Data of type F5Telemetry_virtualServers (and same for F5Telemetry_AVR) was dropped: The number of custom fields xxx is above the limit of 500 fields per data type

[danielcayer]

Environment

• Telemetry Streaming Version:
• BIG-IP Version:

Summary

The F5Telemetry_virtualServers table in Microsoft Sentinel is not populating even when utilizing the workaround proposed in article K42151385 (https://support.f5.com/csp/article/K42151385). All corresponding messages for tables F5Telemetry_virtualServers and also F5Telemetry_AVR from my BIG-IP using TS result in errors which can be seen in the Azure portal.
I am under impression this issue should have been resolved since the 1.24.0 release of TS (as per final comment in issue number 156 - #156)
This issue is easily reproducible. The impact is low since the alternate workaround is sufficient for the time being (as documented here: https://support.f5.com/csp/article/K42151385).

Steps To Reproduce

Steps to reproduce the behavior:

1. Submit the following declaration:

curl -sku username:password --location --request POST 'https://Management_IP_Address/mgmt/shared/telemetry/declare' \
--header 'Content-Type: application/json' \
--data-raw '{
    "class": "Telemetry",
    "controls": {
        "class": "Controls",
        "logLevel": "debug",
        "debug": true
    },
    "My_Listener": {
    "class": "Telemetry_Listener",
    "port": 6514
    },
    "Poller": {
    "class": "Telemetry_System_Poller",
    "interval": 60,
    "enable": true,
    "trace": false,
    "allowSelfSignedCert": false,
    "host": "localhost",
    "port": 8100,
    "protocol": "http",
    "actions": [
    {
    "enable": true,
    "includeData": {},
    "locations": {
    "system": true,
    "virtualServers": true,
    "httpProfiles": true,
    "clientSslProfiles": true,
    "serverSslProfiles": true
    }
    }
    ]
    },
    "Pull_Consumer": {
    "class": "Telemetry_Pull_Consumer",
    "type": "default",
    "systemPoller": [
    "Poller"
    ]
    },
   
    "Azure_Consumer": {
    "class": "Telemetry_Consumer",
    "type": "Azure_Log_Analytics",
    "workspaceId": "xxxxxxxxxxxxxxxxxxxxx",
    "passphrase": {
    "cipherText": " xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx="
    },
   "format": "propertyBased"
    },
    "schemaVersion": "1.12.0"
}'

2. Observe the following error response:
   In the Azure portal browse to Microsoft Sentinel -> [click to select your workspace] -> Genera/Logs and run the "operation" query.

OperationStatus: Failed

Detail: Data of type F5Telemetry_virtualServers was dropped: The number of custom fields 501 is above the limit of 500 fields per data type. See https://aka.ms/AA593as to find instructions for removing unnecessary custom fields for this type.

OperationCategory: Ingestion
OperationKey: Custom log ingestion
CorrelationId: 5f384089-cba0-41ae-9846-ddb60b571b64
Type: Operation

Expected Behavior

All LTM and AVR logs should be logged in Azure Log Analytics and Sentinel without exceeding the 500 fields limit.

Actual Behavior

The Virtual server stats are not logged in Azure.
The following error is reported in Azure log:
Data of type F5Telemetry_virtualServers was dropped: The number of custom fields 501 is above the limit of 500 fields per data type. See https://aka.ms/AA593as to find instructions for removing unnecessary custom fields for this type.

[joernott]

The same happens with elasticsearch as well:

Fri, 18 Nov 2022 12:55:16 GMT - finest: [telemetry.ElasticSearch.f5telemetry_default::Elasticsearch_cluster] response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded while adding new fields [1001]"}},"status":400}

I edited the elasticsearch index mapping (see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) and set "index.mapping.total_fields.limit": "10000", as even 2k was not enough.

It currently creates 3177 fields on my test ltm which is basically empty

[joernott]

I am currently working with the data modification feature (see https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/data-modification.html) to limit the output to the relevant fields. Maybe, that is also a way for you to reduce the number of fields.

[joernott]

Using the following actions, I have reduced the number of fields from 3177 to 674 for 16 pools with 1 member each. There are still a lot of superfluous infos in there, so it should be possible to condense it even more.

    "My_System": {
        "class": "Telemetry_System",
        "systemPoller": {
            "interval": 60,
            "actions": [
                {
                    "includeData": {},
                    "locations": {
                        "pools": {
                            ".*": {
                                "activeMemberCnt": true,
                                "availabilityState": true,
                                "enabledState": true,
                                "members": {
                                    ".*": {
                                        "availabilityState": true,
                                        "enabledState": true,
                                        "monitorStatus": true,
                                        "serverside": true
                                    }
                                },
                                "serverside": true,
                                "status": true,
                                "totRequests": true
                            }
                        },
                        "system": {
                            "connectionsPerformance": {
                                ".*": true
                            },
                            "cpu": true,
                            "memory": true,
                            "networkInterfaces": true,
                            "swap": true,
                            "throughputPerformance": true,
                            "tmmCpu": true,
                            "tmmMemory": true,
                            "tmmTraffic": true
                        },
                        "virtualservers": {
                            ".*": {
                                "availabilityState": true,
                                "clientSide": true,
                                "totRequests": true
                            }
                        }
                    }
                }
            ]
        }
    },

[DahlPatric]

Helped me to solve 500 limit.

    "actions": [
          {
            "includeData": {},
            "locations": {
              "telemetryServiceInfo": true,
              "system": true,
              "virtualServers": {
                ".*": {
                  "availabilityState": true,
                  "clientside": true,
                  "status.statusReason": true,
                  "totRequests": true,
                  "name": true
                }
              },
              "pools": true
            }
          }
        ]
      }