Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to use Kubelet to fetch metadata in Kubernetes filter in version 3 #9885

Open
nilushancosta opened this issue Jan 28, 2025 · 0 comments
Open

Unable to use Kubelet to fetch metadata in Kubernetes filter in version 3 #9885

nilushancosta opened this issue Jan 28, 2025 · 0 comments
Labels
status: waiting-for-triage

Comments

@nilushancosta
Copy link

Bug Report

Describe the bug
I am trying to use the Kubelet to fetch metadata in the Kubernetes filter (i.e. Use_Kubelet true). I noticed that as per startup logs, the host used by the Kubernetes filter does not change to the Kubelet in Fluent Bit version 3. It uses the API server as the host. However, with Fluent Bit version 2, the Kubernetes filter uses the Kubelet as expected.

To Reproduce
Install Fluent Bit using the helm chart with different values files.

helm install fluent-bit fluent/fluent-bit --version 0.48.0 -f values.yaml

Please find the different values files I used and the corresponding startup logs below (Please refer the [filter:kubernetes:kubernetes.0] log lines)

1. Fluent Bit 3 with metadata from Kubelet

values

config:
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Use_Kubelet true

  outputs: |
    [OUTPUT] 
        Name stdout 
        Match *

image:
  tag: 3.2.5
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
Fluent Bit v3.2.5
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2025/01/28 11:11:27] [ info] [fluent bit] version=3.2.5, commit=69ab1c11a1, pid=1
[2025/01/28 11:11:27] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/28 11:11:27] [ info] [simd    ] disabled
[2025/01/28 11:11:27] [ info] [cmetrics] version=0.9.9
[2025/01/28 11:11:27] [ info] [ctraces ] version=0.5.7
[2025/01/28 11:11:27] [ info] [input:tail:tail.0] initializing
[2025/01/28 11:11:27] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/01/28 11:11:27] [ info] [input:tail:tail.0] multiline core started
[2025/01/28 11:11:27] [ info] [input:systemd:systemd.1] initializing
[2025/01/28 11:11:27] [ info] [input:systemd:systemd.1] storage_strategy='memory' (memory only)
[2025/01/28 11:11:27] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2025/01/28 11:11:27] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2025/01/28 11:11:27] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2025/01/28 11:11:27] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with Kubelet...
[2025/01/28 11:11:27] [ warn] [filter:kubernetes:kubernetes.0] could not get meta for POD lima-rancher-desktop
[2025/01/28 11:11:27] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/28 11:11:27] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/01/28 11:11:27] [ info] [sp] stream processor started

Here the host is expected to be the Kubelet. But as per logs, it is the API server. However, Fluent Bit is trying to check connectivity with Kubelet

2. Fluent Bit 3 with metadata from API server

values

config:
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Use_Kubelet false

  outputs: |
    [OUTPUT] 
        Name stdout 
        Match *

image:
  tag: 3.2.5
Fluent Bit v3.2.5
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2025/01/28 11:33:35] [ info] [fluent bit] version=3.2.5, commit=69ab1c11a1, pid=1
[2025/01/28 11:33:35] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/28 11:33:35] [ info] [simd    ] disabled
[2025/01/28 11:33:35] [ info] [cmetrics] version=0.9.9
[2025/01/28 11:33:35] [ info] [ctraces ] version=0.5.7
[2025/01/28 11:33:35] [ info] [input:tail:tail.0] initializing
[2025/01/28 11:33:35] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/01/28 11:33:35] [ info] [input:tail:tail.0] multiline core started
[2025/01/28 11:33:35] [ info] [input:systemd:systemd.1] initializing
[2025/01/28 11:33:35] [ info] [input:systemd:systemd.1] storage_strategy='memory' (memory only)
[2025/01/28 11:33:35] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443  
[2025/01/28 11:33:35] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2025/01/28 11:33:35] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2025/01/28 11:33:35] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...  
[2025/01/28 11:33:35] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2025/01/28 11:33:35] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/28 11:33:35] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/01/28 11:33:35] [ info] [sp] stream processor started

Here the output is as expected. Host is the API server and Fluent Bit checks connectivity with the API server

3. Fluent Bit 2 with metadata from Kubelet

values

config:
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Use_Kubelet true

  outputs: |
    [OUTPUT] 
        Name stdout 
        Match *

image:
  tag: 2.2.3
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
Fluent Bit v2.2.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2025/01/28 11:39:59] [ info] [fluent bit] version=2.2.3, commit=be238e162c, pid=1
[2025/01/28 11:39:59] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/28 11:39:59] [ info] [cmetrics] version=0.6.6
[2025/01/28 11:39:59] [ info] [ctraces ] version=0.4.0
[2025/01/28 11:39:59] [ info] [input:tail:tail.0] initializing
[2025/01/28 11:39:59] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/01/28 11:39:59] [ info] [input:tail:tail.0] multiline core started
[2025/01/28 11:39:59] [ info] [input:systemd:systemd.1] initializing
[2025/01/28 11:39:59] [ info] [input:systemd:systemd.1] storage_strategy='memory' (memory only)
[2025/01/28 11:39:59] [ info] [filter:kubernetes:kubernetes.0] https=1 host=127.0.0.1 port=10250
[2025/01/28 11:39:59] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2025/01/28 11:39:59] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2025/01/28 11:39:59] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with Kubelet...
[2025/01/28 11:39:59] [ warn] [filter:kubernetes:kubernetes.0] could not get meta for POD lima-rancher-desktop
[2025/01/28 11:39:59] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/28 11:40:00] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/01/28 11:40:00] [ info] [sp] stream processor started

Here the output is as expected. Host is the Kubelet and Fluent Bit checks connectivity with it

4. Fluent Bit 2 with metadata from API server

values

config:
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Use_Kubelet false

  outputs: |
    [OUTPUT] 
        Name stdout 
        Match *

image:
  tag: 2.2.3
Fluent Bit v2.2.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2025/01/28 11:41:13] [ info] [fluent bit] version=2.2.3, commit=be238e162c, pid=1
[2025/01/28 11:41:13] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/28 11:41:13] [ info] [cmetrics] version=0.6.6
[2025/01/28 11:41:13] [ info] [ctraces ] version=0.4.0
[2025/01/28 11:41:13] [ info] [input:tail:tail.0] initializing
[2025/01/28 11:41:13] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2025/01/28 11:41:13] [ info] [input:tail:tail.0] multiline core started
[2025/01/28 11:41:13] [ info] [input:systemd:systemd.1] initializing
[2025/01/28 11:41:13] [ info] [input:systemd:systemd.1] storage_strategy='memory' (memory only)
[2025/01/28 11:41:13] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2025/01/28 11:41:13] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2025/01/28 11:41:13] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2025/01/28 11:41:13] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2025/01/28 11:41:13] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2025/01/28 11:41:13] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/28 11:41:13] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/01/28 11:41:13] [ info] [sp] stream processor started

Here the output is as expected. Host is the API server and Fluent Bit checks connectivity with the API server

Expected behavior
With Fluent Bit 3, when I configure the Kubernetes filter to fetch metadata from Kubelet, I expected the filter's logs to indicate that the host is the Kubelet

Screenshots

Your Environment

  • Version used: Fluent Bit 3.2.5 and 2.2.3
  • Configuration: See the values files above
  • Environment name and version (e.g. Kubernetes? What version?):
    Kubernetes v1.29.6
  • Server type and version:
  • Operating System and version:
  • Filters and plugins: Kubernetes filter

Additional context
could not get meta for POD lima-rancher-desktop error in the logs might have something to do with my Rancher Desktop setup. Hence please ignore it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nilushancosta