clickhouse: unsupported column type "count` Int16"

[kubevalet]

What happened:

A query executed through the Grafana datasource (e.g. via explore panel) fails with error:

clickhouse: unsupported column type "count` Int16"

but if I run this exact query from the clickhouse client CLI, it works fine. Also, when I run the query from Grafana, it never shows up in the system.query_log table. Also, some other queries hitting clickhouse from grafana actually work, so I suspect this is table schema dependent?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

My test query is very simple:

SELECT timestamp,severity,message,pod_name as pod,host,`log.id` as request_id,log FROM k8s.dist_pod_logs WHERE `log.kubernetes.namespace_name` = 'foo' ORDER BY timestamp asc LIMIT 10

My table definition is:

SET allow_experimental_full_text_index = true;
SET allow_experimental_object_type = 1;

CREATE TABLE k8s.pod_logs ON CLUSTER logs
(
    timestamp DateTime,
    log JSON,
    host LowCardinality(String),
    pod_name LowCardinality(String),
    severity LowCardinality(String),
    message String CODEC(ZSTD(1)),
    INDEX inv_idx(message) TYPE full_text(0) GRANULARITY 1
)
Engine = MergeTree
ORDER BY tuple(host, pod_name, timestamp)
TTL timestamp + toIntervalDay(8);

CREATE TABLE k8s.dist_pod_logs ON CLUSTER logs
(
    timestamp DateTime,
    log JSON,
    host LowCardinality(String),
    pod_name LowCardinality(String),
    severity LowCardinality(String),
    message String CODEC(ZSTD(1))
)
ENGINE = Distributed('logs', 'k8s', 'pod_logs', rand());

Anything else we need to know?:

Environment:

• Grafana version: Grafana v10.3.3 (252761264e)
• Plugin version: 4.3.1
• OS Grafana is installed on: Kubernetes (1.28) / AWS Linux
• User OS & Browser: Mac / Chrome
• Others:

[kubevalet]

also, how do you turn on debug logging for this plugin so I can see the query it is sending to clickhouse?

[kubevalet]

I have this for grafana itself:

    [log]
    level = debug
    mode = console

but don't see anything about the query it's sending to clickhouse ?

[kubevalet]

Figured out it was my JSON column log .. after removing that from the projected fields, it's working fine. Debugging issues with this plugin is a horrible user experience btw :-(

[SpencerTorres]

Apologies for the trouble debugging. I'm still not sure why it was reporting an issue with Int16, but I will say that the JSON type is deprecated. A new JSON type is being developed, but isn't available as a stable feature yet. As for debug logging, I'm unsure if the plugin reports different messages based on Grafana's debug level. You can sometimes see what query it sends based on the error response in the HTTP response from the plugin (check your network log in the browser).

ClickHouse has some excellent features for handling JSON data stored in strings, perhaps that would be a good alternative?
Here's our new observability guide: https://clickhouse.com/docs/en/observability/schema-design

Some points may be relevant for your logging use case. Let me know if you have any more questions. Thanks!