Support complex MQTT Payloads

[Appelg]

Hello!

Nice plugin. Can you please update the documentation with information on how to format the payload?
No matter what I try, all I get is "Data does not have a time field"

Can I have complex objects with multiple values? Can I instruct it to parse the json?
E.g

{
  foo: { 
   bar: 1337
  }
  baz: 555
}

Thanks!

[ryantxu]

Currently the plugin supports simple values ie "555" or a simple map[string]number -- we should be able to support a more complex shape

[Appelg]

Thanks!

The no-data was solved by setting a low timerange, like 5 mins. I used it in an existing dashboard which defaulted to 30d.

[whbruce]

I'm new to grafana. Could you give more specific examples of the what the data should look like (i.e. what does map[string]number mean?) . I can generate data points with mosquitto_pub -t grafana -m 555 but want to attach a time to the data.

[atifali]

currently we support very basic JSON structures like:

{
    'value1': 1.0,
    'value2': 2,
    'value3': 3.33,
    ...
}

notice how the key is string and the value is number in the JSON, that is what map[string]number (golang lingo) means in this context.

we do plan to bring support for more complex JSON structures soon (contributions are always welcome too 🙂)

as far as the timestamp is concerned currently we just naively use time.Now() for every message coming in, see:

mqtt-datasource/pkg/mqtt/client.go

Line 98 in 46b6178

Timestamp: time.Now(),

however i do agree we should have a way to set that based on the incoming message in cases where the data is delayed and needs a timestamp on the publisher side to be propagated. any suggestions on this?

[whbruce]

Thanks for the detailed reply. I'm currently sending just raw data. Once I understand more about Grafana I'll try out your more complex json payload and recommend how time might be encoded.

Re: contributions your typescript and go code (and how they interact) is beyond me at this point (I'm a C++/Java/Python guy).

[atifali]

support for more data types with the JSON payload would be super handy as part of this enhancement, see: #19

[rafaelschlatter]

Hi @atifali,

How do you plan to support more complex json? will that be arbitrary json? Do you have plans to support timestamps that are part of the MQTT message?

This has caught my attention, I think it is awesome: https://grafana.com/blog/2021/08/12/streaming-real-time-sensor-data-to-grafana-using-mqtt-and-grafana-live/

But we can currently not use it, because our json is more complex and also subject to change.

[atifali]

Hi @rafaelschlatter really appreciate your enthusiasm and patience with this. We are currently in the midst of incorporating a very versatile and flexible automatic JSON processor within Live "pipelines" in Grafana core (see: https://github.com/grafana/grafana/tree/main/pkg/services/live/pipeline), which will eventually be used here. This means that virtually any message that has a valid JSON format would work. Stay tuned!

[rafaelschlatter]

Sounds awesome, thanks for the update, @atifali !

[Appelg]

@rafaelschlatter that sounds awesome, nice work!

[andrerodi]

Any news on this? There is an open pull request #22

[grabouillon06]

Hello, what is the status for the integration of the Live "pipelines" JSON processor in mqtt-datasource? Do you have any news? thanks and have a good day.

[wz2b]

One idea about complex mqtt payloads would be to implement something like what Amazon did for the rules processing feature of IoT Core. Essentially, it allows you to use a language very close to SQL to "select" data out of a JSON object of arbitrary structure and depth, pull out the data you want, rename the fields, and do some basic math on them. Some of what they do probably wouldn't be necessary here (I don't think you really need WHERE clauses, for example).

[DonatoD]

Sorry, which the state of this integration? I tried to understand it by clicking on the various links: closed, enhancement, open ....
It's Like a stuck loop

[OlivierGre]

@atifali Hello, What is the status of this? Thanks

[pkkrusty]

Any progress on this?