Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added json_path function #4225

Merged
merged 3 commits into from
Jun 23, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ Main (unreleased)
components and modifies attributes of a span, log, or metric. (@ptodev)


- Added json_path function to river stdlib. (@jkroepke)

### Enhancements

- Attributes and blocks set to their default values will no longer be shown in the Flow UI. (@rfratto)
Expand Down
41 changes: 41 additions & 0 deletions docs/sources/flow/reference/stdlib/json_path.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
aliases:
- ../../configuration-language/standard-library/json_path/
title: json_path
---

# json_path

The `json_path` function lookup values using [jsonpath](https://goessner.net/articles/JsonPath/) syntax.

The function expects 2 strings. First string is the json string to lookup values from. Second string is the jsonpath expression.

`json_path` always return a list of values. If the jsonpath expression does not match any values, an empty list is returned.

A common use case of `json_path` is to decode and filter the output of a [`local.file`][] or [`remote.http`][] component to a River value.

> Remember to escape double quotes when passing JSON string literals to `json_path`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's implied, but could we also mention that we can pass the string export of another component directly if it's valid JSON?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do a bit copy/paste from json_encode. Is this enough?

jkroepke marked this conversation as resolved.
Show resolved Hide resolved
>
> For example, the JSON value `{"key": "value"}` is properly represented by the
> string `"{\"key\": \"value\"}"`.

## Examples

```
> json_path("{\"key\": \"value\"}", ".key")
["value"]


> json_path("[{\"name\": \"Department\",\"value\": \"IT\"},{\"name\":\"TestStatus\",\"value\":\"Pending\"}]", "[?(@.name == \"Department\")].value")
["IT"]

> json_path("{\"key\": \"value\"}", ".nonexists")
[]

> json_path("{\"key\": \"value\"}", ".key")[0]
value

```
jkroepke marked this conversation as resolved.
Show resolved Hide resolved

[`local.file`]: {{< relref "../components/local.file.md" >}}
[`remote.http`]: {{< relref "../components/remote.http.md" >}}
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ require (
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/ncabatoff/process-exporter v0.7.10
github.com/nerdswords/yet-another-cloudwatch-exporter v0.51.0
github.com/ohler55/ojg v1.18.7
github.com/oklog/run v1.1.0
github.com/olekukonko/tablewriter v0.0.5
github.com/oliver006/redis_exporter v1.49.0
Expand Down
9 changes: 2 additions & 7 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1712,8 +1712,6 @@ github.com/grafana/ckit v0.0.0-20230518140533-fbd338b33964 h1:rJIMh9u7TBoq58tE8c
github.com/grafana/ckit v0.0.0-20230518140533-fbd338b33964/go.mod h1:HcHGszFkeYCuFbWt6qt72rnZRHwbR2BhQBGHWqe4AqU=
github.com/grafana/cloudflare-go v0.0.0-20230110200409-c627cf6792f2 h1:qhugDMdQ4Vp68H0tp/0iN17DM2ehRo1rLEdOFe/gB8I=
github.com/grafana/cloudflare-go v0.0.0-20230110200409-c627cf6792f2/go.mod h1:w/aiO1POVIeXUQyl0VQSZjl5OAGDTL5aX+4v0RA1tcw=
github.com/grafana/dnsmasq_exporter v0.2.1-0.20211118155541-751b01d21de9 h1:/ovWD85B27b/xytKaxhP/LvXF8fxWIhEwE55mFpCQsE=
github.com/grafana/dnsmasq_exporter v0.2.1-0.20211118155541-751b01d21de9/go.mod h1:Jj5TSVVJE6t8Brq/ZkUbQ1n260dilriL0tWNaHjVDUs=
github.com/grafana/dskit v0.0.0-20210908150159-fcf48cb19aa4/go.mod h1:m3eHzwe5IT5eE2MI3Ena2ooU8+Hek8IiVXb9yJ1+0rs=
github.com/grafana/dskit v0.0.0-20211021180445-3bd016e9d7f1/go.mod h1:uPG2nyK4CtgNDmWv7qyzYcdI+S90kHHRWvHnBtEMBXM=
github.com/grafana/dskit v0.0.0-20230201083518-528d8a7d52f2 h1:IOks+FXJ6iO/pfbaVEf4efNw+YzYBYNCkCabyrbkFTM=
Expand Down Expand Up @@ -2321,14 +2319,11 @@ github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyex
github.com/microsoft/go-mssqldb v0.19.0 h1:LMRSgLcNMF8paPX14xlyQBmBH+jnFylPsYpVZf86eHM=
github.com/microsoft/go-mssqldb v0.19.0/go.mod h1:ukJCBnnzLzpVF0qYRT+eg1e+eSwjeQ7IvenUv8QPook=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/miekg/dns v1.1.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/miekg/dns v1.1.25/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso=
github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso=
github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI=
github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
github.com/miekg/dns v1.1.50/go.mod h1:e3IlAVfNqAllflbibAZEWOXOQ+Ynzk/dDozDxY7XnME=
github.com/miekg/dns v1.1.54 h1:5jon9mWcb0sFJGpnI99tOMhCPyJ+RPVz5b63MQG0VWI=
github.com/miekg/dns v1.1.54/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY=
github.com/miekg/dns v1.1.55 h1:GoQ4hpsj0nFLYe+bWiCToyrBEJXkQfOOIvFGFy0lEgo=
github.com/miekg/dns v1.1.55/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY=
github.com/miekg/pkcs11 v1.0.3/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WTFxgs=
Expand Down Expand Up @@ -2468,6 +2463,8 @@ github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
github.com/observiq/ctimefmt v1.0.0 h1:r7vTJ+Slkrt9fZ67mkf+mA6zAdR5nGIJRMTzkUyvilk=
github.com/observiq/ctimefmt v1.0.0/go.mod h1:mxi62//WbSpG/roCO1c6MqZ7zQTvjVtYheqHN3eOjvc=
github.com/ohler55/ojg v1.18.7 h1:sC7zy0usEiWa6bvx3NU1yZH4kCA2F3Qzs6iiDX4+xdk=
github.com/ohler55/ojg v1.18.7/go.mod h1:uHcD1ErbErC27Zhb5Df2jUjbseLLcmOCo6oxSr3jZxo=
github.com/oklog/oklog v0.3.2/go.mod h1:FCV+B7mhrz4o+ueLpx+KqkyXRGMWOYEvfiXtdGtbWGs=
github.com/oklog/run v0.0.0-20180308005104-6934b124db28/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA=
github.com/oklog/run v1.0.0/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA=
Expand Down Expand Up @@ -2812,8 +2809,6 @@ github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O
github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.8.0/go.mod h1:z7EfXMXOkbkqb9IINtpCn86r/to3BnA0uaxHdg830/4=
github.com/prometheus/procfs v0.10.1 h1:kYK1Va/YMlutzCGazswoHKo//tZVlFpKYh+PymziUAg=
github.com/prometheus/procfs v0.10.1/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM=
github.com/prometheus/procfs v0.11.0 h1:5EAgkfkMl659uZPbe9AS2N68a7Cc1TJbPEuGzFuRbyk=
github.com/prometheus/procfs v0.11.0/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM=
github.com/prometheus/statsd_exporter v0.22.7/go.mod h1:N/TevpjkIh9ccs6nuzY3jQn9dFqnUakOjnEuMPJJJnI=
Expand Down
16 changes: 16 additions & 0 deletions pkg/river/internal/stdlib/stdlib.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import (

"github.com/grafana/agent/pkg/river/internal/value"
"github.com/grafana/agent/pkg/river/rivertypes"
"github.com/ohler55/ojg/jp"
"github.com/ohler55/ojg/oj"
)

// Identifiers holds a list of stdlib identifiers by name. All interface{}
Expand Down Expand Up @@ -79,6 +81,20 @@ var Identifiers = map[string]interface{}{
return res, nil
},

"json_path": func(jsonString string, path string) (interface{}, error) {
jsonPathExpr, err := jp.ParseString(path)
if err != nil {
return nil, err
}

jsonExpr, err := oj.ParseString(jsonString)
if err != nil {
return nil, err
}

return jsonPathExpr.Get(jsonExpr), nil
},

"coalesce": value.RawFunction(func(funcValue value.Value, args ...value.Value) (value.Value, error) {
if len(args) == 0 {
return value.Null, nil
Expand Down
26 changes: 26 additions & 0 deletions pkg/river/vm/vm_stdlib_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,32 @@ func TestStdlibCoalesce(t *testing.T) {
}
}

func TestStdlibJsonPath(t *testing.T) {
tt := []struct {
name string
input string
expect interface{}
}{
{"json_path with simple json", `json_path("{\"a\": \"b\"}", ".a")`, []string{"b"}},
{"json_path with simple json without results", `json_path("{\"a\": \"b\"}", ".nonexists")`, []string{}},
{"json_path with json array", `json_path("[{\"name\": \"Department\",\"value\": \"IT\"},{\"name\":\"ReferenceNumber\",\"value\":\"123456\"},{\"name\":\"TestStatus\",\"value\":\"Pending\"}]", "[?(@.name == \"Department\")].value")`, []string{"IT"}},
{"json_path with simple json and return first", `json_path("{\"a\": \"b\"}", ".a")[0]`, "b"},
}

for _, tc := range tt {
t.Run(tc.name, func(t *testing.T) {
expr, err := parser.ParseExpression(tc.input)
require.NoError(t, err)

eval := vm.New(expr)

rv := reflect.New(reflect.TypeOf(tc.expect))
require.NoError(t, eval.Evaluate(nil, rv.Interface()))
require.Equal(t, tc.expect, rv.Elem().Interface())
})
}
}

func TestStdlib_Nonsensitive(t *testing.T) {
scope := &vm.Scope{
Variables: map[string]any{
Expand Down