jq seems to parse string values as json

[ephemeris-lappis]

Hello.

It seems that jq tries to parse some string values as JSON objects.

For example, with the following text, the filed "Text" is parsed as a JSON, and it is not
[ { "Key": "one", "Text": "ok" }, { "Key": "two", "Text": "ok" } ]

With this text put into a file, it works using the command :

for x in $(cat file.json | jq -c '.[]'); do echo "$(jq .Key <<< $x) $(jq .Text <<< $x)"; done

When the field "Text" value is something like "{a=1, b=2}" (that is not a JSON object, but just a string, in fact a java Properties.toString), and with the same command, the parser fails with these messages :

parse error: Invalid numeric literal at line 1, column 4
parse error: Invalid numeric literal at line 1, column 4

Strange no ?

Does someone know if this is a normal behavior, and if some flag may be used to disable it, or if it's a "little bug" ?

Thanks for your help.

I use jq version jq-1.5-1-a5b5cbe

[itchyny]

This is not a bug. The for loop splits by white spaces.

json='[ { "Key": "one", "{a=1, b=2}": "ok" }, { "Key": "two", "Text": "ok" } ]'
for x in $(echo "$json" | jq -c '.[]'); do printf "[[[%s]]]\n" "$x"; done

[[[{"Key":"one","{a=1,]]]
[[[b=2}":"ok"}]]]
[[[{"Key":"two","Text":"ok"}]]]

The first input {"Key":"one","{a=1, is not a valid JSON. If you use the compact output option -c, you can read each line, or you can also use --raw-output0 option introduced in 1.7.

while read -r x; do jq . <<< "$x"; done < <(echo "$json" | jq -c '.[]')
while read -r -d '' x; do jq . <<< "$x"; done < <(echo "$json" | jq --raw-output0 '.[]')

[emanuele6]

I suggest using <<<"$json"/<<EOF\n$json\nEOF, or at most printf %s\\n "$json" | instead of echo "$json" | because in some system's default sh (Free/NetBSD's ash, debian's dash, MirBSD/android's mksh, but also MacOS's patched version of bash 3.2, etc), echo will expand \ sequences (like echo -e in bash) making for example json='{"a":"1\t2"}' invalid json when used in echo "$json" | jq.

[ephemeris-lappis]

Hello.

Thanks for your answers.

Anyway, I've still some doubt about jq's parsing.

The following command works as expected :

json='[{"Key":"one","Text":"ko"},{"Key":"two","Text":"ok"}]'; for x in $(jq -c .[] <<< $json); do printf "key=%s text=%s\n" "$(jq -r .Key <<< $x)" "$(jq -r .Text <<< $x)"; done

result is :

key=one text=ko
key=two text=ok

But the next command, with just a space in the first "Text" string value produces errors :

json='[{"Key":"one","Text":"k o"},{"Key":"two","Text":"ok"}]'; for x in $(jq -c .[] <<< $json); do printf "key=%s text=%s\n" "$(jq -r .Key <<< $x)" "$(jq -r .Text <<< $x)"; done

This produces a different output :

key= text=
parse error: Invalid numeric literal at line 1, column 2
parse error: Invalid numeric literal at line 1, column 2
key= text=
key=two text=ok

My version of jq has not the "--raw-output0" option, and I don't know if it can change anything to handle in sinle space in the JSON input...

Any supplementary explanation is welcome !

Thanks again.

[wader]

I think it still boils down to shell word splitting:

$ jq -nc '[{"Key":"one","Text":"k o"},{"Key":"two","Text":"ok"}][]'
{"Key":"one","Text":"k o"}
{"Key":"two","Text":"ok"}
$ for x in $(jq -nc '[{"Key":"one","Text":"k o"},{"Key":"two","Text":"ok"}][]'); do echo "$x"; done
{"Key":"one","Text":"k
o"}
{"Key":"two","Text":"ok"}

[ephemeris-lappis]

Does it mean that it's impossible to get the expected result while the shell doesn't provide correct JSON strings ? Perhaps it's going to appear as a bit stupid, but I don't understand why/how shell's word splitting impacts JSON parsing...

In the "real life", the content I must use as input comes from systems API, and the scripts should be able to read it, parse it and provide warning messages to our monitoring tools. I can't change the data format, and simple bash+jq scripts seemed to be as simple as needed.

[wader]

It's not really jq's fault, the rules for how shell word splitting works sadly makes it a bit tricky. Maybe you can use while read instead? something like:

$ echo '[{"Key":"one","Text":"k o"},{"Key":"two","Text":"ok"}]' | jq -c '.[]' | while read x; do printf "key=%s text=%s\n" "$(jq -r .Key <<< $x)" "$(jq -r .Text <<< $x)"; done
key=one text=k o
key=two text=ok

But i wonder what problem your trying to solve? if your goal is to get that output maybe you don't have to do via the shell? something like:

$ echo '[{"Key":"one","Text":"k o"},{"Key":"two","Text":"ok"}]' | jq -r '.[] | "key=\(.Key) text=\(.Text)"'
key=one text=k o
key=two text=ok