wildq - Command-line TOML/JSON/INI/YAML/XML/HCL processor using jq c bindings

Purpose of this package is to provide a simple wrapper arround jq for different formats. I'm tired of searching a package doing yaml jq, toml jq, ini jq etc. mainly used for scripting.

This script uses:

• @mwilliamson Python bindings on top of @stedolan famous jq lib
• swiss knife for coloration pygments
• binary built with pyinstaller
• easy CLI with click
• for supported types sources, check table Supported file types

Installation

You could install via pipx, pip, brew, deb, rpm or fallback to binary version.

Pipx

pipx install and run python applications in isolated environments

pipx install wildq

Pip

pip install wildq

MacOS

brew install ahmet2mir/tap/wildq
brew install ahmet2mir/tap/wq

Debian (no gpg signature) >= 10

VERSION=$(curl -s "https://api.github.com/repos/ahmet2mir/wildq/releases/latest" | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/\1/')
curl -sL https://github.com/ahmet2mir/wildq/releases/download/v${VERSION}/wildq_${VERSION}-1_amd64.deb -o wildq_${VERSION}-1_amd64.deb
sudo dpkg -i wildq_${VERSION}-1_amd64.deb

wq --help
wildq --help

Centos (no gpg singature) >= 7

VERSION=$(curl -s "https://api.github.com/repos/ahmet2mir/wildq/releases/latest" | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/\1/')
curl -sL https://github.com/ahmet2mir/wildq/releases/download/v${VERSION}/wildq-${VERSION}-1.x86_64.rpm -o wildq-${VERSION}-1.x86_64.rpm
sudo yum install -y ./wildq-${VERSION}-1.x86_64.rpm

wq --help
wildq --help

Gentoo

@zmedico did a portage on app-misc

emerge -av app-misc/wildq

GNU/Linux Binary

Compiled using glibc 2.17, it should work on lot of stable/LTS distros.

mkdir -p ~/bin/
VERSION=$(curl -s "https://api.github.com/repos/ahmet2mir/wildq/releases/latest" | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/\1/')
curl -sL https://github.com/ahmet2mir/wildq/releases/download/v${VERSION}/wildq-${VERSION}-linux-x86_64.tar.gz -o wildq-${VERSION}-linux-x86_64.tar.gz
tar xvfz wildq-${VERSION}-linux-x86_64.tar.gz -C ~/bin
export PATH="~/bin:$PATH"

wq --help
wildq --help

Windows

Wildq use jq.py and it's not yet available on windows platforms. I tried to compile it without windows machine and I failed, and I don't had the time to try to understand how Windows / C binding / Python works. If anybody would contribute, there is an open issue (jq and onigurama are 'compilable' on Windows so I think that someone confortable with that OS could make it);

Supported file types

type	color	ordering	output	source
hcl	json	no	json	pyhcl by @virtuald
ini	yes	no	yes	ConfigParser
json	yes	yes	yes	json
toml	yes	no	yes	toml by @uiri
xml	yes	no	yes	xmldict by @martinblech
yaml	yes	yes	yes	pyyaml

Usage

$ wildq --help
Usage: wildq [OPTIONS] JQ_FILTER [FILE]

Options:
  -c, --compact-output            compact instead of pretty-printed output
  -r, --raw                       output raw strings, not content texts
  -C, --color-output              colorize content (default), mutally
                                  exclusive with --monochrome-output

  -M, --monochrome-output         monochrome (don't colorize content), mutally
                                  exclusive with --color-output

  --hcl                           Combine --input hcl --output json, mutally
                                  exclusive with other Combined options

  --ini                           Combine --input ini --output json, mutally
                                  exclusive with other Combined options

  --json                          Combine --input json --output json, mutally
                                  exclusive with other Combined options

  --toml                          Combine --input toml --output json, mutally
                                  exclusive with other Combined options

  --xml                           Combine --input xml --output json, mutally
                                  exclusive with other Combined options

  --yaml                          Combine --input yaml --output json, mutally
                                  exclusive with other Combined options

  -i, --input [hcl|ini|json|toml|xml|yaml]
                                  Define the content type of file, mutally
                                  exclusive with Combined option

  -o, --output [hcl|ini|json|toml|xml|yaml]
                                  Define the content type of printed output,
                                  mutally exclusive with Combined option
                                  (default input format)

  --version                       Show the version and exit.
  --help                          Show this message and exit.

For backward compatibility in previous version only --[yaml|json|toml|ini|xml|hcl] was possible with default to json output. We still keep Monochrome, raw and json output with thoses options. Output was similar to jq -MCr (no color, no compact and no quote on single value)

But now, by default it's colorized, not raw and if you specify input using -i or --input output will be the same format.

There is also a shorter command wq comming with the package.

Like jq cli, wildq supports both of stdin and file to the function

See examples to get some example.

Content of examples/json.json

{
    "general": {
        "user": "admin"
    },
    "keys": [
        {"key": "value1"},
        {"key": "value2"},
        "alone"
    ]
}

cat examples/json.json | wildq -i json ".keys[]"
{
    "key": "value1"
}
{
    "key": "value2"
}
alone

or

wildq -i json ".keys[]" examples/json.json
{
    "key": "value1"
}
{
    "key": "value2"
}
alone

or

wq -i json ".keys[]" examples/json.json
{
    "key": "value1"
}
{
    "key": "value2"
}
alone

For TOML

cat examples/toml.toml | wildq -i toml ".keys[]"
{
    "key": "value1"
}
{
    "key": "value2"
}
alone

For INI (no array)

cat examples/ini.ini | wildq -i ini ".keys"
{
    "key1": "value1",
    "key2": "value2"
}

For XML

cat examples/xml.xml | wildq -i xml "."
{
    "root": {
        "general": {
            "user": "admin"
        },
        "keys": {
            "element": [
                {
                    "key": "value1"
                },
                {
                    "key": "value2"
                },
                "alone"
            ]
        }
    }
}

For YAML

cat examples/yaml.yaml  | wildq -i yaml ".keys[]"
{
    "key1": "value1"
}
{
    "key2": "value2"
}
alone

For HCL

cat examples/hcl.hcl  | wildq -i hcl ".keys[]"
{
    "key": "value1"
}
{
    "key": "value2"
}

Tips and tricks

Loop on keys in bash without creating a subshell

wildq -i toml "keys[]" examples/toml.toml | while read -r key 
do
    echo "Getting key ${key}"
done

You could also found some examples on some blogs/websites:

• a complete guide with advanced examples on how to read/update files with differents processing tools (Original Japanese) (Translate EN)

• an example of INI manipulation made by AWS ParallelCluster Workshop (workshop) (example 1) (example 2)

TODO

• add tests...
• add more control over filters and files
• use click for the CLI
• support different output
• detect automagically filetype
• support all jq types
• ordering

Contributing

Merge requests are welcome :)

License

Licensed under the terms of the Apache License, Version 2.0.

Repository URL

https://github.com/ahmet2mir/wildq