Feature Request: --format tsv for more convenient processing via CLI
#89
Comments
Manually processing
|
| NAME | TYPE | VALUE |
|---|---|---|
| dualstack.k.sni.global.fastly.net. | A | 151.101.130.137 |
| dualstack.k.sni.global.fastly.net. | A | 151.101.194.137 |
| dualstack.k.sni.global.fastly.net. | A | 151.101.2.137 |
| dualstack.k.sni.global.fastly.net. | A | 151.101.66.137 |
| dualstack.k.sni.global.fastly.net. | AAAA | 2a04:4e42:200::649 |
| dualstack.k.sni.global.fastly.net. | AAAA | 2a04:4e42:400::649 |
| dualstack.k.sni.global.fastly.net. | AAAA | 2a04:4e42:600::649 |
| dualstack.k.sni.global.fastly.net. | AAAA | 2a04:4e42::649 |
| fastly-static.crates.io. | CNAME | dualstack.k.sni.global.fastly.net. |
| static.crates.io. | CNAME | fastly-static.crates.io. |
Reference (comparison of accomplishing the same with doggo JSON output)
As per the original issue of mine, this is the equivalent yq (or jq) for processing the --json output option of doggo - which is far simpler to process.
If q were to alter the --format json / --format yaml output, it should probably be via a separate modifier option to avoid breaking any reliance on the current structure which has much more information available.
# NOTE: The sub operator is only to remove escaped double quote wrapping of TXT values (due to JSON).
# Not doing so will make TXT values inconsistent with the TSV output when processed between these tools.
# jq
doggo example.test NS MX TXT A --json | jq -r '["TYPE", "DATA"], (.[] | [.answers[0].type, .answers[0].address])
| .[1] |= sub("\""; ""; "g")
| @tsv'
# yq
doggo example.test NS MX TXT A --json \
| yq -o=tsv '[["TYPE", "DATA"]] + [.[]
| [.answers[0].type, .answers[0].address]
| .[1] |= sub("\"", "")
]'
|
One last example with a reference input from a standard # --Use this q output for processing in next codefence--
# x3 static.crates.io => fastly-static.crates.io
# x4 fastly-static.crates.io => dualstack.k.sni.global.fastly.net
$ q @1.1.1.1 CNAME A NS TXT AAAA static.crates.io
static.crates.io. 5m CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
fastly-static.crates.io. 24s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m24s CNAME fastly-static.crates.io.
fastly-static.crates.io. 10s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m10s CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42::649
fastly-static.crates.io. 36s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m36s CNAME fastly-static.crates.io.# Pipe into qsv with space as the delimiter,
# which is ok for this example but would break with other records
# `rename --no-headers` is used to prepend headers
# Although you could avoid adding headers and just rearrange/hide columns via select indices
$ qsv fmt -d " " \
| qsv rename --no-headers NAME,TTL,TYPE,DATA \
| qsv select TYPE,NAME,DATA \
| qsv dedup --quiet \
| csview --header-align left --style markdown
| TYPE | NAME | DATA |
|-------|------------------------------------|------------------------------------|
| A | dualstack.k.sni.global.fastly.net. | 151.101.130.137 |
| A | dualstack.k.sni.global.fastly.net. | 151.101.194.137 |
| A | dualstack.k.sni.global.fastly.net. | 151.101.2.137 |
| A | dualstack.k.sni.global.fastly.net. | 151.101.66.137 |
| AAAA | dualstack.k.sni.global.fastly.net. | 2a04:4e42:200::649 |
| AAAA | dualstack.k.sni.global.fastly.net. | 2a04:4e42:400::649 |
| AAAA | dualstack.k.sni.global.fastly.net. | 2a04:4e42:600::649 |
| AAAA | dualstack.k.sni.global.fastly.net. | 2a04:4e42::649 |
| CNAME | fastly-static.crates.io. | dualstack.k.sni.global.fastly.net. |
| CNAME | static.crates.io. | fastly-static.crates.io. |No duplicates 👍
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TL;DR
It'd be great to have a CSV / TSV as a
--formattype as output for feeding into other tools likejq,yq, or anything else that can operate on CSV / TSV formatted data (csview,qsv,csvlens).Justification
--format columnis quite close to this, but you can't just delimit on space (MXrecord with preference,TXT, etc). Whereas TSV uses tabs to delimit,TXTwould not need to be quote wrapped when using--txtconcatfor example (when TXT records are longer than 255 chars).Below I've requested configurable columns but TSV may be simpler (
yqis a Go project with CSV and TSV output if it helps as reference?), where column order and other manipulations (deleting columns) can be performed by an external tool.The current
json+yamloutputs are more complicated in layout thandoggofor this type of processing (although I'm sure it's useful for programmatic access). See the belowyqexample required to manipulate the records just to exclude TTL for a custom format and removal of record duplicates due to a bug withqwhenCNAMErecords are queried with other types. A TSV output that is more aligned with--formattypesprettyandcolumnwould greatly simplify that sort of processing.It'd effectively make
--format columnredundant (but less convenient), presumably the request below for configurable columns to display in this mode wouldn't be a big addition, but I think TSV output is a great value add by itself? (eg: Markdown and CLI table output via piping intocsview)Feature Request
From my original feedback: #72 (comment)
In a follow-up comment I share a way to parse the JSON/YAML output from
q --format yamlfor the desired display format:qdoesn't have column headers likedoggo,so this request is just for the ability to toggle off the TTL column (+nottls/--no-ttls?). There's presently--pretty-ttlsand--short-ttls, but no opt-out option.Similarly since the
NAMEcolumn is all the same in the queries I've been doing, opt-out for that could also be useful? (EDIT: I've noticed this is not the case when resolvingCNAMErecords where the context may be useful,--format columnlacks this too).So instead of support for a
+nottls/--no-ttls, it could instead be an option like--hide-fields/+nofields/--hide-columns(headers is another valid term?) with a list of fields/columns to hidename,ttl? That'd allow for the inverse--fields type,answer, or--headers type,answer/+noheaders name,ttl. You could use the order in the list for column order too :)NOTE: Technically this could also be a different output type like CSV or TSV that both
jqandyqsupport as output and input formats. Then filtering out the columns manually is much simpler andqdoesn't have to worry about display (see thegithub.comexample with largeTXTvalue, which doesn't wrap into column by considering terminal width)The text was updated successfully, but these errors were encountered: