Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding ndjson format #218

Merged
merged 6 commits into from
Oct 9, 2023
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions cli/cmd/transformCmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ var (
}
schema string
input string
ndjson bool
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought we said changing ndjson to stream?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok I changed the command option name but not in code, I'll make that change now

)

func init() {
Expand All @@ -39,6 +40,8 @@ func init() {

transformCmd.Flags().StringVarP(
&input, "input", "i", "", "input file (optional; if not specified, stdin/pipe is used)")
transformCmd.Flags().BoolVarP(
&ndjson, "stream", "", false, "change the output format to ndjson")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change the argument help display text to "if specified, each record will be a standalone/full JSON blob and printed out immediately once transform is done"

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, ndjson -> stream

}

func openFile(label string, filepath string) (io.ReadCloser, error) {
Expand Down Expand Up @@ -86,22 +89,42 @@ func doTransform() error {
if err != nil {
return "", err
}

s := string(b)
if ndjson {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

return s, nil
}

return strings.Join(
strs.NoErrMapSlice(
strings.Split(jsons.BPJ(string(b)), "\n"),
strings.Split(jsons.BPJ(s), "\n"),
func(s string) string { return "\t" + s }),
"\n"), nil
}

record, err := doOne()
if err == io.EOF {
fmt.Println("[]")
if ndjson {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

fmt.Println("")
} else {
fmt.Println("[]")
}
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well this is just written weirdly, why not something inline with:

if !stream {
    println("[]")
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Println writes a new line even with a blank string, should we just leave it blank?

return nil
}
if err != nil {
return err
}
fmt.Printf("[\n%s", record)

lparen := "[\n%s"
delim := ",\n%s"
rparen := "\n]"
if ndjson {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stream

lparen = "%s"
delim = "\n%s"
rparen = ""
}

fmt.Printf(lparen, record)
for {
record, err = doOne()
if err == io.EOF {
Expand All @@ -110,8 +133,8 @@ func doTransform() error {
if err != nil {
return err
}
fmt.Printf(",\n%s", record)
fmt.Printf(delim, record)
}
fmt.Println("\n]")
fmt.Println(rparen)
return nil
}