The What and The Why

Master Foo and the Ten Thousand Lines

Master Foo once said to a visiting programmer: “There is more Unix-nature in one line of shell script than there is in ten thousand lines of C.”

The programmer, who was very proud of his mastery of C, said:“How can this be? C is the language in which the very kernel of Unix is implemented!”

Master Foo replied: “That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C.”

The programmer grew distressed. “But through the C language we experience the enlightenment of the Patriarch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!”

Master Foo replied: “All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C.”

The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”

The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so.

“And how many hours would you require to implement and debug that C program?” asked Nubi.

“Many,” admitted the visiting programmer. “But only a fool would spend the time to do that when so many more worthy tasks await him.”

“And who better understands the Unix-nature? ”Master Foo asked.“Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”

Upon hearing this, the programmer was enlightened.

Source: Rootless Root: The Unix Koans of Master Foo

Sadly, shell scripting, as well as much of Unix philosophy, is a dying craft even among people who routinely write and maintain shell scripts for Unix-like operating systems. Shell languages are typically seen as confusing, complex, unpleasantly surprising, and for good reason. All popular, widely supported shell languages are basically ancient technology that have collected a lot of cruft over the past few decades.

Their syntaxes don't look like those of any modern programming language. Their basic data type is the string. All other data types were somehow hacked together on top of strings. Common operations such as using a variable or a default value if the variable isn't set use weird syntax. Arrays either aren't supported at all or look nothing like the arrays we, as everyday programmers, are used to. Hash maps are but a pipe dream (pun intended).

And yet, "there is more Unix-nature in one line of shell script than there is in ten thousand lines of C." So, does that mean Unix-nature is crap?

Not really.

So what's this Unix-nature that shell scripts have, but other languages don't? Shell scripting provides a fantastic framework for writing software that adheres to the following aspects (or "rules") of Unix philosophy, as described by wizardly hacker Eric S. Raymond in his enlightening book, The Art of Unix Programming:

• Rule of Modularity: Write simple parts connected by clean interfaces.
• Rule of Composition: Design programs to be connected with other programs.
• Rule of Simplicity: Design for simplicity; add complexity only where you must.
• Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
• Rule of Transparency: Design for visibility to make inspection and debugging easier.
• Rule of Robustness: Robustness is the child of transparency and simplicity.
• Rule of Silence: When a program has nothing surprising to say, it should say nothing.
• Rule of Repair: When you must fail, fail noisily and as soon as possible.
• Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
• Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
• Rule of Extensibility: Design for the future, because it will be here sooner than you think.

While other programming languages may (and do) also enable many of those characteristics, no programming language use all of them as strong guiding principles like shell languages do.

Blastoise Shell is an attempt to bring those good parts of shell scripting to a true, modern programming language environment: NodeJS. And thankfully, not too much is needed to accomplish that goal. We just need a very terse, but also very pleasant and safe way to:

• Execute other programs as child processes.
• Easily and freely pipe data between our Node scripts' functions and variables, child processes, and files in the file system.
• Let them inherit our Node script's stdin, stderr, and stdout streams by default.
• Process data from streams line by line.
• Handle errors.

By default, Node's child process APIs and existing related libraries are not terse, nor pleasant, nor, as is the case of some child process invocation APIs, safe either.

If we can pull this out using Blastoise Shell, we'll be able to write shell script-like Node scripts that combine most of the power of shell scripting with the full power of JavaScript and its rich data types and data structures, and the full power of Node and its extremely rich npm module ecosystem.