Home

Examples

These are all pretty specific use cases, but that's the point - if your problem is common, you will probably find more specific tools to address them. But when you have that weird requirement, that's when a very general tool like piep can save the day:

List the total size of all root directories (ignoring mounts)

ls -1 / | piep 'root=set(pp) | pp = sh("mount").splitlines() \
  | p.split() | p[2] | p[1:] | "/" not in p | bool \
  | root.difference(set(pp)) | "/"+p \
  | sh("du","-hs", *pp, stderr=open(os.devnull), stdout=sys.stdout)'

Explanation:

• root=set(pp) : convert input (ls -1 /) into into a set, for later
• | pp = sh("mount").splitlines() : run mount, and save its lines as the new pp
• | p.split() | p[2] | p[1:] | "/" not in p : grab the path component, trim off the leading slash and remove any paths that have more than one slash
• | bool : filter out the empty result
• | root.difference(set(pp)) : set pp to the difference of the stored root variable and the current pp value
• | "/"+p : add back leading slash
• | sh("du","-hs", *pp, stderr=open(os.devnull), stdout=sys.stdout) : run du -hs on all paths. Print directly to stdout, and ignore stderr

Rename files in a directory so their alphanumeric order matches date order

ls -1rt | piep 'enumerate(pp, 1) | list(pp) | idx, name = p \
  | newname = "%03d-%s" % p | os.rename(name, newname) or newname'

Explanation:

• ls -1rt : list files by modification date
• enumerate(pp, 1) : turn each filename into a pair of (index, filename) with indexes starting from 1
• list(pp) : make sure the input sequence is fully consumed before proceeding (you could run into issues if we start renaming things before ls is complete)
• idx, name = p : extract the index and name from the tuple
• newname = "%03d-%s" % p : format a new filename (using 3 digits for the index, increase this if you have more than 1000 files)
• os.rename(name, newname) or newname : perform the rename (since rename returns None, also return newname so we can see it in the output)

live memory usage graph (for a single process)

Requires xmgrace program

env PYTHONUNBUFFERED=1 piep -m itertools -m time \
-e PID=$(pgrep 'process-name') \
-e 'START=time.time()' \
-e 'def delay(n,x): time.sleep(n); return x' -n \
'pp = itertools.repeat(None) | delay(1,time.time()) \
  | x = int(p - START) | y = sh("ps","-o","rss=","-p", str(PID)) \
  | y = float(str(y)) / 1000 \
  | ["g0.s0 point {}, {}".format(x,y), "autoscale", "redraw"] \
  | pp.merge() \
  | itertools.chain(["yaxis label \"RSS (mb)\"", "xaxis label \"time (s)\""], pp)' \
| xmgrace -barebones -dpipe -

Explanation:

• Command line:
• env PYTHONUNBUFFERED=1: write each output line immediately
• piep -m itertools -m time: import itertools and time modules
• -e PID=$(pgrep 'process-name'): lookup PID for the given process name (you can also just use a raw number if you know it)
• -e 'START=time.time()': set START time (seconds)
• -e 'def delay(n,x): time.sleep(n); return x': create a small helper function which sleeps before returning the given value
• -n: no-input (this is a self-constructing pipe)
• Pipeline:
• pp = itertools.repeat(None): create an infinite stream of None elements
• | delay(1,time.time()): Sleep for 1 second and then return the current time
• | x = int(p - START): Set x to the seconds since monitoring started
• | y = sh("ps","-o","rss=","-p", str(PID)): Set y to the current memory usage (rss) of the target pid
• | y = float(str(y)) / 1000: convert ps output into a number, and divide it by 1000 to get mb
• | ["g0.s0 point {}, {}".format(x,y), "autoscale", "redraw"]: format each x & y value for the xmgrace program, including instructions to rescale & redraw the graph after each point
• | pp.merge(): Merge each array value into one long sequence
• | itertools.chain(["yaxis label \"RSS (mb)\"", "xaxis label \"time (s)\""], pp)': prefix the output with a header to set axis labels
• Graphing:
• | xmgrace -barebones -dpipe -: pipe everything into xmgrace, which will do the actual graphing