DEinspanjer on Useful Unix commands for data science

Comment on Useful Unix commands for data science

One of my favorite little tools that makes all these others better is pv -- PipeViewer.

Use it any place in a pipeline to see a progress meter on stderr. Very handy when grepping through a bunch of big log files looking for stuff. Here is a quick strawman example:

  pv /data/*.log.gz | zgrep -c 'hello world'
  241MiB 0:00:15 [15.8MiB/s] [==>       ]  2% ETA 0:12:12

gwu78 13y

BSD has a progress meter utility. It's called progress(1).

   progress -zf /data/*.log.gz grep -c 'hello world'

   progress -f /data/*.log.gz zgrep -c 'hello world'

The second form will show the progress of the decompression process.

You can also adjust buffer size, set the length for the time estimate (otherwise we have to fstat the input), and display progress to stderr instead of stdout.

gaadd33 13y

Which BSD has that? I don't seem to find it in my FreeBSD installs.

gwu78 13y

It's actually in FreeBSD's base, but it's part of the ftp(1) program.

   http://svnweb.freebsd.org/base/vendor/tnftp/dist/src/progressbar.c

   http://ftp.netbsd.org/pub/NetBSD/NetBSD-release-6/src/usr.bin/{Makefile,progress.c}

Not sure if you prefer binary installs or whether you compile your installs yourself... but I'm sure you could get this to compile on FreeBSD with a little work.

pjungwir 13y

Shameless plug: here is a similar tool I wrote that prints not a progress bar but the contents flowing through the pipe, to help in debugging:

https://github.com/pjungwir/stutter

Comments