joshbaptiste on Many times faster (de)compression using multiple processors.

Comment on Many times faster (de)compression using multiple processors.

Had to try this on my quad core laptop, as I never heard of these tools .

  josh@snoopy:~/Downloads $ grep -m2 -i intel  /proc/cpuinfo 
  vendor_id       : GenuineIntel
  model name      : Intel(R) Core(TM) i5-2410M CPU @ 2.30GHz

  josh@snoopy:~/Downloads $ ls -l test 
  -rw-r--r-- 1 josh josh 1073741824 2012-03-07 20:06 test

  josh@snoopy:~/Downloads $ time gzip test
  real    0m16.430s
  user    0m10.210s
  sys     0m0.490s

  josh@snoopy:~/Downloads $ time pigz test 
  real    0m5.028s
  user    0m16.040s
  sys     0m0.620s

Looks good.. although the man page describes it as being "an almost compatible replacement for the gzip program".

kristianp 14y

That's actually a dual core with hyperthreading. (I'm not just being pedantic, I'm curious as to how well hyperthreading works.) http://ark.intel.com/products/52224

I wonder what the speed looks like with two threads? (pigz -p 2) ?

alecco 14y

But what's the compression ratio and what's the test data?

joshbaptiste 14y

Test data

  josh@snoopy:~/Downloads $ dd if=/dev/zero of=test bs=1024K count=1000
  
  josh@snoopy:~/Downloads $ gzip -l test.gz 
  compressed        uncompressed  ratio   uncompressed_name
  1176024          1048576000  99.9% test.txt

with -p2 it runs ~2 sec slower

alecco 14y

For that kind of data you can just use RLE instead. It's the trivial case (an edge case in fact).

Comments