--------------------------------------------------------------------------------
JSON
packTime 0.612724065781 s - 163205.601975 items/s
unpackTime 0.782174110413 s - 127848.772631 items/s
size 174.26637
Baseline cPickle, ascii protocol:
--------------------------------------------------------------------------------
cPickle
packTime 2.41442704201 s - 41417.6938297 items/s
unpackTime 0.875658035278 s - 114199.831408 items/s
size 286.26637
cPickle, highest protocol:
--------------------------------------------------------------------------------
cPickleHP
packTime 1.02942800522 s - 97141.3245929 items/s
unpackTime 0.583297967911 s - 171438.965162 items/s
size 198.26637
For giggles, I evened the playing field and let cPickle at the same data structure as JSON, ascii protocol and highest protocol:
--------------------------------------------------------------------------------
cPickleJsonData
packTime 0.642832040787 s - 155561.629874 items/s
unpackTime 0.478959083557 s - 208786.101847 items/s
size 205.53356
--------------------------------------------------------------------------------
cPickleHPjsonData
packTime 0.285845041275 s - 349839.897708 items/s
unpackTime 0.340456962585 s - 293722.881273 items/s
size 175.26637
My conclusions? Serializing dictionaries is easier than serializing objects, and using a non-ascii protocol offers some obvious benefits. JSON is not obviously better than cPickle when comparing apples to apples.
It's also worth noting that comparing the json module to the pure python Pickle module isn't a fair fight either; json is at least partially written in C.
Comments
This benchmark doesn't specify a pickle protocol, which forces Python to use a big, inefficient format.
I filed a bug. https://github.com/benfred/bens-blog-code/issues/1
I also went and tested it.
Baseline JSON:
Baseline cPickle, ascii protocol: cPickle, highest protocol: For giggles, I evened the playing field and let cPickle at the same data structure as JSON, ascii protocol and highest protocol: My conclusions? Serializing dictionaries is easier than serializing objects, and using a non-ascii protocol offers some obvious benefits. JSON is not obviously better than cPickle when comparing apples to apples.It's also worth noting that comparing the json module to the pure python Pickle module isn't a fair fight either; json is at least partially written in C.
I wonder how ujson (pure-C) compares? https://pypi.python.org/pypi/ujson
This is a huge thing he missed. Anyone using pickle for non-trivial work is doing cPickle with PROTOCOL_HIGHEST.