> It would be interesting to know why a portable, cross-language serialization format beats out the language-specific one.
Because protobuf parses messages with a fixed schema in a very structured format. Pickle, OTOH, is an interpreted bytecode microlanguage used to describe arbitrary python objects (for instance, pickle can call python functions: http://nadiana.com/python-pickle-insecure)
Also, pickle supports references (so if an object is referenced two times in the same serialization stream, it is
serialized only once) and this has a cost at serialization time (need to keep the set of seen references)
Perl's Storable can also serialize code references which get eval'ed during deserialization, and can also serialize multiple references once, though you have to be more explicit about that. And Storable is still 2x-3x faster than the already quite fast JSON::XS.
It would seem the bytecode interpreter architecture in Pickle is the limiting factor. If anybody has some good profiling data on Pickle though, I'd love to see it.
Comments
> It would be interesting to know why a portable, cross-language serialization format beats out the language-specific one.
Because protobuf parses messages with a fixed schema in a very structured format. Pickle, OTOH, is an interpreted bytecode microlanguage used to describe arbitrary python objects (for instance, pickle can call python functions: http://nadiana.com/python-pickle-insecure)
Also, pickle supports references (so if an object is referenced two times in the same serialization stream, it is serialized only once) and this has a cost at serialization time (need to keep the set of seen references)
Perl's Storable can also serialize code references which get eval'ed during deserialization, and can also serialize multiple references once, though you have to be more explicit about that. And Storable is still 2x-3x faster than the already quite fast JSON::XS.
It would seem the bytecode interpreter architecture in Pickle is the limiting factor. If anybody has some good profiling data on Pickle though, I'd love to see it.