For a long time (much longer than I expected it would take) I've been working on a protobuf implementation in C that does not use Google's C++ implementation at all. I've been through about three rewrites and I finally have the interface right. I'm hoping it will be usable with Python soon (weeks).
(if anyone's looking at the code, I'm working on the src-refactoring branch at the moment)
The benefits of my approach are:
* you can avoid depending on a 1MB C++ library. upb is more like 30k compiled.
* you can avoid doing any code generation. instead you just load the .proto schema at runtime, so you don't have to get a C++ compiler involved.
* Google's protobuf library does have a dynamic/reflection option that avoids my previous point, but it is ~10x slower than generating C++ code. My library, last time I benchmarked it, was 70-90% of the speed of generated C++.
I read through your upb code about 3-4 months ago, was initially impressed, but couldn't get the Python extension to work. Certain abstractions really lost me, like pushing and pulling between sources and sinks. Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? But maybe you've refactored since.
Sorry, I should be clearer about the current state of the code, which for the Python extension is: currently completely broken. Since I was focusing on the core interfaces, the more peripheral pieces (like the language extensions) are totally broken at the moment.
> Certain abstractions really lost me, like pushing and pulling between sources and sinks.
Hopefully more documentation will make this clear. Making sources and sinks a general abstraction makes the event-based interface independent of any specific serialization (like protobuf binary format, protobuf text format, a JSON serialization of the same schema, etc). The key thing about protobufs is that a .proto file defines a typed tree structure, and the core interfaces of upb let you iterate over that tree structure, regardless of how exactly that tree structure was serialized.
> Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes?
In my most recent interface, the upb_src does indeed run an event loop, and calls callbacks for every input value, or on a submessage start, or on a submessage end. For a while I wanted to make upb_src a pull-based interface instead, to give the application more control over the main loop, but this created more problems than it solved.
Comments
For a long time (much longer than I expected it would take) I've been working on a protobuf implementation in C that does not use Google's C++ implementation at all. I've been through about three rewrites and I finally have the interface right. I'm hoping it will be usable with Python soon (weeks).
https://github.com/haberman/upb/wiki
(if anyone's looking at the code, I'm working on the src-refactoring branch at the moment)
The benefits of my approach are:
* you can avoid depending on a 1MB C++ library. upb is more like 30k compiled.
* you can avoid doing any code generation. instead you just load the .proto schema at runtime, so you don't have to get a C++ compiler involved.
* Google's protobuf library does have a dynamic/reflection option that avoids my previous point, but it is ~10x slower than generating C++ code. My library, last time I benchmarked it, was 70-90% of the speed of generated C++.
Here's a fast Python C extension for protobuf that's already usable:
https://github.com/acg/lwpb
I read through your upb code about 3-4 months ago, was initially impressed, but couldn't get the Python extension to work. Certain abstractions really lost me, like pushing and pulling between sources and sinks. Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? But maybe you've refactored since.
> but couldn't get the Python extension to work.
Sorry, I should be clearer about the current state of the code, which for the Python extension is: currently completely broken. Since I was focusing on the core interfaces, the more peripheral pieces (like the language extensions) are totally broken at the moment.
> Certain abstractions really lost me, like pushing and pulling between sources and sinks.
Hopefully more documentation will make this clear. Making sources and sinks a general abstraction makes the event-based interface independent of any specific serialization (like protobuf binary format, protobuf text format, a JSON serialization of the same schema, etc). The key thing about protobufs is that a .proto file defines a typed tree structure, and the core interfaces of upb let you iterate over that tree structure, regardless of how exactly that tree structure was serialized.
> Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes?
In my most recent interface, the upb_src does indeed run an event loop, and calls callbacks for every input value, or on a submessage start, or on a submessage end. For a while I wanted to make upb_src a pull-based interface instead, to give the application more control over the main loop, but this created more problems than it solved.
Can you clarify what you mean by "70-90% of the speed of generated C++"?
Suppose that the generated C++ takes 1.0 seconds. Does your implementation take 0.7-0.9s or 1.7-1.9s or something else?
If the generated C++ can parse 1MB/s, I can parse at 700-900kB/s. 70-90% of the speed, not the time. So 1.1-1.4 seconds, in your example.
Looks interesting. I might need to dig in.
I like the license, too.