Just finished removing this encoding in our production services.
It panics on malformed input which is a no go for us since high availability is really important for us, and it showed quite a lot in the performance and memory profiles (roughly 5 times the time and memory as doing the same with JSON).
The code was converting some data to gob, and storing it in the database for later.
We now just do the same but in json, it’s human readable and Postgres validates that the data is valid JSON.
That's interesting because I've had basically the opposite experience. I used encoding/json with BadgerDB and saw that json.Unmarshal in a hot loop was using about 68% of total CPU time in a profile taken from production. By switching to gob it significantly decreased to around 28% (for gob's decode function). I've read that decoding interfaces in gob is slow[1], maybe that accounts for my difference as I don't have any in this particular struct. Also this was a very read-heavy service, so that could be a major difference as well.
I've been considering adopting the gob package. I haven't used it before, so I only know what's in the docs -- and all of your claims are surprising to me. Could you share more information?
How is it possible that they were getting malformed input? This was happening in go-to-go communication, or was there some kind of cross-language interop?
Any idea why the performance was so much slower than JSON in your case? The technique described in the OP would seem to make that impossible.
Do you think it's possible the database column type or collation was somehow affecting the gob?
The column type was bytea (basically blob) so it should be stored as is by the database. The profiling showed the hotspots in the gob package directly.
The docs explicitly mention that invalid input will make it panic and that can be confirmed by reading the code or fuzzing the input.
From my understanding, there is no compile time schema so everything is done with runtime reflection and that is bound to not be super fast. Granted, JSON is the same on paper, I would guess that the JSON package had more eyes on it and optimizations.
In our case, everything was using JSON except this one component due to some historical oddity so it was also a win in terms of simplifying.
Comments
Just finished removing this encoding in our production services.
It panics on malformed input which is a no go for us since high availability is really important for us, and it showed quite a lot in the performance and memory profiles (roughly 5 times the time and memory as doing the same with JSON).
The code was converting some data to gob, and storing it in the database for later.
We now just do the same but in json, it’s human readable and Postgres validates that the data is valid JSON.
And unmarshaling it does not panic.
That's interesting because I've had basically the opposite experience. I used encoding/json with BadgerDB and saw that json.Unmarshal in a hot loop was using about 68% of total CPU time in a profile taken from production. By switching to gob it significantly decreased to around 28% (for gob's decode function). I've read that decoding interfaces in gob is slow[1], maybe that accounts for my difference as I don't have any in this particular struct. Also this was a very read-heavy service, so that could be a major difference as well.
[1]: https://groups.google.com/g/golang-nuts/c/12qhqiG1J70
Have you tried the superset of json from AWS?
https://amazon-ion.github.io/ion-docs/
I've been considering adopting the gob package. I haven't used it before, so I only know what's in the docs -- and all of your claims are surprising to me. Could you share more information?
How is it possible that they were getting malformed input? This was happening in go-to-go communication, or was there some kind of cross-language interop?
Any idea why the performance was so much slower than JSON in your case? The technique described in the OP would seem to make that impossible.
Do you think it's possible the database column type or collation was somehow affecting the gob?
The column type was bytea (basically blob) so it should be stored as is by the database. The profiling showed the hotspots in the gob package directly.
The docs explicitly mention that invalid input will make it panic and that can be confirmed by reading the code or fuzzing the input.
From my understanding, there is no compile time schema so everything is done with runtime reflection and that is bound to not be super fast. Granted, JSON is the same on paper, I would guess that the JSON package had more eyes on it and optimizations.
In our case, everything was using JSON except this one component due to some historical oddity so it was also a win in terms of simplifying.
If the issue is panic why not create a wrapper func with recover and present the same interface that you want?