Great achievement. To be honest I wouldn't recommend Capnp. The C++ API is very awkward.
The zero copy parsing is less of a benefit than you'd expect - pretty unlikely you're going to want to keep your data as a Capnp data structure because of how awkward it is to use. 99% of the time you'll just copy it into your own data structures anyway.
There's also more friction with the rest of the world which has more or less settled on Protobuf as the most popular binary implementation of this sort of idea.
I only used it for serialisation. Maybe the RPC stuff is more compelling.
I really wish Thrift had taken off instead of Protobuf/gRPC. It was so much better designed and more flexible than anything I've seen before or since. I think it died mainly due to terrible documentation. I guess it also didn't have a big name behind it.
I do agree that the API required for zero-copy turns out a bit awkward, particularly on the writing side. The reading side doesn't look much different. Meanwhile zero-copy is really only a paradigm shift in certain scenarios, like when used with mmap(). For network communications it doesn't change much unless you are doing something hardcore like RDMA. I've always wanted to add an optional alternative API to Cap'n Proto that uses "plain old C structures" (or something close to it) with one-copy serialization (just like protobuf) for the use cases where zero-copy doesn't really matter. But haven't gotten around to it yet...
That said I personally have always been much more excited about the RPC protocol than the serialization. I think the RPC protocol is actually a paradigm shift for almost any non-trivial use case.
I've always been excited about zero copy messages in the context of its potential in database systems; the thought of tuples working their way all the way from btree nodes in a pager, to query results on the network without copies seems fantastic.
But every time I've tried to prototype or implement around this model I've run into conceptual blocks. It's a tricky paradigm to fully wrap one's head around, and to squeeze into existing toolsets.
One thing about google proto is that, at least in many languages, every message throws off a ton of garbage that stresses the GC. On the send side, you can obviously re-use objects, but on the receive side no.
More and more languages are being built on top of the "upb" C library for protobuf (https://github.com/protocolbuffers/upb) which is designed around arenas to avoid this very problem.
Currently Ruby, PHP, and Python are backed by upb.
Disclosure: I work on the protobuf team, and created the upb library.
This is also because Google's Protobuf implementations aren't doing a very good job with avoiding unnecessary allocations. Gogoproto is better and it is possible to do even better, here is an example prototype I have put together for Go (even if you do not use the laziness part it is still much faster than Google's implementation): https://github.com/splunk/exp-lazyproto
What part of the industry are you in where flatbuffers is seen as the de facto standard? Personally I've never randomly encountered a project using flatbuffers. I see protobuf all the time.
(I've randomly run into Cap'n Proto maybe 2-3 times but to be fair I'm probably more likely to notice that.)
Flatbuffers seems to have penetration in the games industry. And it sounds like from other posters that Facebook uses it.
I recently started a job doing work on autonomy systems that run in tractors, and was surprised to see we use it (flatbuffers) in the messaging layer (in both C++ and Rust)
As of the last time I was close, flatbuffer usage is or was near ubiquitous for use in FB's (ha ha, go figure) mobile apps, across Android and iOS at least.
I find MessagePack to be pretty great if you don't need schema. JSON serialization is unreasonably fast in V8 though and even message pack can't beat it; though it's often faster in other languages and saves on bytes.
It depends on your data. We ran comparisons on objects with lots of numbers and arrays (think GeoJSON) and messagepack came out way ahead. Of course, something like Arrow may have fared even better with its focus on columnar data, but we didn't want to venture that far afield just yet.
Encoding JSON or MessagePack will be about the same speed, although I would expect MessagePack to be marginally faster from what I’ve seen over the years. It’s easy to encode data in most formats, compression excluded.
Parsing is the real problem with JSON, and no, it isn’t even close. MessagePack knows the length of every field, so it is extremely fast to parse, an advantage that grows rapidly when large strings are a common part of the data in question. I love the simple visual explanation of how MessagePack works here: https://msgpack.org/
Anyone who has written parsing code can instantly recognize what makes a format like this efficient to parse compared to JSON.
With some seriously wild SIMD JSON parsing libraries, you can get closer to the parsing performance of a format like MessagePack, but I think it is physically impossible for JSON to be faster. You simply have to read every byte of JSON one way or another, which takes time. You also don’t have any ability to pre-allocate for JSON unless you do two passes, which would be expensive to do too. You have no idea how many objects are in an array, you have no idea how long a string will be.
MessagePack objects are certainly smaller than JSON but larger than compressed JSON. Even compressed MessagePack objects are larger than the equivalent compressed JSON, in my experience, likely because the field length indicators add a randomness to the data that makes compression less effective.
For applications where you need to handle terabytes of data flowing through a pipeline every hour, MessagePack can be a huge win in terms of cost due to the increased CPU efficiency, and it’s a much smaller lift to switch to MessagePack from JSON than to switch to something statically typed like Protobuf or CapnProto, just due to how closely MessagePack matches JSON. (But, if you can switch to Protobuf or CapnProto, those should yield similar and perhaps even modestly better benefits.)
Compute costs are much higher than storage costs, so I would happily take a small size penalty if it reduced my CPU utilization by a large amount, which MessagePack easily does for applications that are very data-heavy. I’m sure there is at least one terribly slow implementation of MessagePack out there somewhere, but most of them seem quite fast compared to JSON.
Also take note of the “ShamatonGen” results, which use codegen before compile time to do things even more efficiently for types known ahead of time, compared to the normal reflection-based implementation. The “Array” results are a weird version that isn’t strictly comparable, the encoding and decoding steps assume that the fields are in a fixed order, so the encoded data is just arrays of values, and no field names. It can be faster and more compact, but it’s not “normal” messagepack.
I’ve personally seen crazy differences in performance vs JSON.
If you’re not handling a minimum of terabytes of JSON per day, then the compute costs from JSON are probably irrelevant and not worth thinking too hard about, but there can be other benefits to switching away from JSON.
Size savings depends I guess on the workload. That home page example gets larger gzip'd so raw msgpack is smaller. Another comment says their data was considerably smaller vs json.
Sometimes you can't gzip for various reasons. There were per-message deflate bugs in Safari and Brave somewhat recently. Microsoft is obsessed with the decades old CRIME/BREACH for some reason(I've never heard any other company or individual even mention them) so signalR still doesn't have the compression option yet..
The zero copy parsing is less of a benefit than you'd expect - pretty unlikely you're going to want to keep your data as a Capnp data structure because of how awkward it is to use. 99% of the time you'll just copy it into your own data structures anyway.
There's also more friction with the rest of the world which has more or less settled on Protobuf as the most popular binary implementation of this sort of idea.
I only used it for serialisation. Maybe the RPC stuff is more compelling.
I really wish Thrift had taken off instead of Protobuf/gRPC. It was so much better designed and more flexible than anything I've seen before or since. I think it died mainly due to terrible documentation. I guess it also didn't have a big name behind it.