In my experience, the biggest issue with TCP is that it assumes that the traffic is a single continuous byte stream, whereas the majority of applications send messages.
Vint mentioned the lack of encryption as a mistake, but even on private networks, the 16-bit TCP checksums are too weak to protect against bit-flips. Many orgs noticed that enforcing encryption reduces "random" crashes and errors, because it also enforces a strong checksum, typically 128 or 256 bit. However, even a 32-bit checksum would have sufficed for ordinary use.
Almost all "real" protocols built on top of TCP eventually end up reinventing the same wheels: message boundaries, logical sub-streams with priorities, and strong checksums or encryption. Examples include Citrix's ICA, HTTP 3.0, gRPC, etc...
IMHO TCP made the correct call here. TCP is a transport, not an application protocol. You are supposed to run your protocol over top of it, including things like blocking, substreams, etc... It's easy to add features to a simple transport, it's not so easy to work around features when you don't want them, plus they end up complicating the stack and become a possible source of bugs. Even "simple" TCP proved to be quite a challenge to implement back in the 90s, with many many buggy stacks out on the Internet.
By being just a stream of bytes you are free to do whatever you want with the protocol. Sure that may mean implementing your own form of blocking but that's better than being stuck in a protocol where you are forced to work around the message oriented features even though you are just streaming data.
The checksum does hail from an earlier era where 16 bit machines were commonplace and the total amount of data sent is minimal. Also, there is some assumption that the underlying transport is going to have its own checksums. You will note that IPv6 ditches the checksum entirely.
It sounds fine from a theoretical standpoint to use a pure byte stream rather than reliable messaging, which is what most applications actually want. However, messaging protocols can run byte streams at 0 overhead (you still need a message sequence number/tag, and this field turns into the same thing as the TCP seq num for a byte stream), while byte streams need some overhead to turn into messaging protocols, since there is otherwise no protocol-level delimiter between messages.
This seems like nearly nothing, but when you think about the lost compute power and network bandwidth throughout the history of TCP to this distinction, it's actually a very significant chunk of power, CO2, and wasted human life. Not to mention the lost human life that comes from HOL blocking.
The byte stream abstraction idea, I think, was a suspect decision at the time (born out of a desire to look exactly like a serial teletype port despite the fact that it's the wrong abstraction layer to use for that), and clearly wrong for a transport layer today. Almost everything built on top of TCP re-introduces messaging semantics.
Also, IP never had a message checksum, only a header checksum. The checksumming is at the TCP layer and it is still alive.
Sure that's how people are forced to use TCP, but for performance reasons that's not how they want to use it.
IMO something like SCTP would have been better. You generally want multiple independent streams (to avoid head of line blocking), message and stream transmission, and optional reliability on a per-stream/message basis.
This sort of thing is very very common in real time games (e.g. enet). Actually enet is built on UDP, but if you compile it to WASM then the Emscripten UDP implementation uses the WebRTC data channel which uses SCTP!
> By being just a stream of bytes you are free to do whatever you want with the protocol.
Not entirely. Head-of-line blocking is an inherent problem with TCP so you can’t have real priority “streams” like you can with eg QUIC. Another “missing feature” is datagrams, ie simply opting out of the retransmission part of tcp.
That said, I still agree with the general statement.
Yes, TCP header had 16 bits allocated to "Urgent Pointer". It's (unfortunately) still there for compatibility, but RFC 9293, which updated RFC 793 after more than 40 years, has the following to say about it:
> As a result of implementation differences and middlebox interactions, new applications SHOULD NOT employ the TCP urgent mechanism. However, TCP implementations MUST still include support for the urgent mechanism.
As it happens, that feature is used in the FTP protocol (RFC 959) to abort the current file transfer (or to immediately print a status message), since some old implementations would have to stop listening on the control connection while a transfer was in progress and could only be woken up with an interrupt. At least vsftpd supports it, I haven't looked at any other implementations. The main difficulty is that the libc API for reading urgent data is somewhat painful, and most higher-level TCP libraries haven't bothered with it for obvious reasons.
Of course, basic ftp:// clients (the only kind that are really used today) would probably rather just close the control connection, which has the same effect.
It turns out that what people wanted was application protocols, not transports.
This is why HTTP(S) is basically the only protocol used inside and outside of the data centre these days. Even storage protocols are migrating towards HTTPS, with the public clouds using S3-like protocols on top of HTTPS even for virtual disks.
This is also because firewall administrators don't want to let anything through except DNS and HTTP. Everything runs on HTTPS because it's the only thing that won't leave your help desk constantly running traces to figure out where you're getting blocked.
It's ironic that in an effort to only allow web traffic on the network, these administrators have instead made it impossible to block anything else, because their actions forced all of those other application to disguise themselves as web traffic.
No they made the wrong decision and SCTP and QUIC fixed almost everything wrong with TCP.
I personally stopped using raw TCP because it is almost always the wrong solution. If I need a bare bones protocol where I want to send raw bytes I now always use websockets because they work exactly as they should.
Streaming is such a niche usecase that is only natural if you don't semantically interpret the data, i.e. arbitrary file transfers but the vast majority of the complexity isn't there.
> It's easy to add features to a simple transport, it's not so easy to work around features when you don't want them
No, in my experience, you will spend most of your time working around TCP warts.
"No they made the wrong decision and SCTP and QUIC fixed almost everything wrong with TCP."
In the context of 2023, this is a defensible position. I can quibble, but it's defensible.
However, are you sure you want a 1974 take on a "message based protocol"? Trust me, you're not getting SCTP out of that, in a world where having two streams instead of just one is a noticeable resource impact. Heck, even by the 1990s there were still a lot of really bad things coming out of the world of "message based protocols".
You wouldn't be here in 2023 with a message-based TCP. You'd be here in 2023 with some other protocol having become the foundation of the Internet, and it wouldn't be "message based" either. There's no option where someone in 1974 is building with the nearly 50 years of experience that haven't happened yet, and I'm not going to grade TCP on that basis personally.
No, they made TCP for a particular set of use cases, and UDP for a different set, and there's always plain IP for you to build whatever you want. If you were using TCP for the wrong use case, how is that Vint Cerfs (or anyone elses) fault?
This is also how I see it, even though I've only done relatively simple work with plain old TCP. It's just a stream of bytes, you then interpret it however your protocol on top of TCP requires.
> In my experience, the biggest issue with TCP is that it assumes that the traffic is a single continuous byte stream, whereas the majority of applications send messages.
TCP and IP used to be one thing, and they split off TCP in what we now call Layer 4 (where UDP, and others, also exist). So they did think ahead in separating things to a certain extent.
The fact that we have a whole bunch of middle-boxes that don't allow DCCP, SCTP, etc, is hardly the fault of Cerf et al.
Technically true, but parent’s point still stands (even if it should have said “on top of IP”).
Byte streams are not ideal to deal with for protocols. As soon as you have encryption, like tls, you have discrete messages anyway (records) and the stream is largely an illusion.
That said, I still like TCP as a compromise between simplicity and versatility. It’s been holding up incredibly well for a huge variety of use cases.
Vint mentioned the lack of encryption as a mistake, but even on private networks, the 16-bit TCP checksums are too weak to protect against bit-flips. Many orgs noticed that enforcing encryption reduces "random" crashes and errors, because it also enforces a strong checksum, typically 128 or 256 bit. However, even a 32-bit checksum would have sufficed for ordinary use.
Almost all "real" protocols built on top of TCP eventually end up reinventing the same wheels: message boundaries, logical sub-streams with priorities, and strong checksums or encryption. Examples include Citrix's ICA, HTTP 3.0, gRPC, etc...