TCP, Hold the Congestion Control?

In an off-list discussion in the protocols interest groups, I got involved in a rather deep discussion of packet rate, congestion control, network neutrality, jitter and choices in Internet design, which are actually quite interesting to share.

A little background here - one person asked if it was true (it is) that the cwnd (congestion window) internal stack variable doesn't have an immediate impact on the network, because TCP updates its actual rate only once per RTT in the congestion avoidance phase, so the cwnd += SMSS*SMSS/cwnd update with each ACK is only an internal calculation. You got that?

Which went on to the question posed to me - "While I now believe that it would actually be 'legal' according to the spec. to implement a TCP sender like this (no one seems to say that you MUST saturate your window at all times)..."

Wait partner. Going back into the Internet Wayback Machine and chatting with some of the earlier worker bees, it turned out it actually started out this way, and congestion backoff fell right out of this.

When the Internet had 56 kbit/sec lines, we coped, but the core was too slow. But then it moved to T1, which was good, but still not great. At this point, everyone began to realize the core was just too slow for the senders. If you had everyone sending the max, it blew out the routers. The simplest version of TCP didn't have much in the way of policy then - just your initial send - but that's where slow start / slow open began. This all should be in the history sections of some of the TCP books, but you can also ask some of the people who did it, since most of them are still alive and kicking. Some folks (Microsoft in particular) don't like slow start however - they keep the session always open instead, which has it's own problems.

"I am sure that typically, a TCP sender updates not only an internal variable but the actual rate that it uses with every ACK, no matter what state it is in. It does not wait a RTT - the notion that it increases the rate by one segment per RTT is just the outcome of the per-ACK rate updates, seen over a long time interval."

Ahh, the inevitable "rate" issue. It seems reasonable, especially to a physicist, to consider rate in this context. But as a mathematician would say, is it contiguous and deterministic?

The fact of the matter is that rate calculations are fraught with peril. The rate is often too erratic because you have stochastic as well as statistical error. Earlier Internet calculations relied on the variance of RTT stats to get the greatest thruput - not do the greatest rate, but the max stable transmission. If you go beyond it, it becomes more unstable (stochastic).

So unlike a nice physics situation as the rate of speed of a particle in a particular direction, rate in the Internet sense is a statistic which is the compounded randomness of congestion in switches, arrival time variance, ... the list goes on. Error analysis is a fundamental issue, but is typically not studied by CS majors or engineers (but they made us do it in physics), but if it were, I'm sure that the real-world conditions of the Internet would make anyone crazy.

Worse yet, companies like Microsoft encourage the idea of very fat packets for their own purposes and don't worry about how it might affect overall network performance. Other experimental mechanisms distilled down to elementals simply grab whatever bandwidth is available in a forceful way with no fairness implied, like a bunch of thugs shouting "Make way" through the crowd of rabble and trampling over everyone. Again, no overall network performance - or network neutrality - is a consideration, and that is in opposition to the establishment of Internet access itself.

Let's talk jitter - not rate. This implies small packets and handling transparent congestion recovery with TCP without affecting the semantics. In this more subtle approach, we need to plan the staging of session data across the Internet without inducing any more network level jitter (or randomness). We've got to think "stable" - not play with statistics in the hopes that eventually we'll get a clear channel. It's at the the end you get stable bandwidth, but that's not what you start with. Begin with "eliminate the source of the problem" and the compound equation is bettered.

When TCP is used in a highly lossy LAN wireless network, for example, it can do a remarkably good job at coping with this impossible situation. But if we extend the lossy links (like a link or network layer technology) in subsequent hops, ultimately nothing gets through. By reducing the bit error rate on a link, the way TCP performs improves remarkably (this shouldn't be a surprise).

Just consider that if we can do network level bit error rate across network hops using network measures that are adaptive - we can reduce the problem so that the net effect on the TCP connection is that the congestion control / flow control / backoff algorithms see an idealized network where the effect of the changes imparted by the transport layer are due to what the network can actually handle over a larger time granularity, and the smaller elements of the chaos/random average out anyways. Thus, we scale the randomness, so to speak to something more appropriate for the RTT that you see (e.g. 80 millisec RTT - effects of congestion / flow control now should reflect a low-level network with an expectation that the characteristics will not change for a minimum of 2 sample times - information theoretic - and that the low-level transport guarantees that the expectation of transport will be held this way, regardless.).

The problem with optimizing rate is that, like Chaos Theory, we don't necessarily understand the impact of the effect. Anything can destablize and make it more chaotic. We can play with random against random in the hopes of cancelling it out, or you minimize the effects of the noise. A first order approach to doing this was congestion backoff and determining the bounds. It was a good idea for the time, but that time is past - it is too limited.

The limits of this approach is that you can only determine this at the endpoints so you're stuck with an undetermined situation in the middle, as we noticed in the extreme case of a lossy wireless network. If you have a RTT of 80 millisec connection, the only time you can do something is every 80 millisec - there may be many sources of chaos in between, and you can still lose out.

The only realistic approach is to divide that 80 millisec interval into 10 8millisec regions and reconcile at each point using the exact same TCP algorithm - in other words, simply break the problem down with divide and conquer. The limit was to complete full transport processing compliant with all interoperable goals timely and also comply with the end-to-end principle.

This has been done in the lab - we made a 10-fold improvement in the situation and we did it in 1998 at InterProphet! And we didn't rely on statistics in an increasingly complex Internet structure, nor have to prove a new algorithm, since it's the same algorithm. We just constrained in more places.

Actually, the smaller the segments, the better it works, because it adapts for the segments in a realistic way. Also the elements where most of the effect occurs is chiefly as it goes off the network, going from high to lower bandwidth environments at the edge.

Remember, congestion control and backoff are biased historically toward the core. We've got plenty of capacity at the core now. It's the edge that is the problem, and will remain lagging for the next 20 years. That is where the work should concentrate, because that's where the economics dominate. And hence that's where the money is.

Where everyone goes wrong is that they think the point is to eliminate the responsibility for the transport layer and thus effect semantics, because it's larger than 2x RTT level. We see looking at the stability is an enhancement that is bounded by the visibility of the mechanism.

It's an argument between planned use and stochastic use. Some opportunists see the success of this algorithm in dealing with a completely different problem at the core in a different time with a different set of conditions and think they can use butterfly's wings with more powerful tools in a new age of Internet use with a completely different set of conditions and problems emerging. In other words,the statistics are taken far beyond what is meaningful or possible. It's a really difficult problem to discuss, somewhat like dealing with a "chemist" who believes he can transmute metals with the right kind of lodestone. The alchemist mistakes the real success of chemistry in certain reactions with the assumption that all reactions are the same if you find the right catalyst. He doesn't do the hard work of really understanding how things work, and the consequent limits. Too many people don't study history and philosophy, especially in science, but that's where the disconnect comes.

We've got to understand and remove the randomness that clouds the issue - not just "control" it. Perhaps working in plasma physics, I have a pretty good feel for the limits of statistics in controlling randomness, as the history of plasma research demonstrates.

I had to consult with others on this matter today, as it is a deep and thoughtful one, and it has led to a long set of discussions. I hope their insight helps others in this matter as well.