Recall from Chapter 12 that a variable sliding window can be used to implement flow control. In Figure 15-8, a TCP client and server are interacting, providing each other with information about the data flow, including segment sequence numbers, ACK numbers, and window sizes (i.e., available space at the receiver).
&OLHQW 6HUYHU
6HT
$&.
:LQ 'DWD
6HT
$&.
:LQ 'DWD 7&3
6HJPHQW 6&
7&3 6HJPHQW
&6
Figure 15-8 Each TCP connection is bidirectional. Data going in one direction causes the peer to respond with ACKs and window advertisements. The same is true for the reverse direction.
ptg999 Section 15.5 Flow Control and Window Management 701
The two large arrows in Figure 15-8 indicate the direction of data flow (the direction in which TCP segments are sent). Recalling that every TCP connection has data flowing in both directions, we have two arrows, one in the client-to- server direction (C→S) and another in the server-to-client direction (S→C). Every segment contains ACK and window information and may also contain some user data. The fields used in the TCP header are shaded based on the direction of data flow they describe. For example, data flowing in the C→S direction is included in segments flowing along the bottom arrow, but the ACK number and window advertisement for this data are returned in segments following the top arrow.
Every TCP segment (except those exchanged during connection establishment) includes a valid Sequence Number field, an ACK Number or Acknowledgment field, and a Window Size field (containing the window advertisement).
In each of the ssh examples in this chapter so far, we have seen an unchang- ing window advertisement conveyed from one TCP peer to the other. Examples include 8320 bytes, 4220 bytes, and 32,900 bytes. These sizes represent the amount of space the sender of the segment has reserved for storing incoming data the peer sends. When TCP-based applications are not busy doing other things, they are typically able to consume any and all data TCP has received and queued for them, leading to no change of the Window Size field as the connection progresses.
On slow systems, or when the application has other things to accomplish, data may have arrived for the application, been acknowledged by TCP, and be sitting in a queue waiting for the application to read or “consume” it. When TCP starts to queue data in this way, the amount of space available to hold new incoming data decreases, and this change is reflected by a decreasing value of the Window Size field. Eventually, if the application does not read or otherwise consume the data at all, TCP must take some action to cause the sender to cease sending new data entirely, because there would be no place to put it on arrival. This is accomplished by sending a window advertisement of zero (no space).
The Window Size field in each TCP header indicates the amount of empty space, in bytes, remaining in the receive buffer. The field is 16 bits in TCP, but with the Window Scale option, values larger than 65,535 can be used (see Chapter 13).
The largest sequence number the sender of a segment is willing to accept in the reverse direction is equal to the sum of the Acknowledgment Number and Window Size fields in the TCP header (scaled appropriately).
15.5.1 Sliding Windows
Each endpoint of a TCP connection is capable of sending and receiving data. The amount of data sent or received on a connection is maintained by a set of window structures. For each active connection, each TCP endpoint maintains a send window structure and a receive window structure. These structures are similar to the con- ceptual window structures described in Chapter 12, but here we describe them in more detail. Figure 15-9 shows a hypothetical TCP send window structure.
ptg999
TCP maintains its window structures in terms of bytes (not packets). In Fig- ure 15-9 we have numbered the bytes 2 through 11. The window advertised by the receiver is called the offered window and covers bytes 4 through 9, meaning that the receiver has acknowledged all bytes up through and including number 3 and has advertised a window size of 6. Recall from Chapter 12 that the Window Size field con- tains a byte offset relative to the ACK number. The sender computes its usable window, which is how much data it can send immediately. The usable window is the offered window minus the amount of data already sent but not yet acknowledged. The vari- ables SND.UNA and SND.WND are used to hold the values of the left window edge and offered window. The variable SND.NXT holds the next sequence number to be sent, so the usable window is equal to (SND.UNA + SND.WND – SND.NXT).
Over time this sliding window moves to the right, as the receiver acknowl- edges data. The relative motion of the two ends of the window increases or decreases the size of the window. Three terms are used to describe the movement of the right and left edges of the window:
1. The window closes as the left edge advances to the right. This happens when data that has been sent is acknowledged and the window size gets smaller.
2. The window opens when the right edge moves to the right, allowing more data to be sent. This happens when the receiving process on the other end reads acknowledged data, freeing up space in its TCP receive buffer.
3. The window shrinks when the right edge moves to the left. The Host Requirements RFC [RFC1122] strongly discourages this, but TCP must be able to cope with it. Section 15.5.3 on silly window syndrome shows an example where one side would like to shrink the window by moving the right edge to the left but cannot.
2IIHUHG:LQGRZ (SND.WND)
6HQWDQG
$FNQRZOHGJHG
6HQWDQG1RW
$FNQRZOHGJHG
%HLQJ6HQW 8VDEOH:LQGRZ
&DQQRW6HQG8QWLO :LQGRZ0RYHV
/HIW(GJH (SND.UNA)
5LJKW(GJH (SND.UNA + SND.WND)
&ORVHV 6KULQNV 2SHQV SND.NXT
Figure 15-9 The TCP sender-side sliding window structure keeps track of which sequence numbers have already been acknowledged, which are in flight, and which are yet to be sent. The size of the offered window is controlled by the Window Size field sent by the receiver in each ACK.
ptg999 Section 15.5 Flow Control and Window Management 703
Because every TCP segment contains both an ACK number and a window advertisement, a TCP sender adjusts the window structure based on both values whenever an incoming segment arrives. The left edge of the window cannot move to the left, because this edge is controlled by the ACK number received from the other end that is cumulative and never goes backward. When the ACK number advances the window but the window size does not change (a common case), the window is said to advance or “slide” forward. If the ACK number advances but the window advertisement grows smaller with other arriving ACKs, the left edge of the window moves closer to the right edge. If the left edge reaches the right edge, it is called a zero window. This stops the sender from transmitting any data.
If this happens, the sending TCP begins to probe the peer’s window (see Section 15.5.2) to look for an increase in the offered window.
The receiver also keeps a window structure, which is somewhat simpler than the sender’s. The receiver window structure keeps track of what data has already been received and ACKed, as well as the maximum sequence number it is willing to receive. The TCP receiver depends on this structure to ensure the correctness of the data it receives. In particular, it wishes to avoid storing duplicate bytes it has already received and ACKed, and it also wishes to avoid storing bytes that it should not have received (any bytes beyond the sender’s right window edge). The receiver’s window structure is illustrated in Figure 15-10.
5HFHLYH:LQGRZ (RCV.WND)
5HFHLYHG$QG
$FNQRZOHGJHG
&DQQRW$FFHSW
/HIW(GJH 5&91;7
5LJKW(GJH (RCV.NXT+RCV.WND) :LOO6WRUH:KHQ5HFHLYHG
Figure 15-10 The TCP receiver-side sliding window structure helps the receiver know which sequence numbers to expect next. Sequence numbers in the receive window are stored when received. Those outside the window are discarded.
This structure also contains a left and right window edge like the sender’s window, but the in-window bytes (4–9 in this picture) need not be differentiated as they are in the sender’s window structure. For the receiver, any bytes received with sequence numbers less than the left window edge (called RCV.NXT) are dis- carded as duplicates, and any bytes received with sequence numbers beyond the
ptg999 right window edge (RCV.WND bytes beyond RCV.NXT) are discarded as out of
scope. Bytes arriving with any sequence number in the receive window range are accepted. Note that the ACK number generated at the receiver may be advanced only when segments fill in directly at the left window edge because of TCP’s cumulative ACK structure. With selective ACKs, other in-window segments can be acknowledged using the TCP SACK option, but ultimately the ACK number itself is advanced only when data contiguous to the left window edge is received (see Chapter 14 for more details on SACK).
15.5.2 Zero Windows and the TCP Persist Timer
We have seen that TCP implements flow control by having the receiver specify the amount of data it is willing to accept from the sender: the receiver’s adver- tised window. When the receiver’s advertised window goes to zero, the sender is effectively stopped from transmitting data until the window becomes nonzero.
When the receiver once again has space available, it provides a window update to the sender to indicate that data is permitted to flow once again. Because such updates do not generally contain data (they are a form of “pure ACK”), they are not reliably delivered by TCP. TCP must therefore handle the case where such window updates that would open the window are lost.
If an acknowledgment (containing a window update) is lost, we could end up with both sides waiting for the other: the receiver waiting to receive data (because it provided the sender with a nonzero window and expects to see incoming data) and the sender waiting to receive the window update allowing it to send. To pre- vent this form of deadlock from occurring, the sender uses a persist timer to query the receiver periodically, to find out if the window size has increased. The persist timer triggers the transmission of window probes. Window probes are segments that force the receiver to provide an ACK, which also necessarily contains a Win- dow Size field. The Host Requirements RFC [RFC1122] suggests that the first probe should happen after one RTO and subsequent problems should occur at exponen- tially spaced intervals (i.e., similar to the “second part” of Karn’s algorithm, which we discussed in Chapter 14).
Window probes contain a single byte of data and are therefore reliably deliv- ered (retransmitted) by TCP if lost, thereby eliminating the potential deadlock condition caused by lost window updates. The probes are sent whenever the TCP persist timer expires, and the byte included may or may not be accepted by the receiver, depending on how much buffer space it has available. As with the TCP retransmission timer (see Chapter 14), the normal exponential backoff can be used when calculating the timeout for the persist timer. An important difference, how- ever, is that a normal TCP never gives up sending window probes, whereas it may eventually give up trying to perform retransmissions. This can lead to a certain resource exhaustion vulnerability that we discuss in Section 15.7.
ptg999 Section 15.5 Flow Control and Window Management 705
15.5.2.1 Example
To illustrate the use of the dynamic window size adjustment and flow control in TCP, we create a TCP connection and cause the receiving process to pause before consuming data from the network. For this experiment, we use a Mac OS X 10.6 sender and a Windows 7 receiver. The receiver runs our sock program with the –P flag as follows:
C:\> sock -i -s -P 20 6666
This arranges for the receiver to pause 20s prior to consuming data from the net- work. The result is that eventually the receiver’s advertised window begins to close, as shown with packet 125 in Figure 15-11.
Figure 15-11 After a period when the advertised window does not change, acknowledgments continue but the window size grows smaller as the receiver’s buffer fills up. If the receiving application fails to consume any data and the sender continues, the window eventually reaches zero.
ptg999 In this trace we see that for more than 100 packets the receiver’s window
remains pegged at 64KB. This is because of an automatic window adjustment algorithm (see Section 15.5.4) that allocates memory to the receiving TCP even if not requested by the application. However, this eventually runs short, so we see the window begin to reduce starting with packet 125. A large number of ACKs fol- low, each reducing the window further while increasing the ACK number by 2896 bytes per ACK. This indicates that the receiving TCP is storing the data, but the application is not consuming it. If we look further into the trace, we see that even- tually the receiver has no more space to hold the incoming data (see Figure 15-12).
Figure 15-12 The receiver’s buffer has filled up. When the receiving application starts reading again, a win- dow update tells the sender that there is now an opportunity to transfer more data.
Here we can see that packet 151 fills the small 327-byte window, as indicated by the TCP Window Full comment provided by Wireshark. After about 200ms, at time 4.979, a zero window advertisement is produced, indicating that no more data can be received. This is no surprise, given that the sender has filled the last known available window and the receiving application will not consume any data until time 20.143.
After receiving the zero window advertisement, the sending TCP tries to probe the receiver three times at 5s intervals to see if the window has opened. At time 20, as instructed, the receiver begins to consume the data present in TCP’s queue.
This causes two window updates to be sent to the sender, indicating that further data transmission (up to 64KB) is now possible. Such segments are called window updates because they do not acknowledge any new data—they just advance the right edge of the window. At this point, the sender is able to resume normal data transmission and complete the transfer.
ptg999 Section 15.5 Flow Control and Window Management 707
There are numerous points that we can summarize using Figures 15-11 and 15-12:
1. The sender does not have to transmit a full window’s worth of data.
2. A single segment from the receiver acknowledges data and slides the win- dow to the right at the same time. This is because the window advertise- ment is relative to the ACK number in the same segment.
3. The size of the window can decrease, as shown by the series of ACKs in Figure 15-11, but the right edge of the window does not move left, so as to avoid window shrinkage.
4. The receiver does not have to wait for the window to fill before sending an ACK.
In addition to these points, it is instructive to look at the throughput this connec- tion achieves as a function of time. Using Wireshark’s Statistics | TCP Stream Graph
| Throughput Graph function, we observe the time series as shown in Figure 15-13.
Figure 15-13 With a relatively large receive buffer, a significant amount of data can be transferred even before the receiving application reads any data from the network.
ptg999 Here we see an interesting behavior. Even before the receiving application has
consumed any data, the connection still achieves a throughput of approximately 1.3MB/s. This continues until approximately time 0.10. After that, the throughput is essentially zero until the receiver begins consuming data much later (after time 20).
15.5.3 Silly Window Syndrome (SWS)
Window-based flow control schemes, especially those that do not use fixed-size segments (such as TCP), can fall victim to a condition known as the silly window syndrome (SWS). When it occurs, small data segments are exchanged across the connection instead of full-size segments [RFC0813]. This leads to undesirable inef- ficiency because each segment has relatively high overhead—a small number of data bytes relative to the number of bytes in the headers.
SWS can be caused by either end of a TCP connection: the receiver can adver- tise small windows (instead of waiting until a larger window can be advertised), and the sender can transmit small data segments (instead of waiting for addi- tional data to send a larger segment). Correct avoidance of silly window syndrome requires a TCP to implement rules specifically for this purpose, whether operating as a sender or a receiver. TCP never knows ahead of time how a peer TCP will behave. The following rules are applied:
1. When operating as a receiver, small windows are not advertised. The receive algorithm specified by [RFC1122] is to not send a segment advertising a larger window than is currently being advertised (which can be 0) until the window can be increased by either one full-size segment (i.e., the receive MSS) or by one-half of the receiver’s buffer space, whichever is smaller.
Note that there are two cases where this rule can come into play: when buf- fer space has become available because of an application consuming data from the network, and when TCP must respond to a window probe.
2. When sending, small segments are not sent and the Nagle algorithm gov- erns when to send. Senders avoid SWS by not transmitting a segment unless at least one of the following conditions is true:
a. A full-size (send MSS bytes) segment can be sent.
b. TCP can send at least one-half of the maximum-size window that the other end has ever advertised on this connection.
c. TCP can send everything it has to send and either (i) an ACK is not cur- rently expected (i.e., we have no outstanding unacknowledged data) or (ii) the Nagle algorithm is disabled for this connection.
Condition (a) is the most straightforward and directly avoids the high-over- head segment problem. Condition (b) deals with hosts that always advertise tiny windows, perhaps smaller than the segment size. Condition (c) prevents TCP from
ptg999 Section 15.5 Flow Control and Window Management 709
sending small segments when there is unacknowledged data waiting to be ACKed and the Nagle algorithm is enabled. If the sending application is doing small writes (e.g., smaller than the segment size), condition (c) avoids silly window syndrome.
These three conditions also let us answer the following question: If the Nagle algorithm prevents us from sending small segments while there is outstanding unacknowledged data, how small is small? From condition (a) we see that “small”
means that the number of bytes is less than the SMSS (i.e., the largest packet size that does not exceed the PMTU or the receiver’s MSS). Condition (b) comes into play only with older, primitive hosts or when a small advertised window is used because of a limited receive buffer size.
Condition (b) of step 2 requires that the sender keep track of the maximum window size advertised by the other end. This is an attempt by the sender to guess the size of the other end’s receive buffer. Although the size of the receive buffer could decrease while the connection is established, in practice this is rare. Further- more, recall that TCP avoids window shrinkage.
15.5.3.1 Example
We will now present a detailed example to see silly window syndrome avoidance in action; this example also involves the persist timer. We will use our sock pro- gram with a Windows XP sending host and a FreeBSD receiver, doing three 2048- byte writes to the network. The command at the sender is as follows:
C:\> sock -i -n 3 -w 2048 10.0.0.8 6666
The corresponding command at the receiver is
FreeBSD% sock -i -s -P 15 -p 2 -r 256 -R 3000 6666
This fixes the receive buffer at 3000 bytes, causes an initial delay of 15s before reading from the network, injects 2s of delay between each read, and sets each read amount to be 256 bytes. The reason for the initial pause is to let the receiver’s buffer fill, ultimately forcing the transmitter to stop. By having the receiver then perform small reads from the network, we expect to see it perform silly window syndrome avoidance. Figure 15-14 is the trace as displayed by Wireshark.
The contents of the entire connection are displayed in the figure. Packet lengths are described in terms of how many TCP payload bytes are included in each segment. During connection establishment, the receiver advertises a window of 3000 bytes with an MSS of 1460 bytes. The sender sends a 1460-byte packet (packet 4) at time 0.052 and 588 bytes (packet 5) at time 0.053. The sum of these sizes equals the 2048-byte write size used by the application. Packet 6 acknowl- edges both data packets from the sender and provides a window advertisement of 952 bytes (3000 – 1460 – 588 = 952).
The 952-byte window (packet 6) is not as large as a full MSS, so the Nagle algorithm at the sender prevents filling it immediately. Instead, we see a delay