In openssh/1:7.8p1-1, the default for IPQoS changed from
IPQoS lowdelay throughput
to
IPQoS af21 cs1
Good reasons for this change are given in
https://lists.gt.net/openssh/commits/71079.
Now since the old ssh used TOS values, matching them with iptables
naturally involed the tos module. Your match for bulk traffic would
usually look like this:
iptables -m tos --tos Maximize-Throughput ...
Unfortunately, that becomes "08x/0x3f". That interacts badly with DSCP
class af21. IPTOS_DSCP_AF21 is valued 0x48. The Maximize-Throuput match
now matches interactive traffic. This is very bad.
What I don't understand is why this happens though. The 0x3f mask used
by iptables here is supposed to exclude the ECN bits. DSCP is supposed
to coexist with ECN, so it shouldn't be setting any ECN bits. Why would
it match interactive traffic as bulk then? <netinet/ip.h>, which defines
IPTOS_DSCP_AF21 as 0x48, also defines IPTOS_ECN_MASK as 0x3. This
suggests that iptables' ECN mask is wrong. It should be using 0xfc
rather than 0x3f.
Unfortunately, this is deployed now and ssh's new default breaks users
of -m tos (that matched ssh's old default) now. Thus I suggest reverting
the IPQoS change until iptables has been fixed.
And fixing iptables is going to be "interesting". It also defines --tos
Minimize-Cost, which happens to be bit 6 (RFC 1349). Bit 6 and 7 are ECN
bits though. So offering Minimize-Cost with an ECN mask simply won't
work. I guess the best thing we can do here is acknowledge that TOS and
ECN don't work well together. Indeed the relevant RFCs define bit 7 as
"must be zero". This suggests changing the mask to 0xff is in order.
For ssh, I recommend temporarily reverting to the old default to give
iptables some time.
Helmut
the behaviour of already deployed firewalls. It's a thorny situation.
Here's why I conclude the mask is wrong.
Let's take RFC 1349[1], which added the Minimize-Cost bit to ToS. First
off, realise that this conflicts with ECN, that's just part of the can
of worms that is ToS, and it complicates a solution to this issue even
more.
RFC 1349 chapter 3 defines the Type of Service Octet as:
The Type of Service octet consists of three fields:
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| | | |
| PRECEDENCE | TOS | MBZ |
| | | |
+-----+-----+-----+-----+-----+-----+-----+-----+
Note how the bit endianness is different than, for example, usual x86
diagrams. I think this is where the wrong mask stems from: the author of
the wrong mask was accustomed to diagrams with a different bit
endianness and ended up confused.
Now let's look at chapter 4 which defines this TOS field from bits 3
through 6:
1000 -- minimize delay
0100 -- maximize throughput
0010 -- maximize reliability
0001 -- minimize monetary cost
0000 -- normal service
And let's compare that to:
# iptables -m tos --help
iptables v1.6.0
[...]
tos match options:
[!] --tos value[/mask] Match Type of Service/Priority field value
[!] --tos symbol Match TOS field (IPv4 only) by symbol
Accepted symbolic names for value are:
(0x10) 16 Minimize-Delay
(0x08) 8 Maximize-Throughput
(0x04) 4 Maximize-Reliability
(0x02) 2 Minimize-Cost
(0x00) 0 Normal-Service
Take a good look at these hexadecimals corresponding to the symbolic
names. They match the byte from RFC 1349 only if you flip the
bit-endianness such that the least significant bit is on the right
(Minimize-Cost has the lowest numerical value). Note that these
hexadecimals are correct; it is only the mask that is wrong.
This is on stretch/oldstable, but the help is no different on
buster/stable. I'll continue with a stretch system, though.
We can make it more concrete. Let's create an iptables rule with
numerical values that matches DSCP CS6, which corresponds to IP
Precendence 6, numerical value 0xC0, where in the terms of RFC 1349 bits
0 and 1 are set in the PRECEDENCE portion of the ToS octet.
# iptables -I INPUT -m tos --tos 0xc0 -j NFLOG --nflog-group 2
Ping it:
$ ping -Q 0xc0 -c 1 10.0.1.1
And take a look at the packet in the PCAP log file of that nflog, with
Wireshark:
Internet Protocol Version 4, Src: 10.0.1.133, Dst: 10.0.1.1
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 84
Identification: 0x5029 (20521)
Flags: 0x4000, Don't fragment
Time to live: 64
Protocol: ICMP (1)
Header checksum: 0xd33a [validation disabled]
[Header checksum status: Unverified]
Source: 10.0.1.133
Destination: 10.0.1.1
This proves:
- That -m tos --tos 0xc0 matches a packet that has 0xc0 in the DS Field
(because this is the only rule in the firewall logging to that nflog
group)
- That 0xc0 means DSCP CS6, because I believe Wireshark's analysis, it's
been correct in different instances of looking at the DSCP field with
packets generated by several systems.
So that means that the mask 0x3f is a mistake.
But changing the mask to 0xfc will make -m tos --tos Minimize-Cost break
because that is actually one of the ECN bits. I tested it and --tos
0x02/0xfc predictably did not match ping -Q 0x02. Changing the mask to
0xff causes less breakage, but still changes the behaviour on existing
deployments... :-(
Perhaps the best solution is to deprecate the symbolic --tos arguments,
urging everyone to exclusively use the numerical format. Put this in
NEWS so people hopefully notice, perhaps the Release Notes. And then
maybe someday DSCP and ECN will be less broken. Firewalls currently
using symbolic --tos arguments already misqualify ECN and IP Precedence
as well, it's not just DSCP.
HTH,
Peter.
[1] <https://tools.ietf.org/html/rfc1349>
I think I should emphasise the point that DSCP CS6 is the same as IP Precedence 6 *by design*, it's not a coincidence. So the experiment proves that IP Precedence 6 is encoded as 0xc0, which is what proves the layout of the octet. Peter.
G'day, TLDR; I think the current openssh TOS settings of AF21 and CS1 are a poor match for the old TOS bit fields and are causing confusion. IMHO AF22 and AF11 would be a better backwards-compatible choice. I think the iptables use of "08x/0x3f" for Maximize-Throughput is only wrong in that the mask 0x3f includes 1 bit of the DSCP "Precedence" field. I sent a message to 923879@bugs.debian.org which bounced because it's closed/archived, and found this bug as probably the most relevant open bug. Reading this bug it looks to me like RFC 1349 confused Peter Lebbing by using less-common big-endian bit numbering for their bit labels in the TOS field and not making that very clear. Everything else I've seen has the precedence bits in the Most-Significant-Bits of the octet including these; http://www.rhyshaden.com/ipdgram.htm https://my.ezycloud.com.au/knowledgebase/201204284/Quality-of-Service-QoS----DSCP-TOS-CoS-Precedence-Conversion-Chart.html This is the convention ssh is currently using when it sets AF21 as 0x48 and CS1 ax 0x20, and this appears to be what IP tables is also using. This means I believe that iptables use of "08x/0x3f" for Maximize-Throughput is only wrong in that the mask 0x3f includes 1 bit of the DSCP "Precedence" field. I think the correct mask should be 0x1f, though maybe there are reasons why you'd want to explicitly clear the LSB of the DSCP Precedence field that I'm unaware of. I stumbled upon https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879 trying to figure out problems with my QoS firewall setup and a recent Debian ssh which is using AF21 and CS1 for interactive and bulk traffic respectively. This is the new ssh upstream default, and after speaking to one of the ssh developers (dtucker) it seems TOS standards are a mess and everyone they've spoken to have given them different advice. There are now more than two RFCs and 100k of text per TOS header bit. The choice of AF21 and CS1 was taken as an attempt to pick the least-wrong option, and no-one has been able to clearly justify anything better. The reason AF21 is being classified as "Maximize-Throughput" is because AF21 sets the (old?) Maximize-Throughput bit, not the Minimize-Delay bit, despite the various DSCP docs describing AF21 as "interactive" (actually, all of AF2[123] is documented as for "low latency traffic"). Note CS1 doesn't set any of the old Delay/Throughput/Reliablity/Cost TOS bits so is likely classified as "normal" traffic, not "bulk" by most older QoS systems. See the following chart for details; https://my.ezycloud.com.au/knowledgebase/201204284/Quality-of-Service-QoS----DSCP-TOS-CoS-Precedence-Conversion-Chart.html It appears from the settings that AF21 was intended for important high-volume traffic. It looks like AF22 was intended for important low-latency traffic (it sets the Minimize-Delay bit) and is probably a better fit for ssh interactive traffic. Note that the ssh devs probably chose AF21 over AF22 because the DSCP docs say AF22 should be dropped before AF21, implying AF21 is "more important" than AF22; https://en.wikipedia.org/wiki/Differentiated_services#Assured_Forwarding However, I think the DSCP standards structure the drop-order this way to reflect that they are actually equally important (as indicated by the precedence setting of "Immediate), but high-volume traffic is more likely to need the packets during congestion. Note that AF23 which sets both Minimize-Delay AND Maximize-Throughput has an even higher drop probability, probably because that's the price you pay for asking for both high volume AND low latency. The use of CS1 for ssh bulk traffic is also problematic, since it doesn't set any of the old TOS bits and thus looks "normal" to anything only looking at the old Delay/Throughput/Reliablity/Cost TOS bits. The equivalent DSCP "Priority" precedence that does set the Maximize-Throughput bit is AF11, which does seem like a better fit than CS1. Another option would be to set the Minimize-Cost bit, but that bit is not used by DSCP so there is no DSCP class with that bit set. So personally I'm now using the following setting in both /etc/ssh/ssh_config and /etc/ssh/sshd_config to override the defaults. IPQoS af22 af11 Note the client/server settings affect the traffic in each direction independently; a server without this setting will send packets with tos=0x48 (AKA AF21) to a client sending packets with tos=0x50 (AKA AF22) that does have these settings. My only problem now is it seems at least one old Debian ssh package (1:6.7p1-5+deb8u8) and/or the VPS I'm using for one of my machines doesn't seem to honor the IPQoS setting and is just using tos=0x0... but that's mostly unrelated.
Hi,
I make yesterday some tcpdump of ssh interactive and non-interactive
sessions between machines using Debian 13.
I was checking the IPv6 DSCP field and seen unknown values from
Wireshark dissector. I came to debian openssh sources, and it keeps
reverting a 2019 OpenSSH change "as a temporary fix" because there is
bad interactions with iptables -m tos and VMWare Player (has been fixed
in 2019). On the iptables side, the situation is unclear for me.
* This debian bug let me think there is no change considered by
upstream (netfilter developers) nor debian patch produced.
* openssh debian package keep applying
debian/patches/revert-ipqos-defaults.patch
o It is the solution of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923879
* I don't really know the burden around the old TOS to DSCP transition
* I don't even know if this topic was ever relevant in an IPv6 context
* I can't really find a matching bug in upstream bugzilla. I asked
today an account creation there.
Having very little old background on this topic, I see "wrong DSCP
values" (as undeclared in official DSCP registry :
https://www.iana.org/assignments/dscp-registry/dscp-registry.xhtml) on
current debian openssh default configurations, waiting after iptables fix.
Is someone know on which bug number this have been ever reported
upstream ? I can't find with "tos" keyword in Open or Closed tickets
after 2019.
Thanks for all the fishes,
I'm not sure how much this is an issue any more. I reverted that change in forky six months or so ago in favour of upstream's switch from AF21 to EF for interactive traffic, which as far as I know doesn't run into this problem any more (EF is 0xb8, and 0xb8 & 0x3f = 0x38, so that doesn't match Maximize-Throughput = 0x08). After some discussion with OpenSSH upstream, I intend to propose backporting that set of OpenSSH patches to trixie. I think the result will be better across the board.