I’m using setsockopt to set the traffic class on sending and receive
it in control messages on receiving, for both IPv4 and IPv6.

The relevant documentation is the ip(7) manpage and, because the ipv6(7)
manpage doesn’t contain it, RFC3542.


For the receiving side, the corresponding socket options are:

• IPPROTO_IP ⇒ IP_RECVTOS, with an int argument 1 (as in the manpage)
• IPPROTO_IPV6 ⇒ IPV6_TCLASS, with an int argument 1 (as in the RFC)

The receiving CMSG is then supposed to contain the traffic class octet
as first byte in the corresponding CMSG_DATA:

       IP_RECVTOS (since Linux 2.2)
              If enabled, the IP_TOS ancillary message is passed with incoming
              packets.   It  contains  a byte which specifies the Type of Ser‐
              vice/Precedence field of the packet header.

… and…

   In the cmsghdr structure containing this ancillary data, the
   cmsg_level member will be IPPROTO_IPV6, the cmsg_type member will be
   IPV6_TCLASS, and the first byte of cmsg_data[] will be the first byte
   of the integer traffic class.

However, Linux has net/ipv6/datagram.c…

                int tclass = ipv6_get_dsfield(ipv6_hdr(skb));
                put_cmsg(msg, SOL_IPV6, IPV6_TCLASS, sizeof(tclass), &tclass);

… and net/ipv6/ipv6_sockglue.c…

                                int tclass = (int)ip6_tclass(np->rcv_flowinfo);
                                put_cmsg(&msg, SOL_IPV6, IPV6_TCLASS, sizeof(tclass), &tclass);

… both setting them as int, breaking standards/documentation-compliant
code on all big endian platforms. Same in net/ipv4/ip_sockglue.c…

                        int tos = inet->rcv_tos;
                        put_cmsg(&msg, SOL_IP, IP_TOS, sizeof(tos), &tos);
… in one place, but…

        put_cmsg(msg, SOL_IP, IP_TOS, 1, &ip_hdr(skb)->tos);

… in ip_cmsg_recv_tos(), yielding inconsistent results for IPv4(!).


For the sending side, we use:

• IPPROTO_IP, IP_TOS, with an argument…
• IPPROTO_IPV6, IPV6_TCLASS, with an argument…

This is funny. The documentation says to use a byte for IPv4…

       IP_TOS (since Linux 1.0)
              Set or receive the Type-Of-Service (TOS) field that is sent with
              every IP packet originating from this socket.   It  is  used  to
              prioritize  packets  on  the network.  TOS is a byte.  There are

… and setsockopt works with a byte argument, but for IPv6, using a
byte causes EINVAL (but this is because RFC3542 says int, overriding
the itojun draft which said byte).

Looking at the kernel code, IP_TOS indeed reads a byte if the size
given is less than 4, an int otherwise… except bpf_setsockopt, which
expects an int. OK, should be no problem for my userspace code.
IPV6_TCLASS always expects an int. Unexpected but apparently okay
wrt. the documentation.


tl;dr: Receiving traffic class values from IP traffic is broken on
big endian platforms.

I place the following suggestion for discussion, to achieve maximum
portability: put 4 bytes into the CMSG for both IPv4 and IPv6, where
the first and fourth byte are, identically, traffic class, second and
third 0.

Please forward this upstream. Thanks!

[The previous message is archived at <https://bugs.debian.org/966459>.]

ip(7) also doesn't document IP_PKTOPIONS.

[...]
getsockopt(... IP_PKTOPTIONS ...) for stream sockets.  They obviously
ought to be consistent, but mistakes happen.

[...]

Some user-space that uses getsockopt(... IP_PKTOPTIONS ...) for stream
sockets might be broken.

I searched for 'cmsg_type.*IP_TOS' on codesearch.debian.net, and found
only two instances where it was used in conjunction with IP_PKTOPTIONS.

libzorpll reads only the first byte (so is broken on big-endian):
https://sources.debian.org/src/libzorpll/7.0.1.0%7Ealpha1-1.1/src/io.cc/#L239

squid reads an int and then truncates it to a byte (so is fine):
https://sources.debian.org/src/squid/4.12-1/src/ip/QosConfig.cc/#L41
[...]

I see no point in changing the IPv6 behaviour: it seems to be
consistent with itself and with the standard, so only risks breaking
user-space that works today.

As for IPv4, changing the format of the IP_TOS field in the
IP_PKTOPIONS value looks like it would work for the two users found in
Debian.

But you should know that the highest priority for Linux API
compatibility is to avoid breaking currently working user-space.  That
means that ugly and inconsistent APIs won't get fixed if it causes a
regression for the programs people actually use.  If the API never
worked like it was supposed to on some architectures, that's not a
regression, and is lower priority.

Ben.

Ben Hutchings dixit:

Hmm, I don’t use IP_PKTOPIONS though. I’m not exactly sure I found
the correct place in the kernel for what I do.

On the sending side, I use setsockopt with either
IPPROTO_IP,IP_TOS or IPPROTO_IPV6,IPV6_TCLASS to
set the default traffic class on outgoing packets.

On the receiving side I use setsockopt with either
IPPROTO_IP,IP_RECVTOS or IPPROTO_IPV6,IPV6_RECVTCLASS
to set up the socket then recvmsg to get a cmsg(3) of
IPPROTO_IP,IP_TOS/IPPROTO_IPV6,IPV6_TCLASS from which
I read the traffic class octet.

These are where I believe I found inconsistencies
between code and documentation.

OK, I’m currently looking at the datagram case only.
This may change later if there’s enough time.
its first byte, it only works by accident on little endian,
but not elsewhere.

Hrm. It risks breaking userspace that reads an int. But the
RFC clearly says it should read the first byte, not an int.

This is why I just put this up for discussion instead of
requesting a specific change.

That being said, given that the IPv6 API is *only* documented
in the RFC and *not* documented in the Linux manpages…

(Perhaps codesearching for IPV6_TCLASS might also help.
It’s unclear how many users this has…)



In the end, what I really want, is clear documentation for
how I should implement the following file that it works on
Linux, and ideally also other systems implementing the RFC
API (FreeBSD supposedly does but needs testing):

https://github.com/tarent/ECN-Bits/blob/master/linux-c/lib/ecn.c

Given that there’s no documentation, trying to read the
coffee grounds from the kernel source, finding it doesn’t
even match the RFC (which, again, doesn’t match what itojun
proposed, for some reason), does not instigate trust in the
things I *think* I’ve found.

bye,
//mirabilos

The first instance of put_cmsg(...IP_TOS...) you found in
net/ipv4/ip_sockglue.c implements that socket option.

[...]

The RFC says that the IPV6_TCLASS option's value is an int, and that
"the first byte of cmsg_data[] will be the *first byte of the integer*
traffic class" (my emphasis).  We can infer from "the first byte of"
that cmsg_data[] will hold more than one byte.  And "the integer"
suggests that it's a C int, like the socket option.
[...]

No, the wording is *not* clear.

Ben.

for setsockopt (“option's”), not cmsg

Agreed.

So perhaps let’s try to find out what’s actually right…

Thanks for helping,
//mirabilos

For what it's worth, FreeBSD/Darwin and Windows also put 4 bytes of
data in a IPV6_TCLASS cmsg.  So whether or not it's "right", it's
consistent between three independent implementations.

Ben.

Hi Ben,

oh, thank you, I don’t have any of these systems around at the
moment, so checking them was tricky for me.

So basically I should read an int in host endianness then (or
keep the code I currently have that compares byte 0 and 3, using
the one that’s not 0, if any). Great, thank you!

After some minor porting work, it turns out that the current code
does work on MidnightBSD (equivalent to FreeBSD 10.4) for IPv6.
I guess I’ll keep ints then.

bye,
//mirabilos

Actually not. I’ve added some debugging code with…

static size_t
cmsg_actual_data_len(const struct cmsghdr *cmsg)
{
        union {
                const struct cmsghdr *cmsg;
                const unsigned char *uc;
        } ptr[(
                /* compile-time assertions */
                sizeof(socklen_t) <= sizeof(size_t)
            ) ? 1 : -1];
        ptrdiff_t pd;

        ptr[0].cmsg = cmsg;
        pd = CMSG_DATA(cmsg) - ptr[0].uc;
        return ((size_t)cmsg->cmsg_len - (size_t)pd);
}

… and dumping the received data on Linux and MidnightBSD
(the two systems I currently have access) and found varying
results but if this returns 4 I think I better consume an int.

bye,
//mirabilos

Hi,

I see it was forwarded back then to
https://lore.kernel.org/netdev/e67190b7de22fff20fb4c5c084307e0b76001248.camel@decadent.org.uk/
but there was no comment from netdev people.

Should this bug be closed?

Regards,
Salvatore