Hi, sometimes rpc.statd binds to port 631 udp which is used by cups. Therefore cups is unable to bind to its port and no printers get discovered. Rebooting the system helps as rpc.statd uses another port afterwards. Regards, Jan
This is a fundamental problem of the bindresvport() function, and not specific to rpc.statd. Reassigning to general. The 'portreserve' package provides a kluge to avoid this, but it requires other packages to register the ports that must be reserved. It also won't work reliably, because insserv runs init scripts in parallel and there is thus a race condition in the way services claim their ports from the portreserve daemon. A proper fix probably involves using systemd's socket-activation. Yes, I said systemd - which presumably means we'll have to wait another 5 years for this to be fixed. Ben.
Actually, according to the manpage: Unlike some bindresvport() implementations, the glibc implementation ignores any value that the caller supplies in sin->sin_port. Fixing this might be a useful way around the problem. I'd code up a patch, but eglibc won't take it without copyright assignment.
You can't fix that, because it can't rely on existing callers to initialise the field at all. Ben.
Systems running SE Linux tend not to have this problem. In most cases the daemons which use RPC services are not permitted to bind to any of the ports that are reserved for services and therefore such a bind attempt fails with EPERM, glibc will just decrement the port number and try again when this happens. http://etbe.coker.com.au/2007/11/06/squid-and-se-linux/ I mentioned this in the above blog post, I think it was in about 2002 that I wrote the policy to do this.
We could also patch bindresvport() to skip all ports mentioned in /etc/services, to get similar behaviour as with SE Linux. Or patch the programs using it to first try to bind to a static port that does not conflict with those in /etc/services, and if that fails fall back to bindresvport().
That would be a viable option. On my system there are 124 TCP ports listed with numbers <1024 (which seems to be the main problem area). Losing 12% of the address space seems viable. One thing to note when comparing this to SE Linux is that the SE Linux policy labels some ports that aren't in /etc/services but which are in relatively common use. One example is port 24 for LMTP. Also with SE Linux there is an easy way of adding new port labels and as the typical daemon won't be permitted to bind to an unlabeled port the sysadmin is compelled to do the correct thing. Now one could patch bindresvport() to also check /etc/services.local or some other source of configuration information about which ports are likely to be used. But getting the users to accept that will take some effort. Of course most users just don't have enough RPC traffic to generate the problem.
Or use a whitelist rather than pretending that /etc/services was complete anywhere within the last 20 years. Not to mention bindresvport() removes the freedom of the sysadmin to bind services to whatever ports she wishes. Or, say, run multiple instances of a service.
Sure, bindresvport is archaic, but workarounds already exist. In particular, Debian already adds /etc/bindresvport.blacklist and the default already contains port 631. Does the submitter have this file in place with the default contents? That seems like a much worse kluge than the existing blacklist. Allowing packages to register reserved ports however seems a useful feature. Reassign to eglibc as request for supporting /etc/bindresvport.blacklist.d ? Irrelevant. Promoting systemd for its side-effect as an amelioration for an ureliable kluge is not a strong argument. ;) [0] [0] Not intended as an argument against systemd either.
AFAIK /etc/services has always been a complete list of ports assigned by IANA. If someone makes a port commonly used without getting IANA approval that's their problem/mistake. If you make your program use bindresvport() then it means that you don't care what the port number is as long as it's in the reserved range. This generally means that it's a RPC service and the Portmapper will tell everyone which port to use or that there is some other channel to tell the clients which port to connect to (maybe a bit like the FTP two-port setup). If you run multiple instances of a service using RPC then I guess you could use different names with the Portmapper. It seems to me that the only problem is if you run multiple instances of a daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux, or some other method of telling bindresvport() to leave your port alone. That wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one of the people who was ignorant of bindresvport.blacklist).
Except that it should not get into ports used by something else. single man page in Debian (ok, amd64 main), and there is not a single reference to /etc/bindresvport.blacklist. In fact, even bindresvport() is referenced only from its own manpage and from portreserve which is another hack to work around this bug. portreserve is neither recommended/suggested, nor has any data that would allow it to work. No other daemon I know has this problem. If I install daemon foo, I can expect it to not touch any ports it hasn't been configured to use. It's just portmap/SunRPC that uses random scatter-shot that can trample on something else. So what about this: let's reserve a number of ports for portmap's exclusive usage[1]. There's like 900 unused assignments, so there's plenty of space than could be parcelled off. SunRPC has long since degenerated from something with a general purpose to a peculiarity of NFS, so not many ports are needed. Only under a pathological configuration one could exceed any reasonable static limit, and in that case bindresvport() would revert to the blacklist+scattershot. [1]. Unless the sysadmin knowingly takes them for some other purpose; no different from, say, having sshd listen on port 443.
Ignorance means not knowing. Sure there are probably some bug reports about man pages due, but it's still something you or I could have found out. apt-get source libc6 Yes, SunRPC and anything that opens a port for callback. The problem with this theory is the fact that the problem that was reported with CUPS only occurred after bindresvport() had used every port from 1023 down to 631. A casual scan of /etc/services reveals that there are no long contiguous ranges available without reserved ports. If you start at the top the common ports pop3s and imaps could be reached quite quickly. So it seems that some sort of blacklist is the only way to go. The idea of a .d directory for blacklist files such that every package installation that is likely to use some ports will automatically have a reservation is a good one. Of course there's still the corner case of trying to install CUPS (or some other daemon) after a long-running RPC service has grabbed the port. Maybe we should default to having ports such as 631, 993, 995, 873, 587, 636, 546, and 547 reserved at all times. From a quick scan of /etc/services they seem to be the most likely ports to be used in the 500-1024 range.
... Firewall port blocking can also cause such problems (denial of service). While it is a different problem, it has the same roots as SunRPC binding to undesired sockets: applications that use random sockets do not know whether they're going to get a socket they're supposed to use. Intelligent use of conntrack can help on single hosts (reducing the problem to incoming callback connections), but most sites have border policies that forbid any traffic to flow for some ports. It causes minor issues for DNS traffic (timeouts on a small fraction of the queries), for example. Yes. And we can easily maintain a current one for Debian-packaged software, although the initial build of such a blacklist will take some work. That's not such a big problem, as it will be noticed immediately and causes no surprise downtime of a service. Looks good, and we can take extra ports as bug reports. A mail to d-d-a and a short article to planet.d.o and LWN may help to raise awareness of such issues: although this _is_ a longstanding and _well known_ issue, the ways to avoid the worst problems it can cause are _not_ well known.
Oh, I completely missed that.
[...]
That seems like it would be necessary in the general case. However, if
port 631 is already on the list then it has nothing to do with the
current bug report.
In fact, the problem seems to be that bindresvport() supports IPv4 only
and therefore libtirpc (the new SunRPC client library) does not use it
(for either IPv4 or IPv6). glibc declares bindresvport6() for IPv6
addresses, but it doesn't appear to define it. So it seems like we need
to:
1. Add bindresvport6() to glibc
2. Use glibc's bindresvport{,6}() in libtirpc
3. Add configuration directory for reserving more ports (not for this
bug)
Ben.
* Henrique de Moraes Holschuh (hmh@debian.org) [110820 14:39]: Actually, the existing interface net.ipv4.ip_local_port_range seems to work quite well. And there are so many ports that for most servers it seems acceptable to limit the outgoing ports to only a tiny portion of port numbers (like 1/4th or so). Andi
This has nothing to do with bindresvport(). Ben.
No, it doesn't. And we have at least one extremely important protocol that needs as many ports as we can give it (DNS). A blacklist is the way to go, and we already have it. We just need to fill it, make it easier to extend (.d directory), tell people about it, and teach stuff other than SunRPC to use it when necessary.
# cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 The above is from one of my systems. This isn't used for RPC, presumably because they want the special <1024 port numbers that imply root ownership. 65535] Aug 21 11:42:48 ns named[2382]: using default UDP/IPv6 port range: [1024, 65535] BIND seems to use ports >1024 as well, again this is different from the typical RPC issues but does have the potential to cause problems (there are more than a few UDP ports >1024 in /etc/services). Maybe BIND should be patched to use the same port reservation procedure as RPC. Yes.
This depends on eglibc being patched as in #638810. Ben.--- libtirpc-0.2.2.orig/src/bindresvport.c +++ libtirpc-0.2.2/src/bindresvport.c @@ -58,12 +58,8 @@ return bindresvport_sa(sd, (struct sockaddr *)sin); } -#ifdef __linux__ - -#define STARTPORT 600 -#define LOWPORT 512 -#define ENDPORT (IPPORT_RESERVED - 1) -#define NPORTS (ENDPORT - STARTPORT + 1) +/* We now want to call libc's bindresvport() */ +#undef bindresvport int bindresvport_sa(sd, sa) @@ -72,17 +68,7 @@ { int res, af; struct sockaddr_storage myaddr; - struct sockaddr_in *sin; -#ifdef INET6 - struct sockaddr_in6 *sin6; -#endif - u_int16_t *portp; - static u_int16_t port; - static short startport = STARTPORT; socklen_t salen; - int nports = ENDPORT - startport + 1; - int endport = ENDPORT; - int i; if (sa == NULL) { salen = sizeof(myaddr); @@ -97,140 +83,11 @@ switch (af) { case AF_INET: - sin = (struct sockaddr_in *)sa; - salen = sizeof(struct sockaddr_in); - port = ntohs(sin->sin_port); - portp = &sin->sin_port; - break; -#ifdef INET6 + return bindresvport(sd, (struct sockaddr_in *)sa); case AF_INET6: - sin6 = (struct sockaddr_in6 *)sa; - salen = sizeof(struct sockaddr_in6); - port = ntohs(sin6->sin6_port); - portp = &sin6->sin6_port; - break; -#endif + return bindresvport6(sd, (struct sockaddr_in6 *)sa); default: errno = EPFNOSUPPORT; return (-1); } - sa->sa_family = af; - - if (port == 0) { - port = (getpid() % NPORTS) + STARTPORT; - } - res = -1; - errno = EADDRINUSE; - again: - for (i = 0; i < nports; ++i) { - *portp = htons(port++); - if (port > endport) - port = startport; - res = bind(sd, sa, salen); - if (res >= 0 || errno != EADDRINUSE) - break; - } - if (i == nports && startport != LOWPORT) { - startport = LOWPORT; - endport = STARTPORT - 1; - nports = STARTPORT - LOWPORT; - port = LOWPORT + port % (STARTPORT - LOWPORT); - goto again; - } - return (res); -} -#else - -#define IP_PORTRANGE 19 -#define IP_PORTRANGE_LOW 2 - -/* - * Bind a socket to a privileged IP port - */ -int -bindresvport_sa(sd, sa) - int sd; - struct sockaddr *sa; -{ - int old, error, af; - struct sockaddr_storage myaddr; - struct sockaddr_in *sin; -#ifdef INET6 - struct sockaddr_in6 *sin6; -#endif - int proto, portrange, portlow; - u_int16_t *portp; - socklen_t salen; - - if (sa == NULL) { - salen = sizeof(myaddr); - sa = (struct sockaddr *)&myaddr; - - if (getsockname(sd, sa, &salen) == -1) - return -1; /* errno is correctly set */ - - af = sa->sa_family; - memset(sa, 0, salen); - } else - af = sa->sa_family; - - switch (af) { - case AF_INET: - proto = IPPROTO_IP; - portrange = IP_PORTRANGE; - portlow = IP_PORTRANGE_LOW; - sin = (struct sockaddr_in *)sa; - salen = sizeof(struct sockaddr_in); - portp = &sin->sin_port; - break; -#ifdef INET6 - case AF_INET6: - proto = IPPROTO_IPV6; - portrange = IPV6_PORTRANGE; - portlow = IPV6_PORTRANGE_LOW; - sin6 = (struct sockaddr_in6 *)sa; - salen = sizeof(struct sockaddr_in6); - portp = &sin6->sin6_port; - break; -#endif - default: - errno = EPFNOSUPPORT; - return (-1); - } - sa->sa_family = af; - - if (*portp == 0) { - socklen_t oldlen = sizeof(old); - - error = getsockopt(sd, proto, portrange, &old, &oldlen); - if (error < 0) - return (error); - - error = setsockopt(sd, proto, portrange, &portlow, - sizeof(portlow)); - if (error < 0) - return (error); - } - - error = bind(sd, sa, salen); - - if (*portp == 0) { - int saved_errno = errno; - - if (error < 0) { - if (setsockopt(sd, proto, portrange, &old, - sizeof(old)) < 0) - errno = saved_errno; - return (error); - } - - if (sa != (struct sockaddr *)&myaddr) { - /* Hmm, what did the kernel assign? */ - if (getsockname(sd, sa, &salen) < 0) - errno = saved_errno; - return (error); - } - } - return (error); } -#endif --- libtirpc-0.2.2.orig/tirpc/rpc/rpc.h +++ libtirpc-0.2.2/tirpc/rpc/rpc.h @@ -79,6 +79,10 @@ #define UDPMSGSIZE 8800 #endif +/* Change the real name of our bindresvport() function so it can easily + * call libc's bindresvport() */ +#define bindresvport tirpc_bindresvport + __BEGIN_DECLS extern int get_myaddress(struct sockaddr_in *); extern int bindresvport(int, struct sockaddr_in *) __THROW; ---
Hi! TBH, I find having to use an additional registry of ports a bit annoying, when /etc/services is already there. In any case rpc.statd from upstream nfs-utils already honors /etc/services, so here's a patch doing something similar for libtirpc users. Which I've also submitted upstream because at least this can be supported everywhere, compared with the local patch for the blacklist. Ideally to have all implementations be consistent, libtirpc would switch to the libc bindresvport() implementation, and that one would honor /etc/services. Maybe glibc upstream is now more amenable to such change. :/ Attached the patch we are using locally. Thanks, Guillem