I've recently had a 2nd incident where inn2 got weirdly stuck and
caused the system to run out out of processes.

In the broken state, the global process table is full, and innd has
many unreaped nnrp zombies.  (In my installation, nnrpd has been moved
aside and replaced by a wrapper, but that doesn't seem likely to be
relevant.)

In the most recent incident I was able to strace innd.
After I discovered that many zombies had 12430 as parent:

# ps -Hfp 12430
UID        PID  PPID  C STIME TTY          TIME CMD
news     12430     1  0 Apr23 ?        00:06:42 /usr/lib/news/bin/innd
# service innd stop
/usr/sbin/service: 44: Cannot fork
# kill 12430
# ps -Hfp 12430
UID        PID  PPID  C STIME TTY          TIME CMD
news     12430     1  0 Apr23 ?        00:06:42 /usr/lib/news/bin/innd
# strace -p 12430
strace: Process 12430 attached
select(98, [12 14 15 17 30 35 39 42 45 59 65 80 90 97], [], NULL, {tv_sec=293, tv_usec=975911}) = 1 (in [15], left {tv_sec=292, tv_usec=501355})
rt_sigprocmask(SIG_SETMASK, [HUP USR1 PIPE TERM CHLD], NULL, 8) = 0
gettimeofday({tv_sec=1778450718, tv_usec=948130}, NULL) = 0
accept(15, {sa_family=AF_INET, sin_port=htons(38066), sin_addr=inet_addr("172.17.207.6")}, [128->16]) = 34
setsockopt(34, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
fcntl(34, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(34, F_SETFL, O_RDWR)              = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f1261323a90) = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "<59>May 10 23:05:18 innd: SERVER"..., 99, MSG_NOSIGNAL, NULL, 0) = 99
close(34)                               = 0
rt_sigprocmask(SIG_SETMASK, [HUP USR1 PIPE TERM CHLD], NULL, 8) = 0
select(98, [12 14 15 17 30 35 39 42 45 59 65 80 90 97], [], NULL, {tv_sec=300, tv_usec=0}^Cstrace: Process 12430 detached
 <detached ...>

# kill 12430
# strace -p 12430
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
^C-bash: fork: Interrupted system call

# kill -9 12430
-bash: wait_for: No record of process 23234

And it turns out that it *had* died and all was well.  So sending it a
SIGTERM seems to have fixed it.

This strace does IMO show pathological behaviour.  innd calls accept
and fork, but you will observe that it doesn't reap!

Ian.

Hi Ian,
[...]

Looking at what has been fixed since the 2.6.4 version, maybe you
reached the maximum number of file descriptors your system can handle?
I noted in the change log of INN 2.7.2 that "innd no longer malfunctions
nor throttles when the maximum number of file descriptors supported by
the system is reached.  If needing to use more file descriptors than the
default system limit, a new LARGE_FD_SETSIZE option can be set at build
time.  See the documentation for rlimitnofile in inn.conf for more
information."

You can find the mentioned documentation here:
https://www.eyrie.org/~eagle/software/inn/docs/inn.conf.html


It could be unrelated to your issue, though.
Just telling you in case it could hint at something.