#1067104 server stalls: AH00046: child process 2876749 still did not exit, sending a SIGKILL

Package:
apache2
Source:
apache2
Description:
Apache HTTP Server
Submitter:
Yaroslav Halchenko
Date:
2026-01-17 12:13:02 UTC
Severity:
normal
#1067104#5
Date:
2024-03-18 12:59:44 UTC
From:
To:
Server was working just fine for years and recently started to stall
completely after 3-7 days of functioning normally.  error logs get filled up
first with AH03490 and then eventually with AH00045 messages:

    [Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
    ...
    [Sun Mar 17 22:00:42.201774 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
    [Sun Mar 17 22:00:42.995574 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
    [Sun Mar 17 22:00:42.998488 2024] [mpm_event:notice] [pid 2649373:tid 139846579189632] AH00492: caught SIGWINCH, shutting down gracefully
    [Sun Mar 17 22:00:46.358981 2024] [core:warn] [pid 2649373:tid 139846579189632] AH00045: child process 2649375 still did not exit, sending a SIGTERM
    [Sun Mar 17 22:00:46.359064 2024] [core:warn] [pid 2649373:tid 139846579189632] AH00045: child process 2649376 still did not exit, sending a SIGTERM

until I restart the beast.

    $> grep AH03490 error.log | wc -l
    70404
    $> grep AH00045 error.log | wc -l
    48

Server has a number of virtualserver's configured.
Seems has started about a month ago

	$> for e in error.log*; do zgrep AH03490 $e| head -n 1 ; done
	[Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
	[Mon Mar 11 16:47:41.181900 2024] [mpm_event:error] [pid 1172065:tid 140192799893376] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
	[Tue Mar 05 00:00:12.307813 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
	[Sun Feb 25 03:23:33.382200 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
	[Sat Feb 24 01:02:29.148887 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
	[Tue Feb 13 14:28:00.653754 2024] [mpm_event:error] [pid 2434335:tid 140300052350848] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.

and likely after I configured some wsgi

	$> zgrep apache /var/log/dpkg.log.* | grep 2024
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 install libapache2-mod-python:amd64 <none> 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 configure libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 <none>
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-configured libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:34:25 status installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:51:18 status installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:51:19 remove libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 <none>
	/var/log/dpkg.log.2.gz:2024-02-02 12:51:19 status half-configured libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status half-installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status config-files libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 install libapache2-mod-wsgi-py3:amd64 <none> 4.9.4-1+b2
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-installed libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 configure libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 <none>
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-configured libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
	/var/log/dpkg.log.2.gz:2024-02-02 12:52:14 status installed libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2


wsgi is still in use but not mod-python (was enabled and then disabled and
uninstalled) so might be relating to that.  It is in use so can't just
disable wsgi ATM.
#1067104#10
Date:
2024-03-21 12:00:45 UTC
From:
To:
Am 18.03.24 um 13:59 schrieb Yaroslav Halchenko:

Have you tried increasing ServerLimit as the warning suggests?

Apart from that, it is probably the same as
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057126 . mod_wsgi or
some python script is preventing apache processes from dying and they
accumulate until the scroeboard is full. Which versions of the wsgi
related packages are you using?
#1067104#15
Date:
2024-03-21 12:12:07 UTC
From:
To:
I think "my" https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067104
is a duplicate of this one.  I blame mod_wsgi since this I believe
started to happen after I started to use it.

FWIW here is a dirty workaround script I just crafted with chatgpt to
monitor/restart apache2 as soon as it starts happening (doesn't happen upon
every maintenance event for me I believe). Let me know if I should gather any
additional information.

	#!/bin/bash

	set -eu

	# Define the lock file and log directory
	lock_file="/var/log/apache-scoreboard-restart/lock.lck"
	log_dir="/var/log/apache-scoreboard-restart/"

	# Ensure the log directory exists
	mkdir -p "$log_dir"

	# Attempt to acquire a lock
	exec 200>"$lock_file"
	if ! flock -n 200 ; then
		echo "Another instance is running."
		exit 0
	fi

	# Function to perform actions when the specified log line is found
	handle_scoreboard_full() {
		local timestamp=$(date --iso-8601=seconds)
		local log_file="${log_dir}${timestamp}.log"

		echo "Logging system information to $log_file."
		{ ps auxw -H; echo "---"; lsof; } > "$log_file"

		echo "Reloading Apache." >> "$log_file"
		service apache2 reload

		echo "Sleeping for a minute." >> "$log_file"
		sleep 60
	}

	# Monitor the Apache error log
	while true; do
		tail --follow=name /var/log/apache2/error.log | while read line ; do
			if echo "$line" | grep -q "AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit." ; then
			handle_scoreboard_full
			break  # so we start with a fresh tail
			fi
		done
	done
#1067104#20
Date:
2024-03-21 14:16:46 UTC
From:
To:
"All ingenious is simple" -- thanks for sharing.  I might redo following
your example but to check more frequently.
#1067104#25
Date:
2024-03-21 13:48:27 UTC
From:
To:
2024-03-21 13:12 skrev Yaroslav Halchenko:

My workaround is simpler, I have this line in root's crontab:

  5 * * * * curl --silent --max-time 5 --output /dev/null
http://localhost/trac/ || systemctl restart apache2

It seems to restart Apache once every 5-8 days, according to the notices
I see from Zabbix. The frequency might very well be related to the
number of accesses to the server.