#558608 libapache2-mod-wsgi: apache2 reports segfaults following installation

#558608#5
Date:
2009-11-29 15:54:00 UTC
From:
To:
After installing libapache2-mod-wsgi and restaring Apache, the server
immediately crashes. I could reproduce that bug on 2 different machines
running lenny (although it is not 100% deterministic).

gdb backtrace:

#0  0x00007f884cb2eb97 in sem_wait () from /lib/libpthread.so.0
#1  0x00007f8845973bc8 in PyThread_acquire_lock (lock=0x0, waitflag=1) at ../Python/thread_pthread.h:349
#2  0x00007f8845c0e684 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so
#3  0x00007f8845c176b9 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so
#4  0x0000000000438e1d in ap_run_child_init (pchild=0x2360888, s=0x21d8968) at /build/buildd/apache2-2.2.9/server/config.c:156
#5  0x000000000044da29 in child_main (child_num_arg=<value optimized out>) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:518
#6  0x000000000044dfd4 in make_child (s=0x21d8968, slot=0) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:777
#7  0x000000000044ec16 in ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
     at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:912
#8  0x0000000000425be5 in main (argc=3, argv=0x7fff6eef4c38) at /build/buildd/apache2-2.2.9/server/main.c:732

#558608#10
Date:
2009-11-29 22:17:29 UTC
From:
To:
I rebuilt mod-wsgi with DEB_BUILD_OPTIONS=nostrip. Here's another
traceback:

#0  0x00007f06faf51b97 in sem_wait () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00007f06f495dbc8 in PyThread_acquire_lock (lock=0x0, waitflag=1) at ../Python/thread_pthread.h:349
	status = 120
#2  0x00007f06f4bf87d0 in wsgi_python_child_init () from /usr/lib/apache2/modules/mod_wsgi.so
No symbol table info available.
#3  0x00007f06f4c01805 in wsgi_hook_child_init () from /usr/lib/apache2/modules/mod_wsgi.so
No symbol table info available.
#4  0x0000000000438e1d in ap_run_child_init (pchild=0x228af88, s=0x213a968) at /build/buildd/apache2-2.2.9/server/config.c:156
	n = 4
#5  0x000000000044da29 in child_main (child_num_arg=<value optimized out>) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:518
	ptrans = (apr_pool_t *) 0x227be28
	allocator = (apr_allocator_t *) 0x2289d10
	status = <value optimized out>
	i = <value optimized out>
	lr = <value optimized out>
	pollset = <value optimized out>
	sbh = <value optimized out>
	bucket_alloc = <value optimized out>
	last_poll_idx = <value optimized out>
#6  0x000000000044dfd4 in make_child (s=0x213a968, slot=0) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:777
No locals.
#7  0x000000000044ec16 in ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
     at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:912
	status = 11
	pid = {pid = -1, in = 0x0, out = 0x0, err = 0x212e040}
	child_slot = <value optimized out>
	exitwhy = 6
	processed_status = <value optimized out>
	index = <value optimized out>
	remaining_children_to_start = 0
	rv = <value optimized out>
#8  0x0000000000425be5 in main (argc=3, argv=0x7fff03d7aff8) at /build/buildd/apache2-2.2.9/server/main.c:732
	c = 0 '\0'
	configtestonly = 0
	confname = 0x45033a "/etc/apache2/apache2.conf"
	def_server_root = 0x459611 ""
	temp_error_log = 0x0
	error = <value optimized out>
	process = <value optimized out>
	server_conf = (server_rec *) 0x213a968
	pglobal = (apr_pool_t *) 0x2132158
	pconf = (apr_pool_t *) 0x2134168
	plog = (apr_pool_t *) 0x2168308
	ptemp = (apr_pool_t *) 0x216c328
	pcommands = (apr_pool_t *) 0x2136178
	opt = (apr_getopt_t *) 0x2136260
	rv = <value optimized out>
	mod = <value optimized out>
	optarg = 0x7f06f7fc75f0 "H\215=)\020 "

#558608#19
Date:
2009-12-01 16:00:56 UTC
From:
To:
Here's how to reproduce the bug in a clean sid chroot:

# apt-get install apache2
[snip]
Unpacking procps (from .../procps_1%3a3.2.8-2_i386.deb) ...
Selecting previously deselected package libcap2.
Unpacking libcap2 (from .../libcap2_1%3a2.17-2_i386.deb) ...
Selecting previously deselected package libmagic1.
Unpacking libmagic1 (from .../libmagic1_5.03-3_i386.deb) ...
Selecting previously deselected package libpcre3.
Unpacking libpcre3 (from .../libpcre3_7.8-3_i386.deb) ...
Selecting previously deselected package mime-support.
Unpacking mime-support (from .../mime-support_3.46-1_all.deb) ...
Selecting previously deselected package libaprutil1-dbd-sqlite3.
Unpacking libaprutil1-dbd-sqlite3 (from .../libaprutil1-dbd-sqlite3_1.3.9+dfsg-3_i386.deb) ...
Selecting previously deselected package libaprutil1-ldap.
Unpacking libaprutil1-ldap (from .../libaprutil1-ldap_1.3.9+dfsg-3_i386.deb) ...
Selecting previously deselected package apache2.2-bin.
Unpacking apache2.2-bin (from .../apache2.2-bin_2.2.14-3_i386.deb) ...
Selecting previously deselected package apache2-utils.
Unpacking apache2-utils (from .../apache2-utils_2.2.14-3_i386.deb) ...
Selecting previously deselected package apache2.2-common.
Unpacking apache2.2-common (from .../apache2.2-common_2.2.14-3_i386.deb) ...
Selecting previously deselected package apache2-mpm-worker.
Unpacking apache2-mpm-worker (from .../apache2-mpm-worker_2.2.14-3_i386.deb) ...
Selecting previously deselected package apache2.
Unpacking apache2 (from .../apache2_2.2.14-3_i386.deb) ...
Setting up procps (1:3.2.8-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode.
************************************
All rc.d operations denied by policy
************************************
Setting up libcap2 (1:2.17-2) ...
Setting up libmagic1 (5.03-3) ...
Setting up libpcre3 (7.8-3) ...
Setting up mime-support (3.46-1) ...
update-alternatives: using /usr/bin/see to provide /usr/bin/view (view) in auto mode.
Setting up libaprutil1-dbd-sqlite3 (1.3.9+dfsg-3) ...
Setting up libaprutil1-ldap (1.3.9+dfsg-3) ...
Setting up apache2.2-bin (2.2.14-3) ...
Setting up apache2-utils (2.2.14-3) ...
Setting up apache2.2-common (2.2.14-3) ...
Enabling site default.
Enabling module alias.
Enabling module autoindex.
Enabling module dir.
Enabling module env.
Enabling module mime.
Enabling module negotiation.
Enabling module setenvif.
Enabling module status.
Enabling module auth_basic.
Enabling module deflate.
Enabling module authz_default.
Enabling module authz_user.
Enabling module authz_groupfile.
Enabling module authn_file.
Enabling module authz_host.
Setting up apache2-mpm-worker (2.2.14-3) ...
************************************
All rc.d operations denied by policy
************************************
Setting up apache2 (2.2.14-3) ...

# /etc/init.d/apache2 start
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
.

# apt-get install libapache2-mod-wsgi
[snip]
Unpacking libreadline6 (from .../libreadline6_6.0-5_i386.deb) ...
Selecting previously deselected package libdb4.5.
Unpacking libdb4.5 (from .../libdb4.5_4.5.20-13.1_i386.deb) ...
Selecting previously deselected package python2.5-minimal.
Unpacking python2.5-minimal (from .../python2.5-minimal_2.5.4-3_i386.deb) ...
Selecting previously deselected package python2.5.
Unpacking python2.5 (from .../python2.5_2.5.4-3_i386.deb) ...
Selecting previously deselected package python-minimal.
Unpacking python-minimal (from .../python-minimal_2.5.4-2_all.deb) ...
Selecting previously deselected package python.
Unpacking python (from .../python_2.5.4-2_all.deb) ...
Selecting previously deselected package libapache2-mod-wsgi.
Unpacking libapache2-mod-wsgi (from .../libapache2-mod-wsgi_2.6-1_i386.deb) ...
Setting up libreadline6 (6.0-5) ...
Setting up libdb4.5 (4.5.20-13.1) ...
Setting up python2.5-minimal (2.5.4-3) ...
Setting up python2.5 (2.5.4-3) ...
Setting up python-minimal (2.5.4-2) ...
Setting up python (2.5.4-2) ...
Setting up libapache2-mod-wsgi (2.6-1) ...
************************************
All rc.d operations denied by policy
************************************

# /etc/init.d/apache2 reload
Reloading web server config: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
.

# tail -f /var/log/apache2/error.log
[Tue Dec 01 15:31:32 2009] [notice] Apache/2.2.14 (Debian) configured -- resuming normal operations
[Tue Dec 01 15:32:12 2009] [notice] SIGUSR1 received.  Doing graceful restart
apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[Tue Dec 01 15:32:12 2009] [notice] Apache/2.2.14 (Debian) configured -- resuming normal operations
[Tue Dec 01 15:32:12 2009] [notice] child pid 9853 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:12 2009] [notice] child pid 9854 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:14 2009] [notice] child pid 9855 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:15 2009] [notice] child pid 9857 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:16 2009] [notice] child pid 9859 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:17 2009] [notice] child pid 9862 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:18 2009] [notice] child pid 9864 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:19 2009] [notice] child pid 9866 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:20 2009] [notice] child pid 9868 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:21 2009] [notice] child pid 9870 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:22 2009] [notice] child pid 9872 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:23 2009] [notice] child pid 9874 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:24 2009] [notice] child pid 9876 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:24 2009] [notice] child pid 9878 exit signal Segmentation fault (11)
[Tue Dec 01 15:32:24 2009] [notice] child pid 9879 exit signal Segmentation fault (11)

#558608#28
Date:
2010-02-01 20:36:31 UTC
From:
To:
also sprach martin f. krafft <madduck@debian.org> [2010.02.01.1153 +1300]:

Sorry for not commenting about the bump of severity. I made this bug
grave because in current state of affairs, mod_wsgi is unusable in
sid. I did verify it with 2.8-1 and can reproduce the bug at will.

Note that I could not reproduce the problem with 3.1-1/experimental.
Jakub, you just marked this bug as found/3.1-1 yesterday. Can you
please verify this and possibly provide command output?

  Setting up libapache2-mod-wsgi (3.1-1) ...
  Installing new version of config file /etc/apache2/mods-available/wsgi.conf ...
  Reloading web server config: apache2

  [Mon Feb 01 20:31:20 2010] [warn] NameVirtualHost *:801 has no VirtualHosts
  [Mon Feb 01 20:31:20 2010] [notice] SIGUSR1 received.  Doing graceful restart
  [Mon Feb 01 20:31:20 2010] [warn] NameVirtualHost *:801 has no VirtualHosts
  [Mon Feb 01 20:31:20 2010] [warn] mod_wsgi: Compiled for Python/2.5.4.
  [Mon Feb 01 20:31:20 2010] [warn] mod_wsgi: Runtime using Python/2.5.5.
  [Mon Feb 01 20:31:20 2010] [notice] Apache/2.2.14 (Debian) mod_wsgi/3.1 Python/2.5.5 configured -- resuming normal operations

  [no further output]

#558608#33
Date:
2010-02-01 21:09:49 UTC
From:
To:
* martin f krafft <madduck@debian.org>, 2010-02-02, 09:36:

I originally set severity to important because restarting Apache (which
is an obvious thing to try in such a case) solved the problem for me.
Also, I believe that upgrades never trigger this bug.

# apt-get install apache2
[snip]
Unpacking procps (from .../procps_1%3a3.2.8-2_i386.deb) ...
Selecting previously deselected package libcap2.
Unpacking libcap2 (from .../libcap2_1%3a2.17-2_i386.deb) ...
Selecting previously deselected package libmagic1.
Unpacking libmagic1 (from .../libmagic1_5.04-1_i386.deb) ...
Selecting previously deselected package libpcre3.
Unpacking libpcre3 (from .../libpcre3_7.8-3_i386.deb) ...
Selecting previously deselected package mime-support.
Unpacking mime-support (from .../mime-support_3.48-1_all.deb) ...
Selecting previously deselected package libaprutil1-dbd-sqlite3.
Unpacking libaprutil1-dbd-sqlite3 (from .../libaprutil1-dbd-sqlite3_1.3.9+dfsg-3_i386.deb) ...
Selecting previously deselected package libaprutil1-ldap.
Unpacking libaprutil1-ldap (from .../libaprutil1-ldap_1.3.9+dfsg-3_i386.deb) ...
Selecting previously deselected package apache2.2-bin.
Unpacking apache2.2-bin (from .../apache2.2-bin_2.2.14-5_i386.deb) ...
Selecting previously deselected package apache2-utils.
Unpacking apache2-utils (from .../apache2-utils_2.2.14-5_i386.deb) ...
Selecting previously deselected package apache2.2-common.
Unpacking apache2.2-common (from .../apache2.2-common_2.2.14-5_i386.deb) ...
Selecting previously deselected package apache2-mpm-worker.
Unpacking apache2-mpm-worker (from .../apache2-mpm-worker_2.2.14-5_i386.deb) ...
Selecting previously deselected package apache2.
Unpacking apache2 (from .../apache2_2.2.14-5_i386.deb) ...
Setting up procps (1:3.2.8-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode.
************************************
All rc.d operations denied by policy
************************************
Setting up libcap2 (1:2.17-2) ...
Setting up libmagic1 (5.04-1) ...
Setting up libpcre3 (7.8-3) ...
Setting up mime-support (3.48-1) ...
Setting up libaprutil1-dbd-sqlite3 (1.3.9+dfsg-3) ...
Setting up libaprutil1-ldap (1.3.9+dfsg-3) ...
Setting up apache2.2-bin (2.2.14-5) ...
Setting up apache2-utils (2.2.14-5) ...
Setting up apache2.2-common (2.2.14-5) ...
Enabling site default.
Enabling module alias.
Enabling module autoindex.
Enabling module dir.
Enabling module env.
Enabling module mime.
Enabling module negotiation.
Enabling module setenvif.
Enabling module status.
Enabling module auth_basic.
Enabling module deflate.
Enabling module authz_default.
Enabling module authz_user.
Enabling module authz_groupfile.
Enabling module authn_file.
Enabling module authz_host.
Setting up apache2-mpm-worker (2.2.14-5) ...
************************************
All rc.d operations denied by policy
************************************
Setting up apache2 (2.2.14-5) ...

# /etc/init.d/apache2 start
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
.

# apt-get install -t experimental libapache2-mod-wsgi
[snip]
Unpacking libreadline6 (from .../libreadline6_6.1-1_i386.deb) ...
Selecting previously deselected package libdb4.5.
Unpacking libdb4.5 (from .../libdb4.5_4.5.20-13.1_i386.deb) ...
Selecting previously deselected package python2.5-minimal.
Unpacking python2.5-minimal (from .../python2.5-minimal_2.5.5-1_i386.deb) ...
Selecting previously deselected package python2.5.
Unpacking python2.5 (from .../python2.5_2.5.5-1_i386.deb) ...
Selecting previously deselected package python-minimal.
Unpacking python-minimal (from .../python-minimal_2.5.4-9_all.deb) ...
Selecting previously deselected package python.
Unpacking python (from .../python_2.5.4-9_all.deb) ...
Selecting previously deselected package libapache2-mod-wsgi.
Unpacking libapache2-mod-wsgi (from .../libapache2-mod-wsgi_3.1-1_i386.deb) ...
Setting up libreadline6 (6.1-1) ...
Setting up libdb4.5 (4.5.20-13.1) ...
Setting up python2.5-minimal (2.5.5-1) ...
Setting up python2.5 (2.5.5-1) ...
Setting up python-minimal (2.5.4-9) ...
Setting up python (2.5.4-9) ...
Setting up libapache2-mod-wsgi (3.1-1) ...
************************************
All rc.d operations denied by policy
************************************

# /etc/init.d/apache2 reload
Reloading web server config: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
.

# tail -f /var/log/apache2/error.log
[Mon Feb 01 20:52:24 2010] [notice] Apache/2.2.14 (Debian) configured -- resuming normal operations
[Mon Feb 01 20:52:24 2010] [notice] child pid 4727 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:25 2010] [notice] child pid 4728 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:26 2010] [notice] child pid 4730 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:27 2010] [notice] child pid 4732 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:27 2010] [notice] child pid 4733 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:29 2010] [notice] child pid 4735 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:30 2010] [notice] child pid 4737 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:31 2010] [notice] child pid 4739 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:31 2010] [notice] child pid 4741 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:33 2010] [notice] child pid 4743 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:34 2010] [notice] child pid 4746 exit signal Segmentation fault (11)
[Mon Feb 01 20:52:35 2010] [notice] child pid 4748 exit signal Segmentation fault (11)

#558608#38
Date:
2010-02-02 02:43:50 UTC
From:
To:
severity 558608 important
notfound 558608 2.8-2
retitle 558608 libapache2-mod-wsgi: apache2 reports segfaults following installation
thanks

also sprach Jakub Wilk <jwilk@debian.org> [2010.02.02.1009 +1300]:

Indeed, sorry — restarting does seem to fix things.

And indeed, the problem is the same with the experimental version.

The temporary fix in -2 fixes the issue too, but it does cut off all
existing connections to the httpd, which may not be acceptable.