In a web server with ore than 1000 Virtual Hosts in /var/log/apache2/error.log
could be seen multiple strings with

====================
Segmentation fault (11)

[warn] make_sock: problem listening on port 443, file descriptor (1068) larger
than FD_SETSIZE (1024)
====================

The solution is editing __FD_SETSIZE value in /usr/include/bits/typesizes.h:
from
#define __FD_SETSIZE 1024
to
#define __FD_SETSIZE 32768
(value must be much greater than current one 1024) and rebuilding apache,
sometime php,
openssl etc.

Please make default value of __FD_SETSIZE greater than now.

tag 330105 + wontfix
severity 330105 wishlist
thanks

  that's not the right fix. Many syscalls won't work correctly if you
change that (select e.g.). If you need to use more than 1024 open file
descriptors in an application, you have to explicitely support it, as
POSIX does not allow you to do sth like that with select and friends.

  Poll can though. epoll too.

  I also have a hard time understanding why having 1000+ virtualhosts in
apache would create such problems (except if you listen of 1000+
different ports btw, but I don't think it would be a very clever setup
anyway :])

Cheers,

This one time, at band camp, Pierre HABOUZIT said:

Because you have an open fd per virtual host for logfiles.  Whether
apache has all the fd's open itself, or it pushes the logs to a named
pipe that does the splitting for apache, something still has all those
fd's open and you can run into this fairly quickly.

  Oh, good point. Though I suppose it's iff you have one logfile per
virtualhost, which is not necessarily true.

  Anyway , in that case, raising __FD_SETSIZE won't help, as it's a
kernel limitation that forbids a usual process to open more than
getdtablesize() process open, except if you performed a setrlimit
before.

  TTBOMK __FD_SETSIZE is only used for fd_set's (so select, FD_* macros,
...), and redefining it won't work (I tried already in another life)
without recompiling the libc at least -- if not the kernel too, I'm less
sure about that one -- and that would be obvioulsy binary incompatible
with the rest of the linuxes around the globe :)

  Another possibility here, is that apache:
  1. did the setrlimit
  2. opened a lot of fd's for the logfiles (or anything else)
  3. does not have <1024 available fd's anymore, hence cannot use select
     over them => you're screwed.

  If you need to use more than 1024 file descriptors in a program for
this or that reason, you should not use select at all, this is a POSIX
limitation.

  So if the submitter really cares about that problem, I'd suggest him
to reassign the bug to apache, asking for a poll()-based implemenation
(that has no 1024-fd's limitations AFAICT).

"Just" libc and everything that uses select - basically you have to
rebuild the whole archive.

AFAIK sys_select() in the kernel can handle arbitrary large number of
file descriptiors, so one option is not to use glibc's select() wrapper
but do your own. But a much better approach (both maintainability and
performance wise) is to use epoll().

Gabor

Hello,

if you raise FD_SETSIZE in /usr/include/bits/typesizes.h and then
compile a software which uses select(), this software will be able to
handle more than 1024 FDs. Recompiling glibc, the kernel or whatever
else is not necessary.

I tested this solution with UnrealIRCd and 8000 clients. Unfortunately,
I don't know enough about the way glibc works, but I found this in the
FAQ of the debian libc6 source package:

--------
2.25.   I need lots of open files.  What do I have to do?

{AJ} This is at first a kernel issue.  The kernel defines limits with
OPEN_MAX the number of simultaneous open files and with FD_SETSIZE the
number of used file descriptors.  You need to change these values in
your
kernel and recompile the kernel so that the kernel allows more open
files.  You don't necessarily need to recompile the GNU C library since
the
only place where OPEN_MAX and FD_SETSIZE is really needed in the library
itself is the size of fd_set which is used by select.

The GNU C library is now select free.  This means it internally has no
limits imposed by the `fd_set' type.  Instead all places where the
functionality is needed the `poll' function is used.

If you increase the number of file descriptors in the kernel you don't
need
to recompile the C library.
--------

Would it be possible to add a #ifndef FD_SETSIZE around the definition
in /usr/include/bits/typesizes.h in order to give other software the
possibility to change the value without modifying the system headers?


Regards,

Christoph