#785788 ganeti: luxid dies regularly

Package:
ganeti
Source:
ganeti
Submitter:
Joerg Jaspert
Date:
2015-05-22 09:00:18 UTC
Severity:
important
#785788#5
Date:
2015-05-20 08:23:27 UTC
From:
To:
Dear Maintainer,

on the ganeti master node I found that the luxid dies regularly. This
happens somewhere between 1 to 1.5 days, possibly depending on the
number of commands it deals with. A simple /etc/init.d/ganeti restart
"fixes" it and one can use the cluster again, but obviously that is
not a good solution.

The last thing in log when this happens is
------------------------------------------------------------------------
2015-05-07 08:50:05,904644000000 CEST: ganeti-luxid pid=175797 INFO Rereading job 51157
2015-05-07 08:50:05,905380000000 CEST: ganeti-luxid pid=175797 INFO Finished jobs: (51157,JOB_STATUS_SUCCESS)
2015-05-07 08:54:21,256650000000 CEST: ganeti-luxid pid=175797 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-07 08:54:21,257120000000 CEST: ganeti-luxid pid=175797 WARNING Rescheduling jobs:
2015-05-07 08:54:21,257706000000 CEST: ganeti-luxid pid=175797 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-07 08:54:21,257828000000 CEST: ganeti-luxid pid=175797 WARNING Rescheduling jobs:
2015-05-07 08:54:21,258016000000 CEST: ganeti-luxid pid=175797 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-07 08:54:21,258163000000 CEST: ganeti-luxid pid=175797 WARNING Rescheduling jobs:
2015-05-07 08:54:21,258431000000 CEST: ganeti-luxid pid=175797 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-07 08:54:21,258550000000 CEST: ganeti-luxid pid=175797 WARNING Rescheduling jobs:
2015-05-07 08:54:21,258755000000 CEST: ganeti-luxid pid=175797 INFO Waiting jobs: []; running jobs: [36853,36858,36883,36884,36886,36917,37010,44515,44518,44528,44529,44532,44533,44534,44535,
44540,44554]
2015-05-07 08:55:03,053666000000 CEST: ganeti-luxid pid=175797 INFO Successfully handled QueryGroups
2015-05-07 08:55:03,400245000000 CEST: ganeti-luxid pid=175797 INFO Successfully handled Query
2015-05-07 08:55:04,784546000000 CEST: ganeti-luxid pid=175797 INFO Successfully handled Query
2015-05-07 08:55:05,230757000000 CEST: ganeti-luxid pid=175797 INFO Successfully handled Query
2015-05-07 08:55:05,574790000000 CEST: ganeti-luxid pid=175797 INFO New jobs enqueued: 51158
2015-05-07 08:55:05,575083000000 CEST: ganeti-luxid pid=175797 INFO Starting jobs: 51158
2015-05-07 08:55:05,575246000000 CEST: ganeti-luxid pid=175797 INFO Successfully handled SubmitJob
ganeti-luxid: file descriptor 1025 out of range for select (0--1024).
Recompile with -threaded to work around this.
------------------------------------------------------------------------

Note that the part with the failed started jobs due to "Resource
temporarily unavailable" happens at other times too, at which luxid
does NOT die, so while that is another thing to look at (wtf, there
are enough resources), I dont think it is the cause. Especially as the
failure happens without that too:

------------------------------------------------------------------------
2015-05-13 08:40:05,864418000000 CEST: ganeti-luxid pid=844184 INFO Finished jobs: (51941,JOB_STATUS_SUCCESS)
2015-05-13 08:42:47,965607000000 CEST: ganeti-luxid pid=844184 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-13 08:42:47,965896000000 CEST: ganeti-luxid pid=844184 WARNING Rescheduling jobs:
2015-05-13 08:42:47,966336000000 CEST: ganeti-luxid pid=844184 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-13 08:42:47,966457000000 CEST: ganeti-luxid pid=844184 WARNING Rescheduling jobs:
2015-05-13 08:42:47,966727000000 CEST: ganeti-luxid pid=844184 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-13 08:42:47,966844000000 CEST: ganeti-luxid pid=844184 WARNING Rescheduling jobs:
2015-05-13 08:42:47,967252000000 CEST: ganeti-luxid pid=844184 INFO Waiting jobs: []; running jobs: [36853,36858,36883,36884,36886,36917,37010,44515,44518,44528,44529,44532,44533,44534,44535,
44540,44554]
2015-05-13 08:42:47,967498000000 CEST: ganeti-luxid pid=844184 WARNING Starting jobs failed: connect: resource exhausted (Resource temporarily unavailable)
2015-05-13 08:42:47,967651000000 CEST: ganeti-luxid pid=844184 WARNING Rescheduling jobs:
2015-05-13 08:45:02,422865000000 CEST: ganeti-luxid pid=844184 INFO Successfully handled QueryGroups
ganeti-luxid: file descriptor 1025 out of range for select (0--1024).
Recompile with -threaded to work around this.
------------------------------------------------------------------------
#785788#10
Date:
2015-05-20 15:15:46 UTC
From:
To:
Hi,

It looks like ganeti-luxid is leaking file descriptors. Can you run a
couple of jobs (e.g. gnt-cluster verify) and attach the output of lsof -n -p
$(pidof ganeti-luxid) as it changes before and after each job is run?

Thanks,
Apollon
#785788#17
Date:
2015-05-21 12:15:06 UTC
From:
To:
Am 2015-05-20 17:15, schrieb Apollon Oikonomopoulos:

Lets see.

[root@bl11c02::~]# lsof -n -p $(pidof ganeti-luxid)

                                        [8/12]
COMMAND      PID        USER   FD   TYPE             DEVICE SIZE/OFF
  NODE NAME
ganeti-lu 771867 gnt-masterd  cwd    DIR              253,0     4096
     2 /
ganeti-lu 771867 gnt-masterd  rtd    DIR              253,0     4096
     2 /
ganeti-lu 771867 gnt-masterd  txt    REG              253,3 14752984
261890 /usr/lib/ganeti/2.11/usr/sbin/ganeti-luxid
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    22928
13241 /lib/x86_64-linux-gnu/libnss_dns-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    47616
13221 /lib/x86_64-linux-gnu/libnss_files-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3  3456528
262019 /usr/lib/locale/locale-archive
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14112
   854 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    72808
261463 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    68072
260968 /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14320
    79 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    35400
260933 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14672
    65 /lib/x86_64-linux-gnu/libcom_err.so.2.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   162632
260914 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   868096
260929 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   109360
264038 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    80712
13243 /lib/x86_64-linux-gnu/libresolv-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   131107
13245 /lib/x86_64-linux-gnu/libpthread-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   108216
265445 /usr/lib/x86_64-linux-gnu/librtmp.so.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   520224
16367 /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   785552
260899 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   257288
260920 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   323088
260957 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    59640
260890 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   167080
261333 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   211376
261345 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
ganeti-lu 771867 gnt-masterd  mem    REG              253,0  1599504
13230 /lib/x86_64-linux-gnu/libc-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14768
13236 /lib/x86_64-linux-gnu/libdl-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    31744
13244 /lib/x86_64-linux-gnu/librt-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   530736
13076 /lib/x86_64-linux-gnu/libm-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    52192
269315 /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   456248
270483 /usr/lib/x86_64-linux-gnu/libgmp.so.10.0.5
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   410216
261374 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   248968
  1313 /lib/x86_64-linux-gnu/libpcre.so.3.13.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    92752
   442 /lib/x86_64-linux-gnu/libz.so.1.2.7
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   136936
13242 /lib/x86_64-linux-gnu/ld-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    26066
264000 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   151984
261232 /usr/lib/locale/C.UTF-8/LC_CTYPE
ganeti-lu 771867 gnt-masterd    0r   CHR                1,3      0t0
  1028 /dev/null
ganeti-lu 771867 gnt-masterd    1w   REG              253,2  1422584
134791 /var/log/ganeti/luxi-daemon.log
ganeti-lu 771867 gnt-masterd    2w   REG              253,2  1422584
134791 /var/log/ganeti/luxi-daemon.log
ganeti-lu 771867 gnt-masterd    3uW  REG               0,15        8
85284709 /run/ganeti/ganeti-luxid.pid
ganeti-lu 771867 gnt-masterd    4r  0000                0,9        0
  6859 anon_inode
ganeti-lu 771867 gnt-masterd    5u  unix 0xffff881fac6ce040      0t0
105707057 /var/run/ganeti/socket/ganeti-query
ganeti-lu 771867 gnt-masterd    6u  unix 0xffff880f3b798780      0t0
105708616 socket
ganeti-lu 771867 gnt-masterd    8wW  REG              253,2        0
131339 /var/lib/ganeti/queue/lock
[root@bl11c02::~]# gnt-cluster verify
Submitted jobs 52967, 52968
Waiting for job 52967 ...
Thu May 21 14:02:47 2015 * Verifying cluster config
Thu May 21 14:02:47 2015 * Verifying cluster certificate files
Thu May 21 14:02:47 2015 * Verifying hypervisor parameters
Thu May 21 14:02:47 2015 * Verifying all nodes belong to an existing
group
Waiting for job 52968 ...
[...]
Thu May 21 14:09:17 2015 * Other Notes
Thu May 21 14:09:17 2015 * Hooks Results

[root@bl11c02::~]# lsof -n -p $(pidof ganeti-luxid)

                                       [7/568]
COMMAND      PID        USER   FD   TYPE             DEVICE SIZE/OFF
  NODE NAME
ganeti-lu 771867 gnt-masterd  cwd    DIR              253,0     4096
     2 /
ganeti-lu 771867 gnt-masterd  rtd    DIR              253,0     4096
     2 /
ganeti-lu 771867 gnt-masterd  txt    REG              253,3 14752984
261890 /usr/lib/ganeti/2.11/usr/sbin/ganeti-luxid
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    22928
13241 /lib/x86_64-linux-gnu/libnss_dns-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    47616
13221 /lib/x86_64-linux-gnu/libnss_files-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3  3456528
262019 /usr/lib/locale/locale-archive
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14112
   854 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    72808
261463 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    68072
260968 /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14320
    79 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    35400
260933 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14672
    65 /lib/x86_64-linux-gnu/libcom_err.so.2.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   162632
260914 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   868096
260929 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   109360
264038 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    80712
13243 /lib/x86_64-linux-gnu/libresolv-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   131107
13245 /lib/x86_64-linux-gnu/libpthread-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   108216
265445 /usr/lib/x86_64-linux-gnu/librtmp.so.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   520224
16367 /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   785552
260899 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   257288
260920 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   323088
260957 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    59640
260890 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   167080
261333 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   211376
261345 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
ganeti-lu 771867 gnt-masterd  mem    REG              253,0  1599504
13230 /lib/x86_64-linux-gnu/libc-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14768
13236 /lib/x86_64-linux-gnu/libdl-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    31744
13244 /lib/x86_64-linux-gnu/librt-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   530736
13076 /lib/x86_64-linux-gnu/libm-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    52192
269315 /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   456248
270483 /usr/lib/x86_64-linux-gnu/libgmp.so.10.0.5
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   410216
261374 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   248968
  1313 /lib/x86_64-linux-gnu/libpcre.so.3.13.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    92752
   442 /lib/x86_64-linux-gnu/libz.so.1.2.7
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   136936
13242 /lib/x86_64-linux-gnu/ld-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    26066
264000 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   151984
261232 /usr/lib/locale/C.UTF-8/LC_CTYPE
ganeti-lu 771867 gnt-masterd    0r   CHR                1,3      0t0
  1028 /dev/null
ganeti-lu 771867 gnt-masterd    1w   REG              253,2  1441391
134791 /var/log/ganeti/luxi-daemon.log
ganeti-lu 771867 gnt-masterd    2w   REG              253,2  1441391
134791 /var/log/ganeti/luxi-daemon.log
ganeti-lu 771867 gnt-masterd    3uW  REG               0,15        8
85284709 /run/ganeti/ganeti-luxid.pid
ganeti-lu 771867 gnt-masterd    4r  0000                0,9        0
  6859 anon_inode
ganeti-lu 771867 gnt-masterd    5u  unix 0xffff881fac6ce040      0t0
105707057 /var/run/ganeti/socket/ganeti-query
ganeti-lu 771867 gnt-masterd    6u  unix 0xffff88037cde7080      0t0
105743850 socket
ganeti-lu 771867 gnt-masterd    7u  unix 0xffff880fc1d1e480      0t0
105743851 socket
ganeti-lu 771867 gnt-masterd    8wW  REG              253,2        0
131339 /var/lib/ganeti/queue/lock
ganeti-lu 771867 gnt-masterd    9r   CHR                1,9      0t0
  1033 /dev/urandom
ganeti-lu 771867 gnt-masterd   10u  unix 0xffff880fc1d1e800      0t0
105743852 socket
ganeti-lu 771867 gnt-masterd   13u  unix 0xffff881038652c40      0t0
105737175 socket
ganeti-lu 771867 gnt-masterd   14u  unix 0xffff880e7824c8c0      0t0
105743842 socket
ganeti-lu 771867 gnt-masterd   16u  unix 0xffff880e794c17c0      0t0
105737177 socket
ganeti-lu 771867 gnt-masterd   18u  unix 0xffff881038b1d740      0t0
105743845 socket
ganeti-lu 771867 gnt-masterd   19u  unix 0xffff88037b592800      0t0
105737171 socket
ganeti-lu 771867 gnt-masterd   22u  unix 0xffff880fff90e400      0t0
105743848 socket
ganeti-lu 771867 gnt-masterd   23u  unix 0xffff88037cde7780      0t0
105743849 socket


There. In the time it took to run the cluster verify it had some
gnt-instance list too, I can't forbid others to work for this...
#785788#22
Date:
2015-05-21 15:18:09 UTC
From:
To:
Control: tags 785788 - moreinfo
--

Not having done anything special, ie. just usual daily processes, an
lsof now shows:

[root@bl11c02::~]# lsof -n -p $(pidof ganeti-luxid)

                                     [154/780]
COMMAND      PID        USER   FD   TYPE             DEVICE SIZE/OFF
  NODE NAME
ganeti-lu 771867 gnt-masterd  cwd    DIR              253,0     4096
     2 /
ganeti-lu 771867 gnt-masterd  rtd    DIR              253,0     4096
     2 /
ganeti-lu 771867 gnt-masterd  txt    REG              253,3 14752984
261890 /usr/lib/ganeti/2.11/usr/sbin/ganeti-luxid
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    22928
13241 /lib/x86_64-linux-gnu/libnss_dns-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    47616
13221 /lib/x86_64-linux-gnu/libnss_files-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3  3456528
262019 /usr/lib/locale/locale-archive
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14112
   854 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    72808
261463 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    68072
260968 /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14320
    79 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    35400
260933 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14672
    65 /lib/x86_64-linux-gnu/libcom_err.so.2.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   162632
260914 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   868096
260929 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   109360
264038 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    80712
13243 /lib/x86_64-linux-gnu/libresolv-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   131107
13245 /lib/x86_64-linux-gnu/libpthread-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   108216
265445 /usr/lib/x86_64-linux-gnu/librtmp.so.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   520224
16367 /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   785552
260899 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   257288
260920 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   323088
260957 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    59640
260890 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   167080
261333 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   211376
261345 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
ganeti-lu 771867 gnt-masterd  mem    REG              253,0  1599504
13230 /lib/x86_64-linux-gnu/libc-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    14768
13236 /lib/x86_64-linux-gnu/libdl-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    31744
13244 /lib/x86_64-linux-gnu/librt-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   530736
13076 /lib/x86_64-linux-gnu/libm-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    52192
269315 /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   456248
270483 /usr/lib/x86_64-linux-gnu/libgmp.so.10.0.5
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   410216
261374 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   248968
  1313 /lib/x86_64-linux-gnu/libpcre.so.3.13.1
ganeti-lu 771867 gnt-masterd  mem    REG              253,0    92752
   442 /lib/x86_64-linux-gnu/libz.so.1.2.7
ganeti-lu 771867 gnt-masterd  mem    REG              253,0   136936
13242 /lib/x86_64-linux-gnu/ld-2.13.so
ganeti-lu 771867 gnt-masterd  mem    REG              253,3    26066
264000 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
ganeti-lu 771867 gnt-masterd  mem    REG              253,3   151984
261232 /usr/lib/locale/C.UTF-8/LC_CTYPE
ganeti-lu 771867 gnt-masterd    0r   CHR                1,3      0t0
  1028 /dev/null
ganeti-lu 771867 gnt-masterd    1w   REG              253,2  1542255
134791 /var/log/ganeti/luxi-daemon.log
ganeti-lu 771867 gnt-masterd    2w   REG              253,2  1542255
134791 /var/log/ganeti/luxi-daemon.log
ganeti-lu 771867 gnt-masterd    3uW  REG               0,15        8
85284709 /run/ganeti/ganeti-luxid.pid
ganeti-lu 771867 gnt-masterd    4r  0000                0,9        0
  6859 anon_inode
ganeti-lu 771867 gnt-masterd    5u  unix 0xffff881fac6ce040      0t0
105707057 /var/run/ganeti/socket/ganeti-query
ganeti-lu 771867 gnt-masterd    6u  unix 0xffff880e791ce400      0t0
105764432 socket
ganeti-lu 771867 gnt-masterd    7u  unix 0xffff880fc1d1e480      0t0
105743851 socket
ganeti-lu 771867 gnt-masterd    8wW  REG              253,2        0
131339 /var/lib/ganeti/queue/lock
ganeti-lu 771867 gnt-masterd    9r   CHR                1,9      0t0
  1033 /dev/urandom
ganeti-lu 771867 gnt-masterd   10u  unix 0xffff880e974964c0      0t0
105764433 socket
ganeti-lu 771867 gnt-masterd   11u  unix 0xffff880fff90e080      0t0
105743922 socket
ganeti-lu 771867 gnt-masterd   12u  unix 0xffff881fffa00740      0t0
105763576 socket
                                         [103/780]
ganeti-lu 771867 gnt-masterd   13u  unix 0xffff881038652c40      0t0
105737175 socket
ganeti-lu 771867 gnt-masterd   14u  unix 0xffff880e7824c8c0      0t0
105743842 socket
ganeti-lu 771867 gnt-masterd   15u  unix 0xffff880e791ce080      0t0
105743924 socket
ganeti-lu 771867 gnt-masterd   16u  unix 0xffff880e794c17c0      0t0
105737177 socket
ganeti-lu 771867 gnt-masterd   17u  unix 0xffff880e97495b80      0t0
105787417 socket
ganeti-lu 771867 gnt-masterd   18u  unix 0xffff881038b1d740      0t0
105743845 socket
ganeti-lu 771867 gnt-masterd   19u  unix 0xffff88037b592800      0t0
105737171 socket
ganeti-lu 771867 gnt-masterd   20u  unix 0xffff880e97496bc0      0t0
105743926 socket
ganeti-lu 771867 gnt-masterd   21u  unix 0xffff880fef05d040      0t0
105803188 socket
ganeti-lu 771867 gnt-masterd   22u  unix 0xffff880fff90e400      0t0
105743848 socket
ganeti-lu 771867 gnt-masterd   23u  unix 0xffff88044767ab00      0t0
105743918 socket
ganeti-lu 771867 gnt-masterd   24u  unix 0xffff8806ea38cac0      0t0
105744314 socket
ganeti-lu 771867 gnt-masterd   25u  unix 0xffff8806ea38c3c0      0t0
105744315 socket
ganeti-lu 771867 gnt-masterd   26u  unix 0xffff88037fe94500      0t0
105764425 socket
ganeti-lu 771867 gnt-masterd   27u  unix 0xffff880fc1d1e800      0t0
105787419 socket
ganeti-lu 771867 gnt-masterd   28u  unix 0xffff88037b592b80      0t0
105744317 socket
ganeti-lu 771867 gnt-masterd   29u  unix 0xffff880e74553840      0t0
105764428 socket
ganeti-lu 771867 gnt-masterd   30u  unix 0xffff880ecfc87c00      0t0
105798782 socket
ganeti-lu 771867 gnt-masterd   31u  unix 0xffff880ecfc87500      0t0
105744309 socket
ganeti-lu 771867 gnt-masterd   32u  unix 0xffff881ef4db8880      0t0
105763579 socket
ganeti-lu 771867 gnt-masterd   33u  unix 0xffff881e177fe880      0t0
105763766 socket
ganeti-lu 771867 gnt-masterd   34u  unix 0xffff880ff5d633c0      0t0
105787410 socket
ganeti-lu 771867 gnt-masterd   35u  unix 0xffff881ffe6ad080      0t0
105763581 socket
ganeti-lu 771867 gnt-masterd   36u  unix 0xffff880e72aa1500      0t0
105798784 socket
ganeti-lu 771867 gnt-masterd   37u  unix 0xffff881e177fe500      0t0
105763767 socket
ganeti-lu 771867 gnt-masterd   38u  unix 0xffff88044767a780      0t0
105787414 socket
ganeti-lu 771867 gnt-masterd   39u  unix 0xffff880e72aa1c00      0t0
105798786 socket
ganeti-lu 771867 gnt-masterd   40u  unix 0xffff882038928780      0t0
105763584 socket
ganeti-lu 771867 gnt-masterd   41u  unix 0xffff880f48bd9480      0t0
105803298 socket
ganeti-lu 771867 gnt-masterd   42u  unix 0xffff880e7418bac0      0t0
105908230 socket
ganeti-lu 771867 gnt-masterd   43u  unix 0xffff880ffa4b5880      0t0
105798776 socket
ganeti-lu 771867 gnt-masterd   44u  unix 0xffff880e9741a740      0t0
105803191 socket
ganeti-lu 771867 gnt-masterd   45u  unix 0xffff88044767a400      0t0
105803236 socket
ganeti-lu 771867 gnt-masterd   46u  unix 0xffff881038b1d040      0t0
105798779 socket
ganeti-lu 771867 gnt-masterd   47u  unix 0xffff881e186aaac0      0t0
105763757 socket
ganeti-lu 771867 gnt-masterd   48u  unix 0xffff880e7824c540      0t0
105803193 socket
ganeti-lu 771867 gnt-masterd   49u  unix 0xffff880f48bd9b80      0t0
105803237 socket
ganeti-lu 771867 gnt-masterd   50u  unix 0xffff881f6d04f4c0      0t0
105763760 socket
ganeti-lu 771867 gnt-masterd   51u  unix 0xffff880fb5463140      0t0
105803300 socket
ganeti-lu 771867 gnt-masterd   52u  unix 0xffff880e794c1b40      0t0
105803292 socket
ganeti-lu 771867 gnt-masterd   53u  unix 0xffff881f5dae43c0      0t0
105763763 socket
ganeti-lu 771867 gnt-masterd   54u  unix 0xffff880e74553140      0t0
105803196 socket
ganeti-lu 771867 gnt-masterd   55u  unix 0xffff881cbd268bc0      0t0
105815624 socket
ganeti-lu 771867 gnt-masterd   56u  unix 0xffff880e7824c1c0      0t0
105803293 socket
ganeti-lu 771867 gnt-masterd   57u  unix 0xffff880ff5d63740      0t0
105803229 socket
ganeti-lu 771867 gnt-masterd   58u  unix 0xffff880fb54634c0      0t0
105803302 socket
ganeti-lu 771867 gnt-masterd   59u  unix 0xffff88037cde7780      0t0
105803295 socket
ganeti-lu 771867 gnt-masterd   60u  unix 0xffff8810386528c0      0t0
105803232 socket
ganeti-lu 771867 gnt-masterd   61u  unix 0xffff880ff5d63040      0t0
105803296 socket
ganeti-lu 771867 gnt-masterd   62u  unix 0xffff880e9741a040      0t0
105833221 socket
ganeti-lu 771867 gnt-masterd   63u  unix 0xffff881cbd2684c0      0t0
105815626 socket
                                          [52/780]
ganeti-lu 771867 gnt-masterd   64u  unix 0xffff881e1a0847c0      0t0
105850353 socket
ganeti-lu 771867 gnt-masterd   65u  unix 0xffff881e175eac40      0t0
105821482 socket
ganeti-lu 771867 gnt-masterd   66u  unix 0xffff880ffa4b5180      0t0
105833224 socket
ganeti-lu 771867 gnt-masterd   67u  unix 0xffff881ef4db8180      0t0
105815616 socket
ganeti-lu 771867 gnt-masterd   68u  unix 0xffff881e175ea8c0      0t0
105821483 socket
ganeti-lu 771867 gnt-masterd   69u  unix 0xffff881e3b437180      0t0
105815618 socket
ganeti-lu 771867 gnt-masterd   70u  unix 0xffff880ecff51400      0t0
105838446 socket
ganeti-lu 771867 gnt-masterd   71u  unix 0xffff8805ea57c500      0t0
105833226 socket
ganeti-lu 771867 gnt-masterd   72u  unix 0xffff881ef2126040      0t0
105815621 socket
ganeti-lu 771867 gnt-masterd   73u  unix 0xffff880ecfc87180      0t0
105838436 socket
ganeti-lu 771867 gnt-masterd   74u  unix 0xffff8804c0ad88c0      0t0
105833228 socket
ganeti-lu 771867 gnt-masterd   75u  unix 0xffff880fc1d1e100      0t0
105858133 socket
ganeti-lu 771867 gnt-masterd   76u  unix 0xffff880fb8a28880      0t0
105874504 socket
ganeti-lu 771867 gnt-masterd   77u  unix 0xffff881ef4db8500      0t0
105821475 socket
ganeti-lu 771867 gnt-masterd   78u  unix 0xffff8805ea57c180      0t0
105858135 socket
ganeti-lu 771867 gnt-masterd   79u  unix 0xffff880e7418b3c0      0t0
105838439 socket
ganeti-lu 771867 gnt-masterd   80u  unix 0xffff881cbd268140      0t0
105821478 socket
ganeti-lu 771867 gnt-masterd   81u  unix 0xffff8804c0ad81c0      0t0
105870555 socket
ganeti-lu 771867 gnt-masterd   82u  unix 0xffff880fb8a28c00      0t0
105874505 socket
ganeti-lu 771867 gnt-masterd   83u  unix 0xffff880e97496140      0t0
105838442 socket
ganeti-lu 771867 gnt-masterd   84u  unix 0xffff880fb8a28500      0t0
105858138 socket
ganeti-lu 771867 gnt-masterd   85u  unix 0xffff880f48bd9100      0t0
105858128 socket
ganeti-lu 771867 gnt-masterd   86u  unix 0xffff880e7418b040      0t0
105870557 socket
ganeti-lu 771867 gnt-masterd   87u  unix 0xffff88037fe94180      0t0
105871211 socket
ganeti-lu 771867 gnt-masterd   88u  unix 0xffff881e3b437c00      0t0
105850344 socket
ganeti-lu 771867 gnt-masterd   89u  unix 0xffff880fb5463840      0t0
105870559 socket
ganeti-lu 771867 gnt-masterd   90u  unix 0xffff881038652540      0t0
105874507 socket
ganeti-lu 771867 gnt-masterd   91u  unix 0xffff881f09349c40      0t0
105850347 socket
ganeti-lu 771867 gnt-masterd   92u  unix 0xffff88037dc1cb80      0t0
105870551 socket
ganeti-lu 771867 gnt-masterd   93u  unix 0xffff8804c0b3d180      0t0
105870849 socket
ganeti-lu 771867 gnt-masterd   94u  unix 0xffff882038928400      0t0
105850350 socket
ganeti-lu 771867 gnt-masterd   95u  unix 0xffff880f48bd9800      0t0
105870839 socket
ganeti-lu 771867 gnt-masterd   96u  unix 0xffff880ecff51b00      0t0
105874498 socket
ganeti-lu 771867 gnt-masterd   97u  unix 0xffff88037dc1c480      0t0
105870018 socket
ganeti-lu 771867 gnt-masterd   98u  unix 0xffff88037b592480      0t0
105870019 socket
ganeti-lu 771867 gnt-masterd   99u  unix 0xffff88037dc1c800      0t0
105874501 socket
ganeti-lu 771867 gnt-masterd  100u  unix 0xffff8803813a7540      0t0
105874883 socket
ganeti-lu 771867 gnt-masterd  101u  unix 0xffff8805ea57c880      0t0
105870842 socket
ganeti-lu 771867 gnt-masterd  102u  unix 0xffff8803813a71c0      0t0
105874884 socket
ganeti-lu 771867 gnt-masterd  103u  unix 0xffff880e72aa1180      0t0
105870022 socket
ganeti-lu 771867 gnt-masterd  104u  unix 0xffff880e77235840      0t0
105870845 socket
ganeti-lu 771867 gnt-masterd  105u  unix 0xffff88037f9a71c0      0t0
105871220 socket
ganeti-lu 771867 gnt-masterd  106u  unix 0xffff8805ea57cc00      0t0
105870014 socket
ganeti-lu 771867 gnt-masterd  107u  unix 0xffff88065238f400      0t0
105908742 socket
ganeti-lu 771867 gnt-masterd  108u  unix 0xffff88037cde7400      0t0
105871214 socket
ganeti-lu 771867 gnt-masterd  109u  unix 0xffff880e7418b740      0t0
105874874 socket
ganeti-lu 771867 gnt-masterd  110u  unix 0xffff8800b8ceec00      0t0
105908232 socket
ganeti-lu 771867 gnt-masterd  111u  unix 0xffff88037b7917c0      0t0
105917711 socket
ganeti-lu 771867 gnt-masterd  112u  unix 0xffff880e74553bc0      0t0
105874877 socket
ganeti-lu 771867 gnt-masterd  113u  unix 0xffff8800b8cee180      0t0
105908234 socket
ganeti-lu 771867 gnt-masterd  114u  unix 0xffff88037dc1c100      0t0
105874879 socket
                                           [1/780]
ganeti-lu 771867 gnt-masterd  115u  unix 0xffff880e7741ec40      0t0
105871217 socket
ganeti-lu 771867 gnt-masterd  116u  unix 0xffff880fb8a28180      0t0
105860786 socket
ganeti-lu 771867 gnt-masterd  117u  unix 0xffff880fef05dac0      0t0
105908734 socket
ganeti-lu 771867 gnt-masterd  118u  unix 0xffff88037f9a7540      0t0
105908226 socket
ganeti-lu 771867 gnt-masterd  119u  unix 0xffff8804c0ad8540      0t0
105860788 socket
ganeti-lu 771867 gnt-masterd  120u  unix 0xffff88037b7910c0      0t0
105917713 socket
ganeti-lu 771867 gnt-masterd  121u  unix 0xffff880e772354c0      0t0
105860790 socket
ganeti-lu 771867 gnt-masterd  123u  unix 0xffff8800b8cee500      0t0
105860782 socket
ganeti-lu 771867 gnt-masterd  124u  unix 0xffff88038057d3c0      0t0
105908953 socket
ganeti-lu 771867 gnt-masterd  125u  unix 0xffff8806ea38c040      0t0
105908737 socket
ganeti-lu 771867 gnt-masterd  126u  unix 0xffff880fef05d740      0t0
105917703 socket
ganeti-lu 771867 gnt-masterd  127u  unix 0xffff8803dabc8b40      0t0
105893283 socket
ganeti-lu 771867 gnt-masterd  128u  unix 0xffff880e77235bc0      0t0
105917705 socket
ganeti-lu 771867 gnt-masterd  129u  unix 0xffff881e13ba8800      0t0
105956691 socket
ganeti-lu 771867 gnt-masterd  130u  unix 0xffff8804c0b3dc00      0t0
105908739 socket
ganeti-lu 771867 gnt-masterd  131u  unix 0xffff88075b72bb00      0t0
105917708 socket
ganeti-lu 771867 gnt-masterd  132u  unix 0xffff8803dabc80c0      0t0
105893285 socket
ganeti-lu 771867 gnt-masterd  133u  unix 0xffff88037cde7080      0t0
105908955 socket
ganeti-lu 771867 gnt-masterd  134u  unix 0xffff8804d014c480      0t0
105893287 socket
ganeti-lu 771867 gnt-masterd  135u  unix 0xffff88007ee944c0      0t0
105923148 socket
ganeti-lu 771867 gnt-masterd  136u  unix 0xffff88037f9a7c40      0t0
105893279 socket
ganeti-lu 771867 gnt-masterd  137u  unix 0xffff8804d014cb80      0t0
105908957 socket
ganeti-lu 771867 gnt-masterd  138u  unix 0xffff88038057d740      0t0
105923138 socket
ganeti-lu 771867 gnt-masterd  139u  unix 0xffff880091a4e480      0t0
105909208 socket
ganeti-lu 771867 gnt-masterd  140u  unix 0xffff880ffa4b5500      0t0
105908947 socket
ganeti-lu 771867 gnt-masterd  141u  unix 0xffff880036762040      0t0
105923295 socket
ganeti-lu 771867 gnt-masterd  142u  unix 0xffff88037fd2fbc0      0t0
105909218 socket
ganeti-lu 771867 gnt-masterd  143u  unix 0xffff88075b72b080      0t0
105908950 socket
ganeti-lu 771867 gnt-masterd  144u  unix 0xffff88075b72b780      0t0
105923141 socket
ganeti-lu 771867 gnt-masterd  145u  unix 0xffff88007ee94840      0t0
105923297 socket
ganeti-lu 771867 gnt-masterd  146u  unix 0xffff881ef54d2140      0t0
105956692 socket
ganeti-lu 771867 gnt-masterd  147u  unix 0xffff880091a4eb80      0t0
105923144 socket
ganeti-lu 771867 gnt-masterd  148u  unix 0xffff881e16af68c0      0t0
105963754 socket
ganeti-lu 771867 gnt-masterd  149u  unix 0xffff880ecff51780      0t0
105923290 socket
ganeti-lu 771867 gnt-masterd  150u  unix 0xffff8800367623c0      0t0
105950911 socket
ganeti-lu 771867 gnt-masterd  151u  unix 0xffff88037ff65ac0      0t0
105909211 socket
ganeti-lu 771867 gnt-masterd  152u  unix 0xffff881e1a0840c0      0t0
105956695 socket
ganeti-lu 771867 gnt-masterd  153u  unix 0xffff881e16af61c0      0t0
105963756 socket
ganeti-lu 771867 gnt-masterd  154u  unix 0xffff8803805e2440      0t0
105909214 socket
ganeti-lu 771867 gnt-masterd  156u  unix 0xffff881e177fe180      0t0
105956687 socket
ganeti-lu 771867 gnt-masterd  157u  unix 0xffff88075b72b400      0t0
105951062 socket
ganeti-lu 771867 gnt-masterd  158u  unix 0xffff8803813a7c40      0t0
105950914 socket
ganeti-lu 771867 gnt-masterd  159u  unix 0xffff881ffe6ad780      0t0
105963746 socket
ganeti-lu 771867 gnt-masterd  160u  unix 0xffff880ecff51080      0t0
105951063 socket
ganeti-lu 771867 gnt-masterd  162u  unix 0xffff881ffdc5a0c0      0t0
105963749 socket
ganeti-lu 771867 gnt-masterd  163u  unix 0xffff88007ee94bc0      0t0
105950916 socket
ganeti-lu 771867 gnt-masterd  164u  unix 0xffff881e186aa740      0t0
105963751 socket
ganeti-lu 771867 gnt-masterd  167u  unix 0xffff880091a4e100      0t0
105950919 socket
ganeti-lu 771867 gnt-masterd  169u  unix 0xffff8800b8cee880      0t0
105951055 socket
ganeti-lu 771867 gnt-masterd  172u  unix 0xffff8804c0b3d880      0t0
105951058 socket


So yeah, it doesn't like to get rid of its sockets.