#748578 libvirt-bin: segfault with RBD storage pool

#748578#5
Date:
2014-05-18 15:50:14 UTC
From:
To:
With storage pool type='rbd' I experience crash in libvirtd
when I attempt to browse storage from virt-manager
(Add Hardware --> Storage --> [Browse]):

Backtrace:
#0  0x00007f0a6f0a4cb7 in virStorageVolTargetDefFormat.isra.4 () from /usr/lib/libvirt.so.0
#1  0x00007f0a6f0a7e06 in virStorageVolDefFormat () from /usr/lib/libvirt.so.0
#2  0x00007f0a64c390ca in storageVolGetXMLDesc () from /usr/lib/libvirt/connection-driver/libvirt_driver_storage.so
#3  0x00007f0a6f0fae0a in virStorageVolGetXMLDesc () from /usr/lib/libvirt.so.0
#4  0x00007f0a6fb1c4e6 in remoteDispatchStorageVolGetXMLDescHelper ()
#5  0x00007f0a6f155732 in virNetServerProgramDispatch () from /usr/lib/libvirt.so.0
#6  0x00007f0a6fb377dd in virNetServerHandleJob ()
#7  0x00007f0a6f046c85 in virThreadPoolWorker () from /usr/lib/libvirt.so.0
#8  0x00007f0a6f04617e in virThreadHelper () from /usr/lib/libvirt.so.0
#9  0x00007f0a6c518062 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f0a6be45a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


## LIBVIRT_DEBUG=1 libvirtd
~~~~
2014-05-18 15:25:05.832+0000: 2555: debug : virEventPollRunOnce:642 : EVENT_POLL_RUN: nhandles=10 timeout=5000
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:89 : Looking up secret by UUID: 161eb53a-a792-436c-8bb7-3f9bd7c143af
2014-05-18 15:25:05.832+0000: 2557: debug : virSecretLookupByUUIDString:15322 : conn=0x7f74241f5dc0, uuidstr=161eb53a-a792-436c-8bb7-3f9bd7c143af
2014-05-18 15:25:05.832+0000: 2557: debug : virSecretLookupByUUID:15283 : conn=0x7f74241f5dc0, uuid=161eb53a-a792-436c-8bb7-3f9bd7c143af
2014-05-18 15:25:05.832+0000: 2557: debug : virObjectRef:296 : OBJECT_REF: obj=0x7f743c91c360
2014-05-18 15:25:05.832+0000: 2557: debug : virAccessManagerCheckSecret:309 : manager=0x7f743c91c360(name=stack) driver=QEMU secret=0x7f74240ddd80 perm=0
2014-05-18 15:25:05.832+0000: 2557: debug : virAccessManagerCheckSecret:309 : manager=0x7f743c91c440(name=none) driver=QEMU secret=0x7f74240ddd80 perm=0
2014-05-18 15:25:05.832+0000: 2557: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f743c91c360
2014-05-18 15:25:05.832+0000: 2557: debug : virObjectNew:202 : OBJECT_NEW: obj=0x7f74240390e0 classname=virSecret
2014-05-18 15:25:05.832+0000: 2557: debug : virObjectRef:296 : OBJECT_REF: obj=0x7f74241f5dc0
2014-05-18 15:25:05.832+0000: 2557: debug : virObjectRef:296 : OBJECT_REF: obj=0x7f743c91c360
2014-05-18 15:25:05.832+0000: 2557: debug : virAccessManagerCheckSecret:309 : manager=0x7f743c91c360(name=stack) driver=QEMU secret=0x7f74240ddd80 perm=3
2014-05-18 15:25:05.832+0000: 2557: debug : virAccessManagerCheckSecret:309 : manager=0x7f743c91c440(name=none) driver=QEMU secret=0x7f74240ddd80 perm=3
2014-05-18 15:25:05.832+0000: 2557: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f743c91c360
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:141 : Found cephx key: AQCzwM9SMB/notarealkeypFhXKhPMvv7bTHw==
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:173 : Found 3 RADOS cluster monitors in the pool configuration
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:197 : RADOS mon_host has been set to: debmain:6789,deblab:6789,debstor:6789,
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:210 : Setting RADOS option client_mount_timeout to 30
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:213 : Setting RADOS option rados_mon_op_timeout to 30
2014-05-18 15:25:05.832+0000: 2557: debug : virStorageBackendRBDOpenRADOSConn:216 : Setting RADOS option rados_osd_op_timeout to 30
2014-05-18 15:25:05.837+0000: 2557: debug : virSecretFree:15752 : secret=0x7f74240390e0
2014-05-18 15:25:05.837+0000: 2557: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f74240390e0
2014-05-18 15:25:05.837+0000: 2557: debug : virObjectUnref:261 : OBJECT_DISPOSE: obj=0x7f74240390e0
2014-05-18 15:25:05.837+0000: 2557: debug : virSecretDispose:655 : release secret 0x7f74240390e0 161eb53a-a792-436c-8bb7-3f9bd7c143af
2014-05-18 15:25:05.837+0000: 2557: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f74241f5dc0
2014-05-18 15:25:05.850+0000: 2557: debug : volStorageBackendRBDRefreshVolInfo:301 : Refreshed RBD image rbd/tempceph (size: 1165084327936 obj_size: 4194304 num_objs: 277778)
2014-05-18 15:25:05.854+0000: 2557: debug : virStorageBackendRBDCloseRADOSConn:256 : Closing RADOS IoCTX
2014-05-18 15:25:05.854+0000: 2557: debug : virStorageBackendRBDCloseRADOSConn:263 : Closing RADOS connection
2014-05-18 15:25:05.855+0000: 2557: debug : virStorageBackendRBDCloseRADOSConn:270 : RADOS connection existed for 0 seconds
Segmentation fault
~~~~

#748578#10
Date:
2014-05-19 04:30:13 UTC
From:
To:
Could you please reproduce this with the libvirt0-dbg package
installed? This will give us line numbers and variable values.
Cheers,
 -- Guido

#748578#15
Date:
2014-05-19 09:43:25 UTC
From:
To:
Dear Guido,

Actually I have libvirt0-dbg installed. I had to upgrade "gdb"
to get the following dump:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f0621de5700 (LWP 10679)]
0x00007f062ae01cb7 in virStorageVolTargetDefFormat (buf=buf@entry=0x7f0621de4a70, def=def@entry=0x7f05fc022a70, type=type@entry=0x7f062af106e7 "target",
    options=0x7f062b1b7d70 <poolTypeInfo+464>, options=0x7f062b1b7d70 <poolTypeInfo+464>) at ../../../src/conf/storage_conf.c:1539
1539    ../../../src/conf/storage_conf.c: No such file or directory.
(gdb) bt
#0  0x00007f062ae01cb7 in virStorageVolTargetDefFormat (buf=buf@entry=0x7f0621de4a70, def=def@entry=0x7f05fc022a70, type=type@entry=0x7f062af106e7 "target",
    options=0x7f062b1b7d70 <poolTypeInfo+464>, options=0x7f062b1b7d70 <poolTypeInfo+464>) at ../../../src/conf/storage_conf.c:1539
#1  0x00007f062ae04e06 in virStorageVolDefFormat (pool=<optimized out>, def=def@entry=0x7f05fc022a40) at ../../../src/conf/storage_conf.c:1651
#2  0x00007f06209960ca in storageVolGetXMLDesc (obj=0x7f0604000bb0, flags=<optimized out>) at ../../../src/storage/storage_driver.c:2613
#3  0x00007f062ae57e0a in virStorageVolGetXMLDesc (vol=vol@entry=0x7f0604000bb0, flags=0) at ../../../src/libvirt.c:14159
#4  0x00007f062b8794e6 in remoteDispatchStorageVolGetXMLDesc (server=<optimized out>, msg=<optimized out>, ret=0x7f06040008c0, args=0x7f0604000960, rerr=0x7f0621de4cc0,
    client=<optimized out>) at ../../../daemon/remote_dispatch.h:13732
#5  remoteDispatchStorageVolGetXMLDescHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f0621de4cc0, args=0x7f0604000960, ret=0x7f06040008c0)
    at ../../../daemon/remote_dispatch.h:13708
#6  0x00007f062aeb2732 in virNetServerProgramDispatchCall (msg=0x7f062bf2c3b0, client=0x7f062bf47c00, server=0x7f062bf2b390, prog=0x7f062bf43a20)
    at ../../../src/rpc/virnetserverprogram.c:437
#7  virNetServerProgramDispatch (prog=0x7f062bf43a20, server=server@entry=0x7f062bf2b390, client=0x7f062bf47c00, msg=0x7f062bf2c3b0) at ../../../src/rpc/virnetserverprogram.c:307
#8  0x00007f062b8947dd in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f062bf2b390) at ../../../src/rpc/virnetserver.c:172
#9  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f062bf2b390) at ../../../src/rpc/virnetserver.c:193
#10 0x00007f062ada3c85 in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf1ffb0) at ../../../src/util/virthreadpool.c:145
#11 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197
#12 0x00007f0628275062 in start_thread (arg=0x7f0621de5700) at pthread_create.c:312
#13 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
--------- (gdb) thread apply all bt Thread 11 (Thread 0x7f0625ded700 (LWP 10670)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b510, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d1b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf201d0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f0625ded700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7f06255ec700 (LWP 10671)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b510, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d1b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf200b0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f06255ec700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7f0624deb700 (LWP 10672)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b510, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d1b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf1ffb0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f0624deb700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7f06245ea700 (LWP 10673)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b510, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d1b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf200b0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f06245ea700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7f0623de9700 (LWP 10675)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b510, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d1b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf201d0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f0623de9700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7f06235e8700 (LWP 10676)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b5a8, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d3b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf200b0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f06235e8700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7f0622de7700 (LWP 10677)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b5a8, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d3b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf1ffb0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f0622de7700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7f06225e6700 (LWP 10678)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b5a8, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d3b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf200b0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f06225e6700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7f0621de5700 (LWP 10679)): #0 0x00007f062ae01cb7 in virStorageVolTargetDefFormat (buf=buf@entry=0x7f0621de4a70, def=def@entry=0x7f05fc022a70, type=type@entry=0x7f062af106e7 "target", options=0x7f062b1b7d70 <poolTypeInfo+464>, options=0x7f062b1b7d70 <poolTypeInfo+464>) at ../../../src/conf/storage_conf.c:1539 #1 0x00007f062ae04e06 in virStorageVolDefFormat (pool=<optimized out>, def=def@entry=0x7f05fc022a40) at ../../../src/conf/storage_conf.c:1651 #2 0x00007f06209960ca in storageVolGetXMLDesc (obj=0x7f0604000bb0, flags=<optimized out>) at ../../../src/storage/storage_driver.c:2613 #3 0x00007f062ae57e0a in virStorageVolGetXMLDesc (vol=vol@entry=0x7f0604000bb0, flags=0) at ../../../src/libvirt.c:14159 #4 0x00007f062b8794e6 in remoteDispatchStorageVolGetXMLDesc (server=<optimized out>, msg=<optimized out>, ret=0x7f06040008c0, args=0x7f0604000960, rerr=0x7f0621de4cc0, client=<optimized out>) at ../../../daemon/remote_dispatch.h:13732 #5 remoteDispatchStorageVolGetXMLDescHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f0621de4cc0, args=0x7f0604000960, ret=0x7f06040008c0) at ../../../daemon/remote_dispatch.h:13708 #6 0x00007f062aeb2732 in virNetServerProgramDispatchCall (msg=0x7f062bf2c3b0, client=0x7f062bf47c00, server=0x7f062bf2b390, prog=0x7f062bf43a20) at ../../../src/rpc/virnetserverprogram.c:437 #7 virNetServerProgramDispatch (prog=0x7f062bf43a20, server=server@entry=0x7f062bf2b390, client=0x7f062bf47c00, msg=0x7f062bf2c3b0) at ../../../src/rpc/virnetserverprogram.c:307 #8 0x00007f062b8947dd in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f062bf2b390) at ../../../src/rpc/virnetserver.c:172 #9 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f062bf2b390) at ../../../src/rpc/virnetserver.c:193 #10 0x00007f062ada3c85 in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf1ffb0) at ../../../src/util/virthreadpool.c:145 #11 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #12 0x00007f0628275062 in start_thread (arg=0x7f0621de5700) at pthread_create.c:312 #13 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7f06215e4700 (LWP 10680)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f062ada363a in virCondWait (c=c@entry=0x7f062bf2b5a8, m=m@entry=0x7f062bf2b4e8) at ../../../src/util/virthread.c:153 #2 0x00007f062ada3d3b in virThreadPoolWorker (opaque=opaque@entry=0x7f062bf200b0) at ../../../src/util/virthreadpool.c:104 #3 0x00007f062ada317e in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:197 #4 0x00007f0628275062 in start_thread (arg=0x7f06215e4700) at pthread_create.c:312 #5 0x00007f0627ba2a3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7f062b7f6840 (LWP 10669)): #0 0x00007f0627b9772d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f062ad68521 in poll (__timeout=5000, __nfds=10, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 virEventPollRunOnce () at ../../../src/util/vireventpoll.c:643 #3 0x00007f062ad66e52 in virEventRunDefaultImpl () at ../../../src/util/virevent.c:308 #4 0x00007f062b895ded in virNetServerRun (srv=srv@entry=0x7f062bf2b390) at ../../../src/rpc/virnetserver.c:1138 #5 0x00007f062b85eb37 in main (argc=<optimized out>, argv=<optimized out>) at ../../../daemon/libvirtd.c:1507 Thank you.
--- The more false we destroy the more room there will be for the true. -- Robert G. Ingersoll, 1902