#1055646 gdb: extremely slow response to basic commands

Package:
gdb
Source:
gdb
Description:
GNU Debugger
Submitter:
Tomasz Pawlak
Date:
2023-11-28 16:30:03 UTC
Severity:
normal
#1055646#5
Date:
2023-11-09 16:17:32 UTC
From:
To:
The gdb compiled for Debian12 is extremely slow - it takes tens of seconds to
minutes
to execute even the simplest commands, like "info args", or "backtrace" - for a
medium size program.

I've modified Codeblocks IDE to measure gdb response times, here's the link:
https://forums.codeblocks.org/index.php/topic,25561.msg174050.html#msg174050

Apparently the problem is with the build flags used to compile gdb.
I've made a test build of the source package:
"gdb-source_13.1-3_all.deb"
build flags:
libbacktrace --with-zstd --with-system-readline --with-system-zlib --with-
xxhash=yes --disable-unit-tests --disable-sim"

The newly built gdb is working as expected - measured response times are close
to zero (see the link posted above)

Regards

#1055646#10
Date:
2023-11-11 12:05:02 UTC
From:
To:
Today I've tried to do the same trick on some other machine (using build
flags from previous mail), but the build failed with error message:
"ar: `u' modifier ignored since `D' is the default (see `U')"
while compiling "libgdbsupport.a"

So I've just spent few hours trying to figure out why it worked on the
other PC ...
Long story short:
During testing, I've first run ./configure from the gdb/gdb directory
with the following flags:
./configure --enable-threading --with-gnu-ld --enable-libbacktrace
--with-xxhash=yes --with-debuginfod=yes

Then I've run ./configure from top /gdb dir (with the aforementioned
flags) and the compilation went fine.

So now the process is reproducible.


Regards

#1055646#15
Date:
2023-11-13 20:46:37 UTC
From:
To:
Hello,

I've just built gdb "the Debian way", to check what are the differences
in the build process:
$> apt source gdb
$> apt build-dep gdb
$> debuild -b -uc -us

 From debuild output:

gdb-default: configured with:
...
      --without-babeltrace
      --with-babeltrace
...
gdb-minimal: configured with:
...
      --without-babeltrace
      --with-babeltrace
      --without-babeltrace
...

This looks bad, but the results are even more "interresting":
The gdb binary installed in the system (**debsums OK**) reports the
following configuration (the "show configuration" command, differences
only):

...
          --with-babeltrace
          --with-mpfr
          --with-xxhash
          --with-python=/usr (relocatable)
          --with-python-libdir=/usr/lib (relocatable)
          --enable-source-highlight
...

The newly "debuild" gdb reports something *different*:
...
          --without-babeltrace  << this: conflicting build flags
          --without-mpfr
          --with-xxhash
          --without-python
          --without-python-libdir
          --disable-source-highlight
...


The gdb which I've compiled using ./configure script (so *without*
Debian patches) reports:
...
          --with-babeltrace
          --with-mpfr
          --without-xxhash  << this: libxxhash-dev is installed, but
flag is disabled ?
          --with-python=/usr
          --with-python-libdir=/usr/lib
          --disable-source-highlight << this
...

Regards

#1055646#20
Date:
2023-11-13 21:20:28 UTC
From:
To:
Hello Tomazzi,

Latest switch is the one that should be in-use

Could you share your conclusion or a patch? I quite do not understand
what you are trying to say with all those examples.

Regards

#1055646#25
Date:
2023-11-15 22:05:57 UTC
From:
To:
Hello Héctor,

Well, I want to fix this bug, because gdb is a very important tool for
me, and compiling gdb from ./configure script is just a hack - not a
real solution (which would eliminate the problem for good).

To do this, I need to have a complete build log with all the
compiler/linker flags, compiler warnings, etc.

Unfortunately, the methods described on the official Debian Wiki page
does not allow to build gdb from source package -> "debuild" fails to
produce the binary shipped by Debian. (I would say that this deserves a
separate BUG report).

Now, the "hacked" gdb is also different - so the first step is to find
differences in the build process, like the exact list of features
enabled/disabled.

So, can You please explain how did You build the gdb?

Regards.


On Mon, 13 Nov 2023 22:20:28 +0100
=?UTF-8?B?SMOpY3RvciBPcsOzbiBNYXJ0w61uZXo=?= <hector.oron@gmail.com> wrote:

 > Hello Tomazzi,
 >
 > On Mon, 13 Nov 2023 at 21:51, tomazzi <tomazzi@wp.pl> wrote:
 > >
 > > Hello,
 > >
 > > I've just built gdb "the Debian way", to check what are the differences
 > > in the build process:
 > > $> apt source gdb
 > > $> apt build-dep gdb
 > > $> debuild -b -uc -us
 > >
 > > From debuild output:
 > >
 > > gdb-default: configured with:
 > > ...
 > > --without-babeltrace
 > > --with-babeltrace
 > > ...
 > > gdb-minimal: configured with:
 > > ...
 > > --without-babeltrace
 > > --with-babeltrace
 > > --without-babeltrace
 > > ...
 > >
 > > This looks bad, but the results are even more "interresting":
 >
 > Latest switch is the one that should be in-use
 >
 > > The gdb binary installed in the system (**debsums OK**) reports the
 > > following configuration (the "show configuration" command, differences
 > > only):
 > >
 > > ...
 > > --with-babeltrace
 > > --with-mpfr
 > > --with-xxhash
 > > --with-python=/usr (relocatable)
 > > --with-python-libdir=/usr/lib (relocatable)
 > > --enable-source-highlight
 > > ...
 > >
 > > The newly "debuild" gdb reports something *different*:
 > > ...
 > > --without-babeltrace << this: conflicting build flags
 > > --without-mpfr
 > > --with-xxhash
 > > --without-python
 > > --without-python-libdir
 > > --disable-source-highlight
 > > ...
 > >
 > >
 > > The gdb which I've compiled using ./configure script (so *without*
 > > Debian patches) reports:
 > > ...
 > > --with-babeltrace
 > > --with-mpfr
 > > --without-xxhash << this: libxxhash-dev is installed, but

#1055646#30
Date:
2023-11-21 12:53:00 UTC
From:
To:
Hello,

Find logs at https://buildd.debian.org/status/logs.php?pkg=gdb&arch=amd64

I am unsure about that, usually you can build using dpkg-buildpackage.

Build daemons build it, see logs posted previously. Build daemons use sbuild.

#1055646#35
Date:
2023-11-23 10:59:32 UTC
From:
To:
Hello,

On Tue, 21 Nov 2023 13:53:00 +0100
=?UTF-8?B?SMOpY3RvciBPcsOzbiBNYXJ0w61uZXo=?= <hector.oron@gmail.com> wrote:

 > Hello,
 >
 > On Wed, 15 Nov 2023 at 23:09, tomazzi <tomazzi@wp.pl> wrote:
 >
 > > To do this, I need to have a complete build log with all the
 > > compiler/linker flags, compiler warnings, etc.
 >
 > Find logs at https://buildd.debian.org/status/logs.php?pkg=gdb&arch=amd64

Many thanks for the link - I have the log, but I need some time to
analyze it.

 > > Unfortunately, the methods described on the official Debian Wiki page
 > > does not allow to build gdb from source package -> "debuild" fails to
 > > produce the binary shipped by Debian. (I would say that this deserves a
 > > separate BUG report).
 >
 > I am unsure about that, usually you can build using dpkg-buildpackage.

The gdb compiled with:

$> dpkg-buildpackage --build=binary --host-type=x86_64-linux-gnu -us -uc

reports configuration identical with the installed gdb, but the files
are of different size: installed 9.9MiB, compiled 9.3MiB. The newly
build gdb is also very slow.

I've also compiled the "hacked" gdb with Debian patches applied, and
it's fast - so no regression here.

Didn't tried sbuild yet.

Regards

#1055646#40
Date:
2023-11-26 18:39:53 UTC
From:
To:
Hello,

I have found the root cause of this problem:
It doesn't matter if the gdb is compiled with sbuild, debuild or
dpkg-buildpackage. It's slow, because it reads BFDs from a *hidden*
directory "/usr/lib/debug/.build-id/"
The gdb compiled using "configure" script had different prefix, so its
"relocatable" paths were pointing to an empty "debug" directory - that's
why it was "fast".

But, this is not the end of problems:
When the gdb is started, it reads symbols from target program and from
libraries which are loaded by the target program. In my test case
(2.5MiB executable with debug symbols included + wxWidgets library with
dbgsym package installed) the symbol table size is ~8MiB, and the gdb
process is using ~50MiB.
This is sufficient to debug the program, but after issuing the "run"
command, the BFDs from "/usr/lib/debug/.build-id/" are read - and gdb
creates a Monster Symbol Table with size over 200MiB (~280Mib used in
total).
Then things are getting only worse - any gdb command for displaying type
or value of target program' variables causes the the memory consumption
to double, in my case from ~280MiB to ~570MiB and the response times
become deadly slow.

But, this is not the end of problems: (2)
In all test cases the gdb was compiled with --enable-threading - and
indeed, gdb creates (num_CPU +1) threads on startup (in my case it's 17
for 16-core CPU). But the threading support in gdb is an illusion -
those additional threads are *never* used, with a single exception -
they are used only after the "run" command is issued, probably to read
the BFDs. For any other gdb command only the main thread is consuming
CPU time.

I thought that this BUG should be easy to eliminate - but it turns out
that there are several serious BUGs in the gdb, and the only way to
eliminate them is to debug the gdb first ;)

Regards

#1055646#45
Date:
2023-11-28 12:37:40 UTC
From:
To:
Hello tomazzi,

Thanks very much! This is a very valuable investigation.
I wanted to drop a link to addr2line bug[1], could that be related?

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=29785

Which several serious bugs? I understand you mean in the upstream GNU
GDB, instead of the Debian GNU GDB package. Are those reported
upstream?

Regards

#1055646#50
Date:
2023-11-28 16:27:25 UTC
From:
To:
Hello Héctor,

On Tue, 28 Nov 2023 13:37:40 +0100  =?UTF-8?B?SMOpY3RvciBPcsOzbiBNYXJ0w61uZXo=?= <hector.oron@gmail.com> wrote:

 > I wanted to drop a link to addr2line bug[1], could that be related?
 >
 > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=29785

Thanks for this link.
It looks like it can be related, but I can't confirm this at the moment.

  > > I thought that this BUG should be easy to eliminate - but it turns out

 > > that there are several serious BUGs in the gdb, and the only way to
 > > eliminate them is to debug the gdb first ;)
 >
 > Which several serious bugs? I understand you mean in the upstream GNU
 > GDB, instead of the Debian GNU GDB package. Are those reported
 > upstream?

Well, I think that at least 2 problems are obvious:
- huge memory consumption (can be related to the upstream bug or just a
memory leak)
- creation of useless threads: I would expect that "enable-threading"
really enables SMP for searching symbol tables, memory ranges, etc.

I haven't reported this yet, because first I want to fully understand
whats going on.
And yes, I mean upstream GNU GDB bugs.

Regards.