Hi,

When profiling with perf (and even oprofile) showing the call graph
can often be invaluable. Unfortunately for anything that goes through
libc that's not efficiently possible as glibc (on at least amd64)
doesn't build with frame pointers enabled.

It is possible to use dwarf unwinding with halfway modern kernel/perf
combinations to get call graphs even in that case, but the overhead is
about a magnitude higher and the profiles are much larger.

As applications have to be built with -fno-omit-frame-pointers anyway
to provide usable call stack it's usually not a problem if some
library isn't. But as so many things that often are bottlenecks
(syscalls, memcpy, string operations, locking, ...) goes through libc
it'd be quite valuable to have a variant of libc built with frame
pointers enabled.

Thanks for considering,

Andres

I would love to bump this bug; I think it would be wonderful to have an
alternative version of libc with frame pointers.

What would it take for such an alternative to exist (can the Debian
alternatives system work for libc)? If other people want this, I'm
interested in investing some time into helping with it.

Best,
~Alex

Hi,

Yea, I'm hitting this more and more often. Especially with the new eBPF
backed profiling tools like bcc, which, for the forseeable future, only
support frame pointer based unwinding. Also fp based unwinding is a lot
more efficient.

I was thinking of adding a libc6-frame-pointers which would replace (and
conflict with) libc6. But that's just because it was what I could think
of.

Maybe it'd be better to let those be co-installed by using a different
triplet and allow to chose which to use via /etc/ld.so.conf/something?
Not pretty either :(

Same here. I'd primarily like some guidance about what approach is more
likely to be accepted.


Greetings,

Andres Freund

Hi,

I've been running into this myself a lot lately and wonder if anything has happened regarding this since 2014, after all it's been six years.
I'm surprised so few people seem to be taking interest in this considering the amount of tools that rely on frame pointers for performant stack traces, which has further increased with the introduction of eBPF.

Hi,

I understand the need for -fno-omit-frame-pointer, however it has a
performance impact, so we do not want to do that by default. OTOH
providing an alternative libc is something tricky if we do not want it
to do it without breaking systems. Someone has to come with a patch that
is well tested.

Aurelien

* Aurelien Jarno:

Most unwinders should be able to use asynchronous unwind tables, which
only impact disk size (and the size of VM mappings).

Experience with perf shows orders of magnitude of overhead of DWARF unwinding over fp based unwinding. The kernel uses ORC for this reason, which is up to 20-40x faster than DWARF, in addition to the removal of the 5-10% DWARF performance penalty caused by .text size increase. It is however not available for user-space programs. Since it uses ORC, DWARF unwinding is also unavailable in (and unlikely to be ever supported by) the kernel, including the bpf_get_stack* helpers used in eBPF. Since most interaction with the kernel is done through libc, this unfortunately makes it impossible to meaningfully inspect user-space programs in response to events, which is one of the biggest selling points of bpf tracing.

Sorry guys, may be I'm telling well known things but Ubuntu has libc6-prof
package which installs alternative version of libc6 with
no-omit-frame-pointer to /lib/libc6-prof/x86_64-linux-gnu/ . Any who need
to profile his program can use it via LD_LIBRARY_PATH environment variable
while other programs by default use optimized version.

Why reinvent the wheel?

Hi all,

Fedora and Ubuntu [1] have now enabled frame pointers on amd64 by default,
providing a great profiling experience out-of-the-box. I think it may be time
for Debian to reconsider its position here; the performance overhead is very
small and meanwhile this lack of frame pointers is preventing other, much more
significant performance improvements by making it hard or impossible to properly
profile applications.

[1]:
https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default

Kind regards,
Ralf

Would it be worth to consider -fno-omit-frame-pointer together with
-momit-leaf-frame-pointer?

Kind regards
Philipp Kern