Hi,
It seems that Debian's libtool has a patch which will reduce
dependencies in binaries it produces.
It would be nice to bring this patch upstream, as the upstream
maintainers not running Debian are libtoolizing without this specific
change, and require the Debian maintainer to relibtoolize to reduce
dependencies, which is a lot of work.
Steve Langasek has brought this issue upstream at:
<http://lists.gnu.org/archive/html/libtool/2005-07/threads.html#00174>
Bye,
One of the knock-on effects of this patch is bug 291641 where ld resolves symbols using installed libraries instead of libraries in the build directory. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=291641 Subversion 1.2 has a Build-Conflicts with Subversion 1.1 to avoid this problem.
After reading bug #291641, I'd have to agree this is a problem with the linker. Has anybody took that to upstream binutils yet? Kurt
Kurt Roeckx <kurt@roeckx.be> writes:
It's not quite so clear to me. Here's the script I've been using to
duplicate the problem:
#!/bin/bash
rm -rf zz
mkdir zz zz/i zz/j zz/k zz/lib zz/bin
R="-rpath $PWD/zz/lib"
echo "int this_is_i() { return 0; }" > zz/i/i.c
echo "int this_is_j() { return this_is_i(); }" > zz/j/j.c
echo "int main() { return this_is_j(); }" > zz/k/k.c
libtool --mode=compile gcc -c zz/i/i.c -o zz/i/i.lo
libtool --mode=compile gcc -c zz/j/j.c -o zz/j/j.lo
libtool --mode=compile gcc -c zz/k/k.c -o zz/k/k.lo
libtool --mode=link gcc zz/i/i.lo -o zz/i/libi.la $R
libtool --mode=link gcc zz/j/j.lo zz/i/libi.la -o zz/j/libj.la $R
libtool --mode=link gcc zz/k/k.lo zz/j/libj.la -o zz/k/k
echo "== first build complete"
libtool --mode=install cp zz/i/libi.la $PWD/zz/lib > /dev/null
libtool --mode=install cp zz/j/libj.la $PWD/zz/lib > /dev/null
libtool --mode=install cp zz/k/k $PWD/zz/bin > /dev/null
echo "== install complete"
echo ldd zz/k/k
libtool --mode=execute ldd zz/k/k
echo ldd zz/bin/k
libtool --mode=execute ldd zz/bin/k
echo "== first build checked"
echo "int this_is_xi() { return 0; }" > zz/i/i.c
echo "int this_is_j() { return this_is_xi(); }" > zz/j/j.c
libtool --mode=compile gcc -c zz/i/i.c -o zz/i/i.lo
libtool --mode=compile gcc -c zz/j/j.c -o zz/j/j.lo
libtool --mode=link gcc zz/i/i.lo -o zz/i/libi.la $R
libtool --mode=link gcc zz/j/j.lo zz/i/libi.la -o zz/j/libj.la $R
libtool --mode=link gcc zz/k/k.lo zz/j/libj.la -o zz/k/k
echo "== second build failed"
rm -rf zz/lib/* zz/bin/*
echo "== uninstall complete"
libtool --mode=link gcc zz/k/k.lo zz/j/libj.la -o zz/k/k $R
echo "== third build complete"
echo ldd zz/k/k
libtool --mode=execute ldd zz/k/k
echo "== third build checked"
It's the second build that fails, when this libtool command:
libtool --mode=link gcc zz/k/k.lo zz/j/libj.la -o zz/k/k
invokes this gcc command:
gcc zz/k/.libs/k.o -o zz/k/.libs/k zz/j/.libs/libj.so -Wl,--rpath -Wl,/home/pm/sw/zz/lib
Note the --rpath to the install directory, that goes straight through
to the linker, and it appears that the command line option overrides
RPATH in the library. Is it a linker bug for the command line option
to have priority? I don't know.
Perhaps libtool should not be passing that --rpath during the link?
Perhaps libtool should be passing the build dir instead? (That might
mean that libtool has to relink the executable at install time to get
the correct final rpath.) Does that make it a libtool bug?
-rpath dir
Add a directory to the runtime library search path. This is used
when linking an ELF executable with shared objects. All -rpath
arguments are concatenated and passed to the runtime linker, which
uses them to locate shared objects at runtime. The -rpath option
is also used when locating shared objects which are needed by
shared objects explicitly included in the link; see the description
of the -rpath-link option.
And -rpath-link has this nice order:
1. Any directories specified by -rpath-link options.
2. Any directories specified by -rpath options. [...]
[...]
5. For a native linker, the contents of the environment variable
"LD_LIBRARY_PATH".
6. For a native ELF linker, the directories in "DT_RUNPATH" or
"DT_RPATH" of a shared library are searched for shared
libraries needed by it. The "DT_RPATH" entries are ignored if
"DT_RUNPATH" entries exist.
7. The default directories, normally /lib and /usr/lib.
8. For a native linker on an ELF system, if the file
/etc/ld.so.conf exists, the list of directories found in that
file.
So it seems that the -rpath option to the install dir might be
the problem. Or maybe an LD_LIBRARY_PATH?
Any idea why libtool passes the install path in the first place?
How should it know where it's going to get installed at this
point?
PS: This might actually be the wrong bug in which we're
discussing this.
Kurt
Kurt Roeckx <kurt@roeckx.be> writes: That's the way libtool works. When a shared library is linked the libtool command must include an rpath to the install directory, the directory path gets stored in the .la file, so when libtool links another library or application the path is available. Or perhaps the bugs should be merged?
Hi there,
Sorry for the long latency.
* Philip Martin wrote on Fri, Oct 07, 2005 at 01:38:48AM CEST:
Hehe, nice script. Take it, do a
sed -e 's/libtool/$libtool/'
on it, and add
: ${libtool=libtool}
somewhere at the top.
(It's also very instructive to look at the output of objdump -p
by the way, in addition to `libtool --mode=execute ldd' which gives a
skewed view through LD_LIBRARY_PATH; so you might want to add that).
Now let's pretend we're not installing to $PWD/zz/lib, but
to /usr/lib. Just for a minute, ok?
Well, to simulate what libtool would do iff the installation path
is in both link editor's and the runtime linker's default search
path (and without actually messing with our installation), we need to
fake libtool knowledge:
sed < libtool > libtool-fake \
-e "s#^\(sys_lib_search_path_spec=.*\)\"\$#\1 $PWD/zz/lib\"#" \
-e "s#^\(sys_lib_dlsearch_path_spec=.*\)\"\$#\1 $PWD/zz/lib\"#"
Now try
libtool=./libtool-fake ./run
(given that your script is named 'run').
It will pass :)
In the following analysis, let 'libdir' be '$PWD/zz/lib'.
- two differences: libj won't be passed the -rpath $libdir,
hence won't have DT_RPATH $libdir. It will still have the DT_RPATH
entry for the uninstalled libi, correctly.
- linking of k (precise: zz/k/.libs/k) won't have *any* -rpath argument.
And it won't be passed '-li' on the command line.
This is precisely the situation in which the GNU ld mechanism
for walking DT_NEEDED + DT_RPATH combinations of indirect dependencies
(libraries pulled in by other libraries) works.
Correct.
It's documented that way.
Well. In above scenario, it would fix the problem.
In general it's harder to avoid. Imagine we have a dependency on
another library, from another package, already installed. If it was
installed in a path which the linker won't find by itself, the -rpath
will be necessary for the final _installed_ executable, so that it can
find its libraries on its own.
What we could do now, to save the day, would be to disallow fast_install.
Relinking upon installation would have to be required then. This would
enable, on Linux, to remove the -rpath to $libdir from the uninstalled.
I believe.
Hmm. Sounds like quite a bit of work. And pretty dangerous, too. We'd
need lots of good tests. One thing to always keep in mind (for upstream
GNU Libtool) is that this change of operation must not change anything
on other systems. A good way to get many people annoyed at you.
Another way of solving the problem in libtool could be to do the grunt
work of traversing all indirect libraries and adding rpaths for them,
too. Ugly. Would likely need '.la' files for all of them so it could
be done portably. OTOH, it may be the only way we could get decent
indirect dependencies support on more than just GNU ld.
See, the way I see it, this whole automatic DT_NEEDED + DT_RPATH walking
by the link editor works best when all installed paths are found by
default, and thus the final program won't need any overrides, which then
make the linker turn off the weaker search methods.
Another thing not to underestimate is library authors that forget to
list all their dependent libraries they use directly. Finding a bug
related to this is bound to be horrific.
Hmm. I'd call it a ld limitation for which libtool does not yet have an
adequate workaround. But that may just be me. :)
Also, it may be helpful to note that there has been quite a bit of
discussion on the libtool list, on and off, about decent support for
indirect dependencies. Not many patches yet, though. :-/
Cheers,
Ralf
* Ralf Wildenhues wrote on Wed, Nov 30, 2005 at 11:07:47PM CET:
Let P be a program, arrows indicate direct linkage.
Make dead sure we find and use the correct (un)installed libraries, for
both the uninstalled and the installed program:
- P --> libA or P --> libA --> libB or P --> libA --> libB
`-> libB
- both uninstalled,
A un-, B installed,
both installed
- A and B (to-be-)installed in same libdir, or different one
- A and B (to-be-)installed in a default-searched dir
(for either the link editor or the runtime linker)
- there exist previously installed versions of one or
both of the libraries
- either link-time or runtime incompatible
- in either the same libdir as the new ones will be,
or other ones which are involved, or the default-
searched dirs (for either the link editor or the
runtime linker)
[ I have no idea how to put the latter in a testsuite run by non-root ]
- maybe combine with (low-priority):
-static, -all-static, or configured --disable-static.
If all these combinations work out fine, then there is still a whole
class of issues yet to be solved: dlopening with library dependencies.
So, now let dotted arrows indicate lt_dlopen()ed modules, M a module,
D another library (libC is on some systems the C++ standard library).
Something like this set of combination, or a bit less:
- P ..> M
P --> libA ..> M
P ..> M --> libB --> libD
`-> libB
P ..> M --> libB --> libD
`-> libB
`-> libD
P --> libA ..> M --> libB -> libD
P --> libA ..> M --> libB
`-> libB
P --> libA ..> M --> libB -> libD
`-> libD
P --> libA ..> M --> libB -> libD
`-> libB
`-> libD
P --> libA ..> M --> libB
`-> libB
P --> libA ..> M --> libB
`-> libB `-> libB
P --> libA ..> M --> libB -> libD
`-> libB
P --> libA ..> M --> libB -> libD
`-> libB
`-> libD
P --> libA ..> M --> libB -> libD
`-> libB `-> libB
`-> libD
- repeat above list of combinations, expanded by M and libD, but do not
try *all* possibilites (around 200M if I'm not mistaken) -- rather
find a good representative subset (maybe a couple hundred).
- It may even be useful to consider the case where installed modules
depend on uninstalled libraries. I don't think this is particularly
elegant, and it's clearly not portable, but disallowing this is likely
to hurt some of the more elaborate packaging schemes (say, browser
with lots of third-party plugins); rather, the lower-level libs (in
this case libB) should be packaged separately from P and libA, so that
M may be built before them but after libB.
- I'm not certain we also need to take into account modules loading
other modules, or multiple modules depending on different lower-level
libraries.
- DESTDIR installs or cross-compilation have not been taken into account
*just* yet.
One reason I'm writing this is that it may be useful to have this set of
tests not only in the "link_all_deplibs=no Debian libtool", but also in
upstream GNU Libtool. I also think that an eventual merge upstream will
be much easier, say, if the link_all_deplibs choice were not one done at
configure time based on $host only, but maybe based on a configuration
switch, or even a "libtool --mode=link" time switch (or an environment
variable containing that switch). The best choice here is not yet clear
to me, as issues like #297726 and #291641 should be fully understood and
fixed first.
This would likely require Debian to change their build machinery a bit,
but I think it has ways to centralize this in a single or few places
(and I think it can be done in a way that it won't need a "flag day").
Anybody with too much time on their hands is cordially invited to write
tests for this, preferably for CVS HEAD with help of the Autotest test
suite (stresstest.at is a good example of how to tackle this).
(I may be abusing the Debian BTS as a TODO list; sorry about that.)
Cheers, and thank you for reading this far,
Ralf
Daniel Shahaf wrote: Thanks, that's useful. Excitingly enough, Philip's test case passes for me! That's because I use gold, which uses --no-copy-dt-needed-entries by default. If I uninstall binutils-gold (so ld.bfd gets used instead), the test case fails. Jonathan $ dpkg-query -W binutils libtool binutils 2.21.90.20111025-1 libtool 2.4.2-1