Dear Maintainer,

I have been attempting to build a loadable module (AKA "loadable built-in") for
ksh and I have been encountering a lot of undefined symbol errors when invoking
the module:


$ builtin -f ./recvmsg.so recvmsg
$ echo hi | recvmsg msgvar fdvar
ksh: symbol lookup error: ./recvmsg.so: undefined symbol: nv_open


After doing some digging I believe this is because symbols from the ksh
executable are not being exported to the loaded modules. (In this case,
nv_open() is one of the routines that modules may use to set shell variables.
It is implemented internally as part of ksh but used by loadable modules.)

Having looked through the code, I believe this to be a platform-specific build
issue. ksh uses this code to load the module:

dllopen(dle->path, (flags | RTLD_GLOBAL | RTLD_PARENT))
(src/lib/libdll/dllplug.c:54)
("dllopen" is defined in src/lib/libdll/dllopen.c: it performs some path
searches and then calls dlopen())

The flag RTLD_PARENT indicates that the loaded module should have access to the
host executable's symbols. However, RTLD_PARENT is not implemented on Linux.
ksh provides a fallback definition for the flag to allow compilation to
succeed:


#define RTLD_PARENT 0       // in ./arch/linux.i386-64/include/ast/dlldefs.h


To enable the equivalent functionality on Linux, the host executable must be
linked with the --export-dynamic flag. I have had some success by adding this
to debian/rules:


LDFLAGS += -Wl,--export-dynamic


With this set, I was able to build and invoke one of the example modules
included in the ksh source distribution (src/lib/libast/comp/open.c) with just
minor alteration:

Compiled as-is, the "open" module still failed to find the function tmfmt()
(defined in src/lib/libast/tm/tmform.c) - but, by disabling the calls to
tmfmt() in open.c I was able to get the rest of the module working:


gcc -c -fPIC -g -I ~/src/ksh-93u+20120801/arch/linux.i386-64/include/ast/ -I
~/src/ksh-93u+20120801/src/cmd/ksh93/include/ -I ~/src/ksh-
93u+20120801/arch/linux.i386-64/src/cmd/ksh93/ -c -o open.o open.c
gcc -shared -L ~/src/ksh-93u+20120801/arch/linux.i386-64/lib -last
-Wl,-soname,open.so -o open.so open.o

# Unaltered open.c:
$ builtin -f ./open.so open
$ open -r f Makefile
$ echo $f
/home/tetsujin/src/ksh-93u+20120801/arch/linux.i386-64/src/cmd/ksh93/ksh:
symbol lookup error: ./open.so: undefined symbol: tmfmt

# With call to tmfmt() replaced with "strcpy(buff, "{time}");" :
$ open -r f Makefile
$ echo $f
( atime='{time}' ctime='{time}' dev=2053 fd=4 gid=1000 ino=2744616
mode=u='rw,g=r,o=r' mtime='{time}' name=open.c nlink=1 size=12936 uid=1000 )
# Success!


Basically it appears as though functions from within ksh (nv_open(), etc.) are
being exported to my loadable modules correctly, but functions from within
libast (for instance, tmfmt() or fprintf()) are not. For my purposes that's a
distinct improvement - though it'd be much better to have ast calls working as
well.  (If I figure that one out, I'll let you know.)

In my previous report I suggesting building one of the example modules from the ksh source distribution (open.c) - that is still a worthwhile test
case but I have another that may be simpler to work with:

This example, when run, takes two arguments, a variable name, and a value, and assigns the value to the variable. It's useful as a test case because
it's simple, more so than open.c. It's a bit limited as a test case because it doesn't include any stdio calls - which is currently where I'm stuck in
terms of getting things working.

Interestingly enough, I discovered that some of the build files in the ksh source distribution do appear to conditionally add -Wl,--export-dynamic to
the linker flags, but for whatever reason that doesn't seem to be taking effect in the actual build.

I think I've somewhat misunderstood the issue. --export-dynamic
appears to not truly be necessary, and when building from upstream
sources the build now includes a few sample loadable modules.

When building the upstream source (the "master" branch of
https://github.com/att/ast, which is still identified as 93u+
2012-08-01) using its own build script ("bin/package make"), ksh is
not linked with --export-dynamic (I don't quite understand how its
loadable modules work without ksh being linked --export-dynamic), but
it does build the sample loadable modules that are included with the
distribution (including "open.c" which I used as an example in the bug
report) - and that build of Korn Shell will be able to load and run
those modules:

$ ./bin/package make   # And then wait a while for it to build...
$ ./arch/linux.i386-64/bin/ksh --version    # Copy I built from Git
  version         sh (AT&T Research) 93u+ 2012-08-01
$ ./arch/linux.i386-64/bin/ksh
$ builtin -f arch/linux.i386-64/lib/ksh/libopen.so open
$ open   # Runs the builtin from libopen.so
Usage: open [-abcirwx] [-m mode] var file

But loading this module from the version from Debian
(93u+20120801-2+b1) fails:

$ ksh --version   # Installed copy
  version         sh (AT&T Research) 93u+ 2012-08-01
$ ksh
$ builtin -f arch/linux.i386-64/lib/ksh/libopen.so open    #
Loading the libopen.so built from git
builtin: arch/linux.i386-64/lib/ksh/libopen.so: libshell.so: cannot
open shared object file: No such file or directory

If I download the Debian ksh source package and build it, and try
that, I get a similar failure. I'm not entirely sure why it works on
the one from upstream (with no --export-dynamic) or why it doesn't
work on the packaged one (which describes itself as the same version
from the package...)

So it's hard for me to gauge exactly what's going on here:
- The installed copy of "ksh" is stripped, and stripping "ksh"
apparently makes it incompatible with modules built from a
non-stripped build. (I don't know if stripping ksh makes it
incompatible with modules entirely...)
- I also rebuilt ksh in the Debian package source tree without using
the Debian build rules, yielding an unstripped 93u+ with Debian
patches. This was also incompatible with the loadable module built
from git.
- I attempted to manually build libopen.so in the Debian build tree,
using compile and link commands from another successful build, and
that also didn't work, so I can't determine if the feature really was
broken on that version, and has since been fixed, or if it's just a
build issue.

The upshot is that it's pretty easy to get a working demonstration of
this feature from the upstream copy in git now. I'll see if I can
figure out why it's still not working with the packaged version.

This report and #864632 relate to the same underlying request:
support for ksh's loadable builtins which needs the AST shared
libraries, development headers and a dynamically linked ksh.
Currently, Debian has only shipped a static ksh to date.

I'm downgrading this to a wishlist and merging the two requests. Not
committing to it right now, but keep an eye on this ticket for
updates.

Anuradha