Package: linux-image-amd64
Version: 5.19.6

During testing systemtap 4.7 with kernel 5.19, I found it can't resolve
some global variables of nginx.
However systemtap 4.7 can resolve them if test with kernel 5.10.

According to output below, systemtap 4.7-1 is only tested with kernel
version from 2.6.32 to 5.18-rc3

% stap -V

With Regards
Yibin

Hi,

Which variables exactly? Can you share a sample script reproducing the
issue?

Thanks for the bug report!

  Emanuele

Hi Emanuele,

The global variable is ngx_cycle in openresty/nginx, openresty is actually
nginx with a lua extension.
The steps to reproduce this bug is as  follows, the scripts used are
attached:


The output is:


However, not all global var access will fail, if I build luajit from source
code, and get a global var named globalL, it will success.

Thanks for investigating.
Yibin

Emanuele Rocca <ema@debian.org> 于2022年9月15日周四 17:15写道：

Hi -

Please raise this with the openresty folks instead of debian.  A
robust stap script must be prepared for intermittent access errors of
@var() type expressions, via constructs such as try/catch.

- FChE

Hi Frank,

openresty is not going to maintain the systemtap scripts anymore, so I have
to debug it.
I tried the try/catch, but shows similar error:
WARNING: fail with read fault [man error::fault] at 0x56354b2f9380


Can you give some clue on why this script works on kernel 5.10, but not on
5.19?

Thanks
Yibin

Frank Ch. Eigler <fche@elastic.org> 于2022年9月20日周二 02:11写道：

Hello Yibin and Frank,

[...]

I can reproduce the regression. The openresty.stp script provided by
Yibin works fine with Linux 5.10 from Debian Bullseye, while it fails
with the 5.19 kernel from testing/sid. In both cases I've used systemtap
4.7-1 from testing/sid.

Here's the output with the Bullseye kernel:

    $ uname -a
    Linux ariel 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

    $ sudo stap --disable-cache ~/Downloads/openresty.stp -x $(pgrep -n nginx)
    WARNING: Kernel function symbol table missing [man warning::symbols]
    WARNING: nginx pid 3224 target 3224
    WARNING: ngx_cycle 0x55caf7ef1b60
    WARNING: module_index 54
    WARNING: lmcf 0x55caf7f048b8
    WARNING: ngx lua 0x7fcfe451b380

And the failure with with the 5.19 kernel from testing/sid:

	$ uname -a
	Linux ariel 5.19.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.19.6-1 (2022-09-01) x86_64 GNU/Linux

	$ sudo stap --disable-cache ~/Downloads/openresty.stp -x $(pgrep -n nginx)
	WARNING: Kernel function symbol table missing [man warning::symbols]
	warning: the compiler differs from the one used to build the kernel
	The kernel was built by: gcc-11 (Debian 11.3.0-5) 11.3.0
	You are using:           gcc-11 (Debian 11.3.0-6) 11.3.0
	WARNING: nginx pid 5978 target 5978
	ERROR: read fault [man error::fault] at 0x5602ca215380 near operator '@var' at /home/ema/Downloads/openresty.stp:18:17
	WARNING: Number of errors: 1, skipped probes: 0
	WARNING: /usr/bin/staprun exited with status: 1
	Pass 5: run failed.  [man error::pass5]
	Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

I've tried booting the 5.19 kernel with all possible values for
preempt=, namely "none", "voluntary", and "full" to no avail.

The 5.19 kernel has this to say when the script fails:

    [   60.484129] kprobes: kprobe jump-optimization is disabled. All kprobes are based on software breakpoint.
    [   60.548807] stap_3811 (openresty.stp): systemtap: 4.7/0.187, base: ffffffffc1153000, memory: 388data/52text/129ctx/262246net/324alloc kb, probes: 2