#849841 'bpf: Invalid argument' thrown when kernel version

#849841#5
Date:
2017-01-01 01:24:10 UTC
From:
To:
Hi, I found bpfcc-tools don't work on linux-image-4.8.0-2-amd64 and
linux-image-4.8.0-2-rt-amd64, for these kernels are signed kernel, I think
bpfcc-tools don't work on all signed kernels on x86_64 platform. bpfcc-tools
works on linux-image-4.8.0-2-amd64-unsigned,
linux-image-4.8.0-2-rt-amd64-unsigned and linux-image-4.7.0-1-amd64, following
is my detailed test log:

##############linux-image-4.8.0-2-amd64########################
root@bcat:~# python /usr/share/doc/bpfcc-tools/examples/hello_world.py
bpf: Invalid argument

Traceback (most recent call last):
 File "/usr/share/doc/bpfcc-tools/examples/hello_world.py", line 11, in <module>
   BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
 File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 203, in __init__
   self._trace_autoload()
 File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 679, in _trace_autoload
   fn = self.load_func(func_name, BPF.KPROBE)
 File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 243, in load_func
   raise Exception("Failed to load BPF program %s" % func_name)
Exception: Failed to load BPF program kprobe__sys_clone
root@bcat:~# uname -a
Linux bcat 4.8.0-2-amd64 #1 SMP Debian 4.8.11-1 (2016-12-02) x86_64 GNU/Linux
root@bcat:~# dpkg -l |grep linux-image
ic  linux-image-4.7.0-1-amd64                     4.7.8-1                             amd64        Linux 4.7 for 64-bit PCs (signed)
ii  linux-image-4.8.0-1-amd64                     4.8.7-1                             amd64        Linux 4.8 for 64-bit PCs (signed)
ii  linux-image-4.8.0-2-amd64                     4.8.11-1                            amd64        Linux 4.8 for 64-bit PCs (signed)
ii  linux-image-amd64                             4.8+77                              amd64        Linux for 64-bit PCs (meta-package)

############## linux-image-4.8.0-2-rt-amd64-unsigned########################
root@bcat:~# python /usr/share/doc/bpfcc-tools/examples/hello_world.py
           sshd-1225  [001] d...2..    86.931062: : Hello, World!
           sshd-2855  [007] d...2..    86.937056: : Hello, World!
           sshd-1225  [002] d...2..    92.551224: : Hello, World!
           sshd-2869  [004] d...2..    92.557394: : Hello, World!
  systemd-udevd-454   [003] d...2..    92.721318: : Hello, World!
^Croot@bcat:~
root@bcat:~# uname -a
Linux bcat 4.8.0-2-rt-amd64 #1 SMP PREEMPT RT Debian 4.8.15-1 (2016-12-19) x86_64 GNU/Linux
root@bcat:~# dpkg -l |grep linux-image
ic  linux-image-4.7.0-1-amd64                     4.7.8-1                             amd64        Linux 4.7 for 64-bit PCs (signed)
rc  linux-image-4.8.0-1-amd64                     4.8.7-1                             amd64        Linux 4.8 for 64-bit PCs (signed)
ii  linux-image-4.8.0-2-amd64                     4.8.11-1                            amd64        Linux 4.8 for 64-bit PCs (signed)
ii  linux-image-4.8.0-2-rt-amd64-unsigned         4.8.15-1                            amd64        Linux 4.8 for 64-bit PCs, PREEMPT_RT
ii  linux-image-amd64                             4.8+77                              amd64        Linux for 64-bit PCs (meta-package)

##############linux-image-4.8.0-2-amd64-unsigned ########################
root@bcat:~# python /usr/share/doc/bpfcc-tools/examples/hello_world.py
           sshd-1108  [005] d...   218.666546: : Hello, World!
           sshd-2701  [007] d...   218.672187: : Hello, World!
           sshd-1108  [005] d...   223.546367: : Hello, World!
           sshd-2707  [007] d...   223.551765: : Hello, World!
console-kit-dae-2622  [002] d...   223.723321: : Hello, World!
console-kit-dae-2622  [002] d...   223.726497: : Hello, World!
console-kit-dae-2622  [002] d...   223.729037: : Hello, World!
           sshd-2707  [000] d...   223.771321: : Hello, World!
           bash-2712  [003] d...   223.773793: : Hello, World!
           bash-2713  [004] d...   223.774088: : Hello, World!
           bash-2712  [003] d...   223.792414: : Hello, World!
           bash-2715  [004] d...   223.792811: : Hello, World!
^Croot@bcat:~# uname -a
Linux bcat 4.8.0-2-amd64 #1 SMP Debian 4.8.15-1 (2016-12-19) x86_64 GNU/Linux
root@bcat:~# dpkg -l |grep linux-image
ic  linux-image-4.7.0-1-amd64                     4.7.8-1                             amd64        Linux 4.7 for 64-bit PCs (signed)
rc  linux-image-4.8.0-1-amd64                     4.8.7-1                             amd64        Linux 4.8 for 64-bit PCs (signed)
rc  linux-image-4.8.0-2-amd64                     4.8.11-1                            amd64        Linux 4.8 for 64-bit PCs (signed)
ii  linux-image-4.8.0-2-amd64-unsigned            4.8.15-1                            amd64        Linux 4.8 for 64-bit PCs
ii  linux-image-4.8.0-2-rt-amd64-unsigned         4.8.15-1                            amd64        Linux 4.8 for 64-bit PCs, PREEMPT_RT
ii  linux-image-amd64                             4.8+77                              amd64        Linux for 64-bit PCs (meta-package)

############## linux-image-4.8.0-2-rt-amd64  ########################
root@bcat:~# python /usr/share/doc/bpfcc-tools/examples/hello_world.py
bpf: Invalid argument

Traceback (most recent call last):
 File "/usr/share/doc/bpfcc-tools/examples/hello_world.py", line 11, in <module>
   BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
 File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 203, in __init__
   self._trace_autoload()
 File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 679, in _trace_autoload
   fn = self.load_func(func_name, BPF.KPROBE)
 File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 243, in load_func
   raise Exception("Failed to load BPF program %s" % func_name)
Exception: Failed to load BPF program kprobe__sys_clone
root@bcat:~# uname -a
Linux bcat 4.8.0-2-rt-amd64 #1 SMP PREEMPT RT Debian 4.8.11-1 (2016-12-02) x86_64 GNU/Linux
root@bcat:~# dpkg -l |grep linux-image
ic  linux-image-4.7.0-1-amd64                     4.7.8-1                             amd64        Linux 4.7 for 64-bit PCs (signed)
rc  linux-image-4.8.0-1-amd64                     4.8.7-1                             amd64        Linux 4.8 for 64-bit PCs (signed)
rc  linux-image-4.8.0-2-amd64                     4.8.11-1                            amd64        Linux 4.8 for 64-bit PCs (signed)
ii  linux-image-4.8.0-2-amd64-unsigned            4.8.15-1                            amd64        Linux 4.8 for 64-bit PCs
ii  linux-image-4.8.0-2-rt-amd64                  4.8.11-1                            amd64        Linux 4.8 for 64-bit PCs, PREEMPT_RT (signed)
rc  linux-image-4.8.0-2-rt-amd64-unsigned         4.8.15-1                            amd64        Linux 4.8 for 64-bit PCs, PREEMPT_RT
ii  linux-image-amd64                             4.8+77                              amd64        Linux for 64-bit PCs (meta-package)


=============linux-image-4.7.0-1-amd64=============================
guoliangc:~# python /usr/share/doc/bpfcc-tools/examples/hello_world.py
       chromium-5847  [003] d... 49706.966231: : Hello, World!
       chromium-5847  [003] d... 49706.966270: : Hello, World!
       chromium-5847  [003] d... 49706.967351: : Hello, World!
       chromium-5847  [003] d... 49706.967484: : Hello, World!
       chromium-5847  [003] d... 49706.967658: : Hello, World!
       chromium-5847  [003] d... 49706.967726: : Hello, World!
       chromium-5847  [003] d... 49706.968571: : Hello, World!
^Cguoliangc:~# uname -a
Linux guoliangc 4.7.0-1-amd64 #1 SMP Debian 4.7.8-1 (2016-10-19) x86_64 GNU/Linux
guoliangc:~# dpkg -l |grep linux-image
ii  linux-image-4.6.0-1-amd64                     4.6.4-1                              amd64        Linux 4.6 for 64-bit PCs
ii  linux-image-4.7.0-1-amd64                     4.7.8-1                              amd64        Linux 4.7 for 64-bit PCs (signed)
ii  linux-image-4.8.0-2-amd64-unsigned            4.8.15-1                             amd64        Linux 4.8 for 64-bit PCs

Thanks and Regards,
--
Liang Guo

#849841#10
Date:
2017-01-01 08:18:50 UTC
From:
To:
I don't think this severity is justified, when the regular kernels work fine
with it.
This seems to be the problem. I am not sure what is causing it on a signed
linux package.

Please also see a similar issue upstream:
https://github.com/iovisor/bcc/issues/283

Going by the bug report upstream, it may not necessarily be a signed linux
package issue; but I'm not sure right now.
-- 
Given the large number of mailing lists I follow, I request you to CC
me in replies for quicker response

#849841#15
Date:
2017-01-01 14:19:34 UTC
From:
To:
Control: reassign -1 src:linux 4.8.11-1
Control: severity -1 normal
Control: tag -1 moreinfo
[...]

The signed and unsigned kernels have exactly the same code.  The only
way they can differe in behaviour is on a system with Secure Boot
enabled, where the signed one could be bootable (and then disable
unsigned modules etc.) while the unsigned one does not.

Given that you've been able to boot unsigned kernels, I don't believe
signing has anything to do with this problem.

You're not comparing the same versions of the signed and unsigned
kernels, so perhaps there was a regression between 4.7 and 4.8 that was
corrected between 4.8.11 and 4.8.15.  Unfortunately we're not able to
provide a signed image for 4.8.15-1 as it failed to build on one
architecture.  This should be corrected in the next version.

Please report whether the next update to linux-signed-4.8.0-2-amd64
fixes this.

Ben.

#849841#26
Date:
2017-01-01 18:14:00 UTC
From:
To:
Hi,
I think I find the problem, when the kernel header don't match the kernel image,
bpfcc will throw Invalid argument exception, following is my test log:

root@debsidamd64:~# /usr/share/doc/bpfcc-tools/examples/hello_world.py
bpf: Invalid argument

Traceback (most recent call last):
  File "/usr/share/doc/bpfcc-tools/examples/hello_world.py", line 11,
in <module>
    BPF(text='int kprobe__sys_clone(void *ctx) {
bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 203, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 679,
in _trace_autoload
    fn = self.load_func(func_name, BPF.KPROBE)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 243,
in load_func
    raise Exception("Failed to load BPF program %s" % func_name)
Exception: Failed to load BPF program kprobe__sys_clone

root@debsidamd64:/home/liang# dpkg -i
linux-headers-4.8.0-2-amd64_4.8.11-1_amd64.deb
linux-headers-4.8.0-2-common_4.8.11-1_amd64.deb
dpkg: 警告: 即将把 linux-headers-4.8.0-2-amd64 从 4.8.15-1 降级到 4.8.11-1
(正在读取数据库 ... 系统当前共安装有 192043 个文件和目录。)
正准备解包 linux-headers-4.8.0-2-amd64_4.8.11-1_amd64.deb  ...
正在将 linux-headers-4.8.0-2-amd64 (4.8.11-1) 解包到 (4.8.15-1) 上 ...
dpkg: 警告: 即将把 linux-headers-4.8.0-2-common 从 4.8.15-1 降级到 4.8.11-1
正准备解包 linux-headers-4.8.0-2-common_4.8.11-1_amd64.deb  ...
正在将 linux-headers-4.8.0-2-common (4.8.11-1) 解包到 (4.8.15-1) 上 ...
正在设置 linux-headers-4.8.0-2-common (4.8.11-1) ...
正在设置 linux-headers-4.8.0-2-amd64 (4.8.11-1) ...
root@debsidamd64:/home/liang# /usr/share/doc/bpfcc-tools/examples/hello_world.py
            sshd-762   [000] d...   275.571167: : Hello, World!
            sshd-1750  [000] d...   275.578291: : Hello, World!
 console-kit-dae-1088  [000] d...   278.033478: : Hello, World!
root@debsidamd64:/home/liang#  dpkg -l |grep 4.8.11
ii  linux-headers-4.8.0-2-amd64                   4.8.11-1
                amd64        Header files for Linux 4.8.0-2-amd64
ii  linux-headers-4.8.0-2-common                  4.8.11-1
                amd64        Common header files for Linux 4.8.0-2
ii  linux-image-4.8.0-2-amd64                     4.8.11-1
                amd64        Linux 4.8 for 64-bit PCs (signed)

It looks not a kernel bug, but a kernel team's bug. If signed and
unsigned kernel use the same header files, they should have the exact
same version in Debian archive.

#849841#31
Date:
2017-01-01 18:23:06 UTC
From:
To:
Control: reassign -1 bpfcc-tools
Control: tag -1 - moreinfo

They match closely enough for module building, so I think bpfcc's
version check might not be correct.

[...]

We can't upload them at the same time, so that's not possible in
general.

Ben.

#849841#42
Date:
2017-01-01 18:45:33 UTC
From:
To:
It's OK, I'll report a bug to bpfcc upstream.
#849841#53
Date:
2017-04-07 08:32:40 UTC
From:
To:
Does this bug still apply with 4.9.18 ?
#849841#58
Date:
2019-02-07 00:41:15 UTC
From:
To:
I am not sure what is going wrong .... Can you confirm this is solved ?

uname -a : Linux david 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux

dpkg -l |grep linux-image
rc  linux-image-4.9.0-4-amd64                 4.9.65-3+deb9u1                   amd64        Linux 4.9 for 64-bit PCs
ii  linux-image-4.9.0-6-amd64                 4.9.88-1+deb9u1                   amd64        Linux 4.9 for 64-bit PCs
ii  linux-image-4.9.0-8-amd64                 4.9.110-3+deb9u6                  amd64        Linux 4.9 for 64-bit PCs
ii  linux-image-amd64                         4.9+80+deb9u6                     amd64        Linux for 64-bit PCs (meta-package)

#849841#63
Date:
2019-02-07 10:12:33 UTC
From:
To:
Hi Khaled,

I don't know. From the looks of it, you seem to be running Debian
stable. bpfcc was never part of stable.


Ritesh
-- 
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System

#849841#68
Date:
2022-03-26 10:44:15 UTC
From:
To:
-- 
My Dear Friend,

Did you receive the message i sent to you?

Regards,
Mrs. Malina Rakovic

#849841#73
Date:
2022-03-26 10:44:15 UTC
From:
To:
-- 
My Dear Friend,

Did you receive the message i sent to you?

Regards,
Mrs. Malina Rakovic