Dear Maintainer,

  * What led up to the situation?
I have two hosts, running several KVM/qemu guests. The first host runs on Ubuntu 20.10 and shows
under /proc/stat in the 9th column the values for the CPU-usage of the guests.

The other host (=this host) under Debian testing does not show these values, rather the guest's
CPU-usage is included in the system-stat (3rd column).

  * What exactly did you do (or not do) that was effective (or
    ineffective)?
I googled an searched a lot to find out, if there is some kind of configuration regarding this,
or known differences because of different kernel/qemu versions.

  * What was the outcome of this action?
Unfortunately I didn’t find anything regarding this issue.

  * What outcome did you expect instead?
I would expect the guest CPU-usage being reported to the host as 'guest' time -> 9th value in
/proc/stat just as it does on my Ubuntu 20.10 box.

Of course between those two boxes are lots of differences (kernel, qemu, libraries, etc.), but
I did not find any clue on where to start searching, nor any hint that it could be a configuration
issue. I suspect that it might be an issue related to the debian's version of qemu and/or kernel
module.

04.04.2021 18:23, Thomas Scholz wrote:

I see what you're talking about. I've no idea what does ubuntu do to provide this information.

In the stock linux kernel, as far as I can see, there's no counting for this info for x86
architecture, - it is only counted for ia64, powerpc and s390x architectures.

I'm not an expert in this area, maybe you can find out what ubuntu does here and ask debian
kernel guys to include the same in debian. As far as I can see, there's nothing qemu can do
about it as long as kernel itself does not provide a way to update this information.

/mjt

04.04.2021 18:23, Thomas Scholz wrote:

I see what you're talking about. I've no idea what does ubuntu do to provide this information.

In the stock linux kernel, as far as I can see, there's no counting for this info for x86
architecture, - it is only counted for ia64, powerpc and s390x architectures.

I'm not an expert in this area, maybe you can find out what ubuntu does here and ask debian
kernel guys to include the same in debian. As far as I can see, there's nothing qemu can do
about it as long as kernel itself does not provide a way to update this information.

/mjt

Thanks a lot for your explanation!

So, if I understand correctly, you say, it is a kernel-issue, because the standard-kernel does not include these counters for x86 and the Ubuntu-team obviously has done some special „hack“ to provide those counters also for x86.

Although I’m a programmer, I have no idea whatsoever of the linux kernel. Could you please point me to these modules you mentioned for ia64/powerpc/s390x that have the counting and the module for x86 that doesn’t? Then I can try to find out what the Ubuntu guys had done there.

If I manage to get to this point, I will get in contact with the Debian kernel team.

Kind regards,
Tom


I see what you're talking about. I've no idea what does ubuntu do to provide this information.

In the stock linux kernel, as far as I can see, there's no counting for this info for x86
architecture, - it is only counted for ia64, powerpc and s390x architectures.

I'm not an expert in this area, maybe you can find out what ubuntu does here and ask debian
kernel guys to include the same in debian. As far as I can see, there's nothing qemu can do
about it as long as kernel itself does not provide a way to update this information.

/mjt

* Michael Tokarev <mjt@tls.msk.ru> [210405 15:40]:

I think this is not quite true. Even on x86, guest_enter_irqoff gets
called by the kvm kernel code (from svm_vcpu_enter_exit). I think
it should just work?

I guess qemu doesn't need to do anything except use KVM.

If it does not work, probably best to ask the kernel folk if they
have any ideas?

Chris

What would be the correct way to do this? Should I just issue a bug-report with "reportbug kernel“?

Hi,
we had Delta in Ubuntu a long long time ago, but afaik there isn't any
delta-need for years.
To be clear, guest time as well as the guest visibility into steal
time is very useful, but there shouldn't be any lack of it in Debian.

I had a try in (an old) debian 10.1 with 5.7.0-2-amd64
And there I can see the accounting moving just fine

ubuntu@debian:~$ cat /proc/stat
cpu  196 0 259 9243 60 0 1 23 0 0
cpu0 82 0 172 4579 39 0 1 10 0 0
qemu-system-x86_64 -nodefaults -nographic -accel kvm
# wait a few seconds, and then abort it
ubuntu@debian:~$ cat /proc/stat
cpu  213 0 283 10926 73 0 4 32 7 0
cpu0 98 0 193 5397 52 0 3 15 7 0
cpu1 115 0 90 5528 20 0 0 16 0 0

The 7 seconds of boring some rom startup code is the accounting that
you are looking for.

Also as you can see in [1] there isn't any architecture note on this
anymore (and I think there only was for steal time). So it really
should work and does so in my experiment above.
The original change is even in the history git pre 2.6.12 and the
latter modifications are not new wither [2][3]

I was updating and rebooting into 5.10.0-5-amd64 and I can confirm
that it no longer works.
(The same experiment as above)

I've looked at the Ubuntu kernels for groovy [4] and hirsute [5], but
found no related Delta on a quick check. IIRC the last upstream change
[6] in that area was around v5.5 - so there should be no change.

IMHO a kernel bug would be appropriate, @mjt I think you could even
just re-target this one, there is no need to make it a different bug.
@Thomas if instead you want to file a new bug, then look at [7]

[1]: https://www.kernel.org/doc/html/latest/filesystems/proc.html#miscellaneous-kernel-statistics-in-proc-stat
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ce0e7b28
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c574358e
[4]: https://kernel.ubuntu.com/git/ubuntu/ubuntu-groovy.git/
[5]: https://kernel.ubuntu.com/git/ubuntu/ubuntu-hirsute.git/
[6]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5720821b
[7]: https://wiki.debian.org/DebianKernelReportingBugs

This made me wonder:

And for [1] account_guest_time it indeed seems that way.
But while ia64/s390/powerpc have special code on their entry/exit path

$ grep -Hrn account_guest_time *
arch/ia64/kernel/time.c:77: account_guest_time(tsk, cycle_to_nsec(ti->gtime));
arch/powerpc/kernel/time.c:430: account_guest_time(tsk,
cputime_to_nsecs(acct->gtime));
arch/s390/kernel/vtime.c:172: account_guest_time(tsk, cputime_to_nsecs(guest));
include/linux/kernel_stat.h:99:extern void account_guest_time(struct
task_struct *, u64);
kernel/sched/cputime.c:140:void account_guest_time(struct task_struct
*p, u64 cputime)

There also are these calls from generic code:

kernel/sched/cputime.c:190: account_guest_time(p, cputime);
kernel/sched/cputime.c:388: account_guest_time(p, cputime);
kernel/sched/cputime.c:678: account_guest_time(tsk, vtime->gtime);

Which match these functions irqtime_account_process_tick,
account_guest_time, vtime_account_guest

Double checked x86 in Ubuntu 5.8.0-49-generic  (groovy) worked fine.
But indeed 5.11.0-13-generic (hirsute) seemed to not account for these
values anymore.

I haven't chased this further down, but these functions sound pretty generic.
@MJT - have you checked those and they all end up only being used in
arch-code/calls?

And as I said, there doesn't seem to be Ubuntu Delta for it [2][3],
yet there it is working.
IMHO this might be an upstream change between 5.8 and 5.10 introducing
the issue.
I'm afraid this really has to be looked at more in depth why it isn't
accounting anymore.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/sched/cputime.c#n140
[2]: https://kernel.ubuntu.com/git/ubuntu/ubuntu-hirsute.git/log/kernel/sched/cputime.c
[3]: https://kernel.ubuntu.com/git/ubuntu/ubuntu-groovy.git/log/kernel/sched/cputime.c

Hi @Christian,

thank you for confirming this bug. The cat /proc/stat looks in my case very similar:

cat /proc/stat
cpu  2116591 89 8538907 52753423 221163 0 36382 0 0 0
cpu0 265391 6 1076646 6580691 28424 0 3086 0 0 0
cpu1 263929 2 1063789 6585041 28533 0 17611 0 0 0
cpu2 266378 4 1071775 6588846 31525 0 2032 0 0 0
cpu3 269310 36 1078628 6569422 28943 0 10916 0 0 0
cpu4 261284 1 1051034 6618690 23751 0 1583 0 0 0
cpu5 264524 0 1068260 6602311 25064 0 514 0 0 0
cpu6 264347 36 1067612 6601253 26841 0 375 0 0 0
cpu7 261424 0 1061160 6607166 28078 0 262 0 0 0

Although several guest are running:

virsh list
 Id   Name        Status
---------------------------
 1    ro****      laufend
 2    rp*****     laufend
 3    T****       laufend
 4    f******     laufend
 5    b******     laufend
 6    T*****      laufend


Is there anything I can/should do in order to re-target this bug? Sorry, I’m a very green newbie
in this low-level linux environment…

I also tried to compile a new „pristine“ kernel following instructions in [1], but failed with the kernel configuration. Unfortunately I wasn’t able to produce a working kernel for my machine.


[1] https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-kernel-org-package

Tom

Thanks,

/mjt

Thanks,

/mjt

No, MJT already forwarded it to upstream (qemu)
It turns out to be a known, but not yet fixed kernel bug [1]
I've collected all the pointers that I could find atm in an Ubuntu
launchpad bug [2] as that is affected just as much with >=5.9 kernels.

It seems we need to wait until the kernel folks agree on some solution :-/
There is not much for you to do, other than (if you want) chime in on
any of the kernel discussions with like "hey this is important to me,
any news?"

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=209831
[2]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1922846

@Christian, thank you very much for your help on this!

Tom