Hi,

I was looking into why one of the loong64 host of ci.d.n was showing a
spike is swap (we have 1.8 TB, it was used all) and loss of
debci-workers and the culprit seems to be src:libipc-shareable-perl. I ran
"""
/usr/bin/autopkgtest --no-built-binaries  --timeout=30600
--needs-internet=skip --timeout-install=9999 --user debci --apt-upgrade 
libipc-shareable-perl -- lxc --sudo --name elbrus
autopkgtest-unstable-loong64
"""
manually on the host and killed it once top reported 70% memory use by
perl (of the 127 GB). Swap plummeted from 74 GB to 10 GB.

I guess this means there's a memory leak somewhere on loong64.
Otherwise, can you please limit the test to a reasonable memory footprint?

Paul

https://ci.debian.net/munin/ci-worker-loong64-01/ci-worker-loong64-01/memory.html

Hrm, annoying.

On amd64 I behaves somewhat sanely, although it also uses lots of
shared memory.

I have a suspicion which test causes this problem, but that's more
guessing.

Did you have a chance to see in top, which process/test was the
culprit?

Also there's a new upstream release, but that's more of a rewrite
and I don't dare to guess if it changes the problem. Maybe we should
just try, but I don't want to kill the poor loong64 CI hosts …


Cheers,
gregor

Hi,
No, but I ran it again to capture it. The output halts here:
t/64-nested_segs_untidy.t ..
ok 1 - Initial array seg count ok
ok 2 - After initial aref add, seg count ok
ok 3 - Adding a new aref to an existing element doesn't create a new seg ok
ok 4 - Same with repurposing the aref again
ok 5 - Same with repurposing the aref again with nested
ok 6 - Nested arrays compare ok
ok 7 - Initial href seg count ok
ok 8 - After initial href add, seg count ok
ok 9 - Adding a new href to an existing key doesn't create a new seg ok
ok 10 - Same with repurposing the href again
ok 11 - Adding a new hash inside of existing does bump seg count
ok 12 - Adding a new hash inside of two level existing does bump seg count
ok 13 - Adding a new hash inside of two level existing twice does bump
seg count
ok 14 - Shared memory hash matches test data ok
1..14
ok

root@elbrus:/# ps faux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         616  0.0  0.0   7776  4304 pts/6    Ss   10:05   0:00 /bin/bash
root         623  0.0  0.0   9552  4000 pts/6    R+   10:08   0:00  \_
ps faux
root         408  0.0  0.0   7488  3744 pts/5    Ss+  10:04   0:00 bash
-c set -a; [ -r /etc/environment ] && . /etc/environment 2>/dev/nul
root         411  0.0  0.0  12288  4832 pts/5    S+   10:04   0:00  \_
su -s /bin/bash root -c set -e; exec /tmp/autopkgtest-lxc.3ruobwtl/d
root         415  0.0  0.0   2688  1744 pts/5    S+   10:04   0:00
\_ /bin/sh /tmp/autopkgtest-lxc.3ruobwtl/downtmp/wrapper.sh --artif
root         427  0.0  0.0   6160  1888 pts/5    S+   10:04   0:00
    \_ tee -a -- /tmp/autopkgtest-lxc.3ruobwtl/downtmp/autodep8-per
root         429  0.0  0.0   6160  1872 pts/5    S+   10:04   0:00
    \_ tee -a -- /tmp/autopkgtest-lxc.3ruobwtl/downtmp/autodep8-per
root         431  0.0  0.0   2688  1728 pts/5    S+   10:04   0:00
    \_ /bin/sh /usr/share/pkg-perl-autopkgtest/runner build-deps
root         438  0.0  0.0   2688  1728 pts/5    S+   10:04   0:00
        \_ /bin/sh /usr/share/pkg-perl-autopkgtest/build-deps.d/smo
root         463  0.1  0.0  22240 11440 pts/5    S+   10:04   0:00
            \_ /usr/bin/perl /usr/bin/prove -I/tmp/autopkgtest-lxc.
root         601 32.8 53.2 1953147760 70989856 pts/5 D+ 10:05   1:12
                  \_ /usr/bin/perl t/65-seg_size.t
root           1  0.1  0.0  18896 12992 ?        Ss   10:04   0:00
/sbin/init
root          43  0.0  0.0  28624 11536 ?        Ss   10:04   0:00  \_
/usr/lib/systemd/systemd-journald
message+      75  0.0  0.0   8784  4160 ?        Ss   10:04   0:00  \_
/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfil
root          76  0.0  0.0  11456  7184 ?        Ss   10:04   0:00  \_
/usr/lib/systemd/systemd-logind
_dhcpcd       90  0.0  0.0  10000  4304 ?        Ss   10:04   0:00  \_
dhcpcd: eth0 [ip4] [ip6]
root          91  0.0  0.0   9824  2352 ?        S    10:04   0:00  |
\_ dhcpcd: [privileged proxy] eth0 [ip4] [ip6]
_dhcpcd      100  0.0  0.0   9824  2208 ?        S    10:04   0:00  |
|   \_ dhcpcd: [DHCP6 proxy] fe80::1266:6aff:fe2f:81c4
_dhcpcd      117  0.0  0.0   9824  2208 ?        S    10:04   0:00  |
|   \_ dhcpcd: [DHCP6 proxy] fc42:5009:ba4b:5ab0:f29:d8d9:95f0:e39f
_dhcpcd      168  0.0  0.0   9824  2144 ?        S    10:04   0:00  |
|   \_ dhcpcd: [BPF ARP] eth0 10.0.193.234
_dhcpcd      500  0.0  0.0   9824  2080 ?        S    10:04   0:00  |
|   \_ dhcpcd: [BOOTP proxy] 10.0.193.234
_dhcpcd       92  0.0  0.0   9808  2032 ?        S    10:04   0:00  |
\_ dhcpcd: [network proxy] eth0 [ip4] [ip6]
_dhcpcd       93  0.0  0.0   9808  1968 ?        S    10:04   0:00  |
\_ dhcpcd: [control proxy] eth0 [ip4] [ip6]
root         132  0.0  0.0   7824  2624 pts/0    Ss+  10:04   0:00  \_
/usr/sbin/agetty --noreset --noclear --keep-baud 115200,57600,38400,
root         133  0.0  0.0  11408  7792 ?        Ss   10:04   0:00  \_
sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups


Well, the package is currently on the reject_list on loong64, so don't
worry about that. I can schedule it when you let me know.

Paul

Control: retitle -1 libipc-shareable-perl: t/65-seg_size.t uses insane amount of memory on loong64

Thanks, much appreciated!
…
…

So it's t/65-seg_size.t again.

"Again" because this test already has a patch to be skipped on 32bit
platforms, which was adopted upstream in the new release (at a
slightly different position in the renamed tests but with the same
effect).

That means that I don't expect any improvement by the new upstream
release at first sight.

Now we either need to find out what's different on loong64 (staring
at the output of `perl -V' didn't show anything obvious to me) or we
can skip the test (in general or on loong64).

Oh, good to know, and thanks for the offer.


Cheers,
gregor