Hi, I somewhat recently switched to the unshare backend, using tarballs
generated by mmdebstrap, and it has been great!

One thing that is a bit less nice than schroot, though, is the inability of the
unshare backend to use eatmydata to speed up package builds. Not only speed,
but I also fear that continuous (and unnecessary) disk writes may degrade my
disk's lifespan ever so slightly.

Is there a way possible to use eatmydata with the unshare backend? So far I
tried to start the build with `eatmydata sbuild`, but it doesn't seem to work.
The eatmydata package is installed in the tarball.

Thanks!


- -- System Information:
Debian Release: trixie/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 6.3.0-1-amd64 (SMP w/4 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages sbuild depends on:
ii  adduser         3.134
ii  libsbuild-perl  0.85.2
ii  perl            5.36.0-7

Versions of packages sbuild recommends:
ii  autopkgtest  5.29
ii  debootstrap  1.0.128+nmu2
ii  schroot      1.6.13-3+b2
ii  uidmap       1:4.13+dfsg1-1+b1

Versions of packages sbuild suggests:
pn  deborphan  <none>
ii  e2fsprogs  1.47.0-2
ii  kmod       30+20230519-1
ii  wget       1.21.3-1+b2

- -- no debconf information
-----BEGIN PGP SIGNATURE-----

iIoEARYIADIWIQS6VuNIvZRFHt7JcAdKkgiiRVB3pwUCZLRGkBQcYW5kcmVhQHBh
cHBhY29kYS5pdAAKCRBKkgiiRVB3p3/IAQDy9M1Se2vwiTLyj5ZXtB2D8/SsZbKK
gmPZJhpBBLueXQD/QHitxrjCK+f5Vg3eoZ+f6uz7PJwQHXjLvd+se4Xylwk=
=d+oF
-----END PGP SIGNATURE-----

Hi Andrea,

* Andrea Pappacoda <andrea@pappacoda.it> [2023-07-16 21:35]:

Given enough RAM you can use a tmpfs like this:

$unshare_tmpdir_template = '/dev/shm/tmp.sbuild.XXXXXXXXXX';

According to my tests that was even faster then eatmydata (and adding
eatmydata did not improve it.

Cheers Jochen

Hi Jochen, thanks for the reply!

That's great, thanks! I also noticed that it is in the sbuild Wiki, but
I guess I missed it :)

Still, it'd be nice to be able to use eatmydata in cases where RAM isn't
enough. But thanks!

I know this isn't the best place to ask, but speaking of the wiki I'd
like to add a section about configuring sbuild to run autopkgtest when
using the unshare backend, since I haven't been able to do it myself and
I think could be helpful to others. If you know how to do it, please let
me know!

Bye :)

I second this request. If /dev/shm is not large enough you get weird errors,
and not everyone will want /tmp to be a tmpfs, or to use /tmp at all

I like the idea of unshare but it is a lot slower than schroot, anything that
can speed it up is great.

I dont know perl, but if there's some documentation of how to add
code to enable this as an optional feature i'd be happy to learn enough to help
out

Hi Richard,

Quoting Richard Lewis (2024-11-17 11:24:24)

I'm not familiar with eatmydata but as far as I understand the docs you run
something under eatmydata by using the eatmydata wrapper program. So if your
chroot has eatmydata installed, maybe try adding this to your ~/.sbuildrc:

$build_env_cmnd = ['eatmydata'];

Does that work?

My system is an ARM Cortex A73 with 3.6 GB of RAM. It is *very* slow. I am not
experiencing unshare being slower than schroot. Can you tell me more about your
setup so that we can find out where the difference is? You are using directory
schroots with overlayfs?

Thanks!

cheers, josch

Hi Andrea,

Quoting Andrea Pappacoda (2024-01-12 19:04:51)

this should become automatic with piuparts 1.5.1 (currently in unstable) and
the next sbuild release with this commit by Jochen:

https://salsa.debian.org/debian/sbuild/-/commit/760f0250cf6a76828526e45828fd4a21fe0404c1

Thanks!

cheers, josch

Hi Johannes,

Thanks for the info! Great job :)

Bye.

I assume no square brackets - i get some perl error with that,
although it claimed to do something (the example in sbuild.conf doesnt
have them)
I am running "gbp buildpackage --mode=unshare --no-run-lintian
--no-run-piuparts --no-run-autopkgtest" some times are below.

schroot (2nd run, in case of caching - directory and overlayfs):

Build needed 00:00:16, 5352k disk space
real    0m20.029s
user    0m12.594s
sys     0m6.416s


unshare (2nd run, in case of caching, no use of eatmydata)
Build needed 00:00:40, 5352k disk space
real    0m49.898s
user    0m26.677s
sys     0m19.721s

unshare with eatmydata (2nd run, in case of caching)
Build needed 00:00:35, 5352k disk space
real    0m44.809s
user    0m24.927s
sys     0m16.344s


I suppose a lot of the difference is uncompressing the tarball - i
didnt try plain.tar yet

Hi Richard,

Quoting Richard Lewis (2024-11-17 20:20:40)

Oh you are right, the option is a string, not a list. Does eatmydata work if
you run it like that?

Wow, this is wild. Your directory schroot with overlayfs is about half a minute
faster. Would you mind helping me getting more to the bottom of this?

My system is an ARM Cortex A73 with 3.6 GB of RAM. To give you an idea about
how slow that system is: its CPU Mark score is about half of my 11 year old
Intel Core i5 Thinkpad. It takes 20 seconds before a video on youtube starts
playing.  On that system, a buildd chroot tarball extracts in under 2 seconds.
So unless your own system is massively slower than mine (raspberry pi 1 would
come to mind), the fact that unshare uses a tarball and you are using a
directory schroot, does not explain the difference of 30 seconds.

Do you have an idea?

Thanks!

cheers, josch

i think quite a bit (but not all - running unzstf takes ~5-8 secs, is
unpacking the .tar.zstd ) is extracting the tarball vs being a directory. i
tried with just a .tar and it saved about 5 seconds (about 1-2s unpacking).

it's a vps so limited ram (1GB i think), a swap file and not proper
hardware -- I dont know the details (and i dont think you can buy it this
low-powered any more),

nproc says 1 so that might be part of the issue? (i know nothing about
hardware -- let me know if there are other commands that help).

(a laptop would be faster, but not available everywhere!)

Hi Richard,

thank you for following up.

Quoting Richard Lewis (2024-11-18 09:55:08)

But 5-8 second unpack time do not explain the 30 second difference.

Are there more packages than the variant buildd installed in your schroot
directory?

Can you compare build logs between your schroot build and the unshare build to
figure out where the extra time is spent? Maybe run sbuild --verbose under
annotate-output?

1 GB RAM and only single core is fine. I don't want to make sbuild less usuable
on setups like yours and the limitations of my own system serve as a motivation
for that.

Thanks!

cheers, josch

it seems there are some differences. intentionally. the schroot sems to
have gcc13 installed and usr-is-merged (both seem to be post usr-merge,the
schroot one is not newly genrated but is up-to-datw)


i did a redirect of the output and then a diff. the main new differences
(apart from the untar) are

- unshare installs fakeroot and libfakeroot along with
sbuild-build-depends-main-dummy
- schroot uses eatmydata by setting
LD_LIBRARY_PATH=/usr/lib/libeatmydata a d LD_PRELOAD=libeatmydata.so,
whereas unshare calls "eatmydata dpkg-buildpackage..."  --- i guess this
may be the main reason because the eatmydata speedup is applying to more
things:  if there is a way to set these variables in unshare it might just
work, i will check that next



Maybe run sbuild --verbose under

i didnt do this yet!
the end has a c15 sec difference in Package-Time. 2s in BUILD-TIME and 4s
in Install-time

On Mon, 18 Nov 2024 at 22:30, Richard Lewis <richard.lewis.debian@googlemail.com> wrote:

thank *you* for your help
$ld_library_path = "/usr/lib/libeatmydata";
$build_environment ={
           "LD_PRELOAD" => "/usr/lib/ibeatmydata.so"
     };
$environment_filter=[];

and the log has the same "Build environment" variables as schroot. But
I get complaints

ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be
preloaded (cannot open shared object file): ignored.

(four times, twice when listing the environment and twice before
running dpkg-buildpackage)

I dont know why schroot works - the libeatmydata is in
/usr/lib/x86_64-linux-gnu/ (seems to be reported as #1016422)

using "LD_PRELOAD" => "/usr/lib/x86_64-linux-gnu/libeatmydata.so"
removes the warnings , and it does make a small difference to the
build. But not much.

I also added "export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so
not clear this made any further difference

I also ran with all the above and calling
eatmydata gbp buildpackage --verbose --no-run-piuparts
--no-run-lintian --no-run-autopkgtest --mode=unshare # same warnings
as above

and this one seems the best
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so gbp buildpackage
--verbose --no-run-piuparts --no-run-lintian --no-run-autopkgtest
--mode=unshare

but no real speedup

I will do this next

On Thu, 21 Nov 2024 at 23:03, Richard Lewis <richard.lewis.debian@googlemail.com> wrote:

Unpacking tarball added: +6s
Installing fakeroot: +3s (i should just add this to the tarball)
(to start of "Build" line = +10s total)
to end of "check disk space": another +1s
(To start of dpkg-buildpackage = +12s total)
running dpkg-buildpackage = +11s total (so i suppose there is some
rounding. the build lines were the same)
by the time the .changes is printed: +14s total (even though schroot
had more packages in the Built-Using header as it had gcc-13 and
gcc-14 installed)
By the time the "Post Build" is printed, +15s difference


I suppose i will try and make a larger tmpfs to use.

Hi,

Quoting Richard Lewis (2024-11-22 23:14:48)

could it be that your schroot is faster because the overlayfs you mount in your
directory-based schroot comes from a tmpfs and not from your disk?

Otherwise I really don't understand where these differences could be coming
from...

Thank you for trying to find this out.

cheers, josch

I remounted /dev/shm larger, it helped, a bit

this might be it - or maybe how overlayfs works is equivalent to that
(one of the things i like about unshare is that it's a lot easier to
understand what is going on!)

when i open a schroot session i see it mounts things under /run (which
is a tmpfs) and under /var/lib (which is ext4)

(from findmnt --output TARGET,FSTYPE --notruncate):

─/run
                     tmpfs
│ └─/run/schroot/mount/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c
           overlay
│   ├─/run/schroot/mount/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/proc
    proc
│   ├─/run/schroot/mount/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/sys
     sysfs
│   ├─/run/schroot/mount/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/dev/pts
 devpts
│   ├─/run/schroot/mount/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/dev/shm
 tmpfs
│   └─/run/schroot/mount/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/build
   ext4
└─/var/lib/schroot/union/underlay/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c
ext4

the main overlay thing has the following mount options
rw,relatime,
lowerdir=/var/lib/schroot/union/underlay/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c,
upperdir=/var/lib/schroot/union/overlay/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/upper,
workdir=/var/lib/schroot/union/overlay/unstable-amd64-sbuild-76deacf6-fa8a-423b-a797-f4945e3ae74c/work

/var is ext4

i dont know if relatime could have some advantages?