- Package:
- debootstrap
- Source:
- debootstrap
- Submitter:
- Phillip Susi
- Date:
- 2026-02-05 08:19:01 UTC
- Severity:
- wishlist
- Tags:
Debootstrap is very, very slow. I watched a server install crawl by for something like an hour at a rate of about 1 package per second. A simple debootstrap chroot takes 10 minutes to setup, with a predownloaded tarball. This slowness is due to dpkg making excessive use of fsync(). Requests to fix dpkg in the past have been rejected and I was told to use eatmydata. After patching debootstrap to add eatmydata to the required list, and activate it during the second stage install, the time to construct the chroot dropped from 10m to 2m. This should also make installing new systems MUCH faster.
This simple two line patch does the trick:--- sid 2013-02-15 11:03:15.384977238 -0500 +++ sid.orig 2013-02-15 10:50:23.381293976 -0500 @@ -16,7 +16,7 @@ esac work_out_debs () { - required="$(get_debs Priority: required) eatmydata" + required="$(get_debs Priority: required)" if doing_variant - || doing_variant fakechroot; then #required="$required $(get_debs Priority: important)" @@ -68,7 +68,7 @@ second_stage_install () { setup_devices - export LD_PRELOAD=/usr/lib/libeatmydata/libeatmydata.so + x_core_install () { smallyes '' | in_target dpkg --force-depends --install $(debfor "$@") }
Le vendredi, 15 février 2013 17.10:56, Phillip Susi a écrit : Reversed patch? OdyX
Woops, yes...
Phillip Susi <psusi@ubuntu.com> (15/02/2013): | base-installer (1.121) unstable; urgency=low | | [ Colin Watson ] | * Merge from Ubuntu: | - Run dpkg with --force-unsafe-io during installation; syncing is | unnecessary in this context and can slow things down quite a bit | (closes: #605384). | […] | | -- Joey Hess <joeyh@debian.org> Wed, 24 Aug 2011 19:11:09 -0400 Mraw, KiBi.
On 02/15/2013 09:09 PM, Cyril Brulebois wrote:
Phillip Susi wrote: I have also been suffering with the much slower dpkg on Sid. Today I performed a test of a default debootstrap chroot and had this benchmark data. I am using a local full mirror so that network performance is not significant in the measurement. This is using the stock Sid version of everything up to date as of today operating on an ext3 filesystem. # time debootstrap wheezy testinstall http://localmirror/ftp.us.debian.org/debian ... real 2m58.577s user 0m49.639s sys 0m8.749s And then using the patched debootstrap script Phillip Susi suggested I get this result: # time debootstrap wheezy testinstall http://localmirror/ftp.us.debian.org/debian ... real 1m17.680s user 0m49.879s sys 0m7.988s Wall clock time is more than 2x faster for me when avoiding the unnecessary fsync() calls. Ideally I would like to see dpkg handle this issue. But since that doesn't seem to be happening it would be nice if the rest of the ecosystem could work around it. Here is the same test again on an ext4 filesystem on the same system and the stock scripts. # time debootstrap wheezy testinstall http://localmirror/ftp.us.debian.org/debian ... real 2m55.402s user 0m52.107s sys 0m7.888s And again with the eatmydata patched script: # time debootstrap wheezy testinstall http://localmirror/ftp.us.debian.org/debian ... real 1m4.819s user 0m48.455s sys 0m6.752s That is 2.7x faster on ext4. Bob
Wow, that's pretty fast. I was testing on a server with a raid5 array and was seeing nearly 10m original time, down to 1-2m patched. And that's with a pre built debootstrap tarball so nothing needed downloaded.
Phillip Susi wrote: 2m58s was the slow time. 1m4.819s was the fast time. :-) This is on a relatively fast machine. But the benefits will be more dramatic on machines with slower I/O bandwidth for disk storage. With --download-only I assume? Or otherwise is fine. Sounds good. Bob
The following patch introduces an --include-early option which allows to introduce eatmydata early enough:--- /usr/sbin/debootstrap 2013-03-22 07:04:12.000000000 +0100 +++ tmp/debootstrap/debootstrap 2013-03-31 14:17:26.425276613 +0200 @@ -40,6 +40,7 @@ EXTRACTOR_OVERRIDE="" UNPACK_TARBALL="" ADDITIONAL="" +ADDITIONAL_EARLY="" EXCLUDE="" VERBOSE="" CERTIFICATE="" @@ -94,6 +95,8 @@ [ --arch=powerpc ]--- debootstrap.8.orig 2013-03-31 21:40:29.694855662 +0200 +++ debootstrap.8 2013-03-31 21:41:19.114966193 +0200 @@ -54,6 +54,12 @@ Comma separated list of packages which will be added to download and extract lists. .IP +.IP "\fB\-\-include-early=alpha,beta\fP" +Comma separated list of packages which will be added to download and extract +lists. These packages are already unpacked during the first stage +which allows, for example, eatmydata to be used early enough in the +process. +.IP .IP "\fB\-\-exclude=alpha,beta\fP" Comma separated list of packages which will be removed from download and extract lists. With this patch, one can use eatmydata debootstrap --include-early=eatmydata ... to get eatmydata. With strace -esync,fsync, I verified that this actually only issues about 20 fsync calls in the final stage of debootstrap. On a rotating hard disk, this reduces the time needed to set up a sid bootstrap from 10 minutes to 4 minutes. On an SSD, the time difference is negligible though. Greetings Marc
The following patch introduces an --include-early option which allows to introduce eatmydata early enough:--- /usr/sbin/debootstrap 2013-03-22 07:04:12.000000000 +0100 +++ tmp/debootstrap/debootstrap 2013-03-31 14:17:26.425276613 +0200 @@ -40,6 +40,7 @@ EXTRACTOR_OVERRIDE="" UNPACK_TARBALL="" ADDITIONAL="" +ADDITIONAL_EARLY="" EXCLUDE="" VERBOSE="" CERTIFICATE="" @@ -94,6 +95,8 @@ [ --arch=powerpc ]--- debootstrap.8.orig 2013-03-31 21:40:29.694855662 +0200 +++ debootstrap.8 2013-03-31 21:41:19.114966193 +0200 @@ -54,6 +54,12 @@ Comma separated list of packages which will be added to download and extract lists. .IP +.IP "\fB\-\-include-early=alpha,beta\fP" +Comma separated list of packages which will be added to download and extract +lists. These packages are already unpacked during the first stage +which allows, for example, eatmydata to be used early enough in the +process. +.IP .IP "\fB\-\-exclude=alpha,beta\fP" Comma separated list of packages which will be removed from download and extract lists. With this patch, one can use eatmydata debootstrap --include-early=eatmydata ... to get eatmydata. With strace -esync,fsync, I verified that this actually only issues about 20 fsync calls in the final stage of debootstrap. On a rotating hard disk, this reduces the time needed to set up a sid bootstrap from 10 minutes to 4 minutes. On an SSD, the time difference is negligible though. Greetings Marc
The following patch introduces an --include-early option which allows to introduce eatmydata early enough:--- /usr/sbin/debootstrap 2013-03-22 07:04:12.000000000 +0100 +++ tmp/debootstrap/debootstrap 2013-03-31 14:17:26.425276613 +0200 @@ -40,6 +40,7 @@ EXTRACTOR_OVERRIDE="" UNPACK_TARBALL="" ADDITIONAL="" +ADDITIONAL_EARLY="" EXCLUDE="" VERBOSE="" CERTIFICATE="" @@ -94,6 +95,8 @@ [ --arch=powerpc ]--- debootstrap.8.orig 2013-03-31 21:40:29.694855662 +0200 +++ debootstrap.8 2013-03-31 21:41:19.114966193 +0200 @@ -54,6 +54,12 @@ Comma separated list of packages which will be added to download and extract lists. .IP +.IP "\fB\-\-include-early=alpha,beta\fP" +Comma separated list of packages which will be added to download and extract +lists. These packages are already unpacked during the first stage +which allows, for example, eatmydata to be used early enough in the +process. +.IP .IP "\fB\-\-exclude=alpha,beta\fP" Comma separated list of packages which will be removed from download and extract lists. With this patch, one can use eatmydata debootstrap --include-early=eatmydata ... to get eatmydata. With strace -esync,fsync, I verified that this actually only issues about 20 fsync calls in the final stage of debootstrap. On a rotating hard disk, this reduces the time needed to set up a sid bootstrap from 10 minutes to 4 minutes. On an SSD, the time difference is negligible though. Greetings Marc
Is this intended to be applied instead of, or on top of my initial patch? If the former ( which I'm guessing it is ), I don't see where in-target is modified to *use* it?
Is this intended to be applied instead of, or on top of my initial patch? If the former ( which I'm guessing it is ), I don't see where in-target is modified to *use* it?
Is this intended to be applied instead of, or on top of my initial patch? If the former ( which I'm guessing it is ), I don't see where in-target is modified to *use* it?
Is this intended to be applied instead of, or on top of my initial patch? If the former ( which I'm guessing it is ), I don't see where in-target is modified to *use* it?
This is intended to be used instead of your patch, and the intended use is eatmydata debootstrap --include-early=... which will invoke deboostrap under eatymdata which will cause all binaries to be started under eatmydata and additionally will cause eatmydata to be installed into the chroot at the earliest convenience so that calls started inside the chroot actually find libeatmydata. Greetings Marc
This is intended to be used instead of your patch, and the intended use is eatmydata debootstrap --include-early=... which will invoke deboostrap under eatymdata which will cause all binaries to be started under eatmydata and additionally will cause eatmydata to be installed into the chroot at the earliest convenience so that calls started inside the chroot actually find libeatmydata. Greetings Marc
Hi, I've tested it but could not get good result as yours. - Use SSD and USB2.0 connected HDD - both ext4 formatted - debootstrap is patched with https://bugs.debian.org/844118 Without eatmydata on SSD --------------------------- real 2m10.057s user 1m47.776s sys 0m8.060s With eatmydata on SSD --------------------------- real 2m8.640s user 1m47.546s sys 0m8.049s Without eatmydata on HDD --------------------------- real 2m25.136s user 1m47.758s sys 0m8.603s With eatmydata on HDD --------------------------- real 2m26.790s user 1m48.699s sys 0m8.470s So I doubt using eatmydata.
Version: 1.0.115
Hi everyone,
I’m appalled that this feature request is still open…
I’ve got a need to do this without patching debootstrap, and have had
success (although I did not time it, but someone else can ;-) with this
sequence of commands (assuming dpkg-deb is available ― which my script
checks earlier):
safe_PATH=/bin:/sbin:/usr/bin:/usr/sbin
# […]
case $TERM in
(Eterm|Eterm-color|ansi|cons25|cons25-debian|cygwin|dumb|hurd|linux|mach|mach-bold|mach-color|mach-gnu|mach-gnu-color|pcansi|rxvt|rxvt-basic|rxvt-m|rxvt-unicode|rxvt-unicode-256color|screen|screen-256color|screen-256color-bce|screen-bce|screen-s|screen-w|screen.xterm-256color|sun|vt100|vt102|vt220|vt52|wsvt25|wsvt25m|xterm|xterm-256color|xterm-color|xterm-debian|xterm-mono|xterm-r5|xterm-r6|xterm-vt220|xterm-xfree86)
# list from ncurses-base (6.1+20181013-2+deb10u1)
;;
(screen.*|screen-*)
# aliases possibly from ncurses-term
TERM=screen ;;
(rxvt.*|rxvt-*)
# let’s hope…
TERM=rxvt ;;
(xterm.*|xterm-*)
# …this works…
TERM=xterm ;;
(linux.*)
# …probably
TERM=linux ;;
(*)
die "Your terminal type '$TERM' is not supported by ncurses-base." \
'Maybe run this script in GNU screen?' ;;
esac
# […]
eatmydata debootstrap --arch=arm64 --include=eatmydata --no-merged-usr \
--force-check-gpg --verbose --foreign buster "$mpt" \
http://deb.debian.org/debian sid
# script specified here as it’s normally what buster symlinks to,
# to achieve compatibility with more host distros
# we need this early
(
set -e
cd "$mpt"
for archive in var/cache/apt/archives/*eatmydata*.deb; do
dpkg-deb --fsys-tarfile "$archive" >a
tar -xkf a
done
rm -f a
) || die 'failure extracting eatmydata early'
# the user can delete this later, from the booted system (or apt will)
cp /usr/bin/qemu-aarch64-static "$mpt/usr/bin/" || die 'cp failed'
echo >&2 'I: second stage bootstrap (under emulation), slooow…'
mount -t tmpfs swap "$mpt/dev/shm" || die 'mount /dev/shm failed'
mount -t proc proc "$mpt/proc" || die 'mount /proc failed'
mount -t tmpfs swap "$mpt/tmp" || die 'mount /tmp failed'
chroot "$mpt" /usr/bin/env -i LC_ALL=C.UTF-8 HOME=/root PATH="$safe_PATH" \
TERM="$TERM" /usr/bin/eatmydata /debootstrap/debootstrap --second-stage || \
die 'debootstrap (second stage) failed'
Of course, “success” here means it does not error out… ☻ (also, as it’s
run under emulation it’s very slooow anyway…)
bye,
//mirabilos
Version: 1.0.115
Hi everyone,
I’m appalled that this feature request is still open…
I’ve got a need to do this without patching debootstrap, and have had
success (although I did not time it, but someone else can ;-) with this
sequence of commands (assuming dpkg-deb is available ― which my script
checks earlier):
safe_PATH=/bin:/sbin:/usr/bin:/usr/sbin
# […]
case $TERM in
(Eterm|Eterm-color|ansi|cons25|cons25-debian|cygwin|dumb|hurd|linux|mach|mach-bold|mach-color|mach-gnu|mach-gnu-color|pcansi|rxvt|rxvt-basic|rxvt-m|rxvt-unicode|rxvt-unicode-256color|screen|screen-256color|screen-256color-bce|screen-bce|screen-s|screen-w|screen.xterm-256color|sun|vt100|vt102|vt220|vt52|wsvt25|wsvt25m|xterm|xterm-256color|xterm-color|xterm-debian|xterm-mono|xterm-r5|xterm-r6|xterm-vt220|xterm-xfree86)
# list from ncurses-base (6.1+20181013-2+deb10u1)
;;
(screen.*|screen-*)
# aliases possibly from ncurses-term
TERM=screen ;;
(rxvt.*|rxvt-*)
# let’s hope…
TERM=rxvt ;;
(xterm.*|xterm-*)
# …this works…
TERM=xterm ;;
(linux.*)
# …probably
TERM=linux ;;
(*)
die "Your terminal type '$TERM' is not supported by ncurses-base." \
'Maybe run this script in GNU screen?' ;;
esac
# […]
eatmydata debootstrap --arch=arm64 --include=eatmydata --no-merged-usr \
--force-check-gpg --verbose --foreign buster "$mpt" \
http://deb.debian.org/debian sid
# script specified here as it’s normally what buster symlinks to,
# to achieve compatibility with more host distros
# we need this early
(
set -e
cd "$mpt"
for archive in var/cache/apt/archives/*eatmydata*.deb; do
dpkg-deb --fsys-tarfile "$archive" >a
tar -xkf a
done
rm -f a
) || die 'failure extracting eatmydata early'
# the user can delete this later, from the booted system (or apt will)
cp /usr/bin/qemu-aarch64-static "$mpt/usr/bin/" || die 'cp failed'
echo >&2 'I: second stage bootstrap (under emulation), slooow…'
mount -t tmpfs swap "$mpt/dev/shm" || die 'mount /dev/shm failed'
mount -t proc proc "$mpt/proc" || die 'mount /proc failed'
mount -t tmpfs swap "$mpt/tmp" || die 'mount /tmp failed'
chroot "$mpt" /usr/bin/env -i LC_ALL=C.UTF-8 HOME=/root PATH="$safe_PATH" \
TERM="$TERM" /usr/bin/eatmydata /debootstrap/debootstrap --second-stage || \
die 'debootstrap (second stage) failed'
Of course, “success” here means it does not error out… ☻ (also, as it’s
run under emulation it’s very slooow anyway…)
bye,
//mirabilos
severity -1 normal thanks Hi. Why this bug ( https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633 ) still is not fixed? I did some experiments and here are results: if I run debootstrap stage of d-i under eatmydata, then its time decreases from 755 s to 370 - 405 s. (I installed Debian to USB storage device). Please, fix this bug. Fix is simple, benefits for many users are big. (So I change severity.) I think eatmydata mode should be enabled by default in debootstrap. Both inside of d-i and outside of it. == Askar Safin https://github.com/safinaskar
Control: tag -1 wishlist Please drop the hyperboles and don't abuse the bts. Cheers, Julien
Debootstrap is used in Debian ISO installer (debian-installer, d-i) for initial target rootfs bootstrapping. With sync enabled, this takes hundreds of minutes to unpack and configure base system on a HDD. With eatmydata it takes about a minute. What could be done to introduce eatmydata into debootstrap? Are propsed patches above (in message 50 and before) are not correct or not conform to Debian style or policy? If so, what should be improved?
Petter Reinholdtsen, le lun. 12 avril 2021 07:34:38 +0200, a ecrit: Please do not confuse "ignore" with "terribly understaffed". Samuel
Thank you for the CC. I am not subscribed to debian-boot@ and it would have take a while before I discovered your reply on the web view. [Samuel Thibault] In my experience, "terribly understaffed" can lead to many things being ignored, and I this fund ignored to be the correct term for it. The staff level is useful as an explanation but do not really change the effect, which is that changes for the better do not get applied. I could apply the debian-cd change myself, if everyone agree that it should be applied, but have not touched debian-cd for so long it seemed safest to only publish the patch as a start. Please let me know via <URL: https://bugs.debian.org/986772 > if you agree or disagree with that change, and I can apply it if no-one objects by the end of the week. I am not sadly qualified at the moment to look at debootstrap, and unlikely to find time to reload that code base into my head any time soon. Sorry about that. Looked at the ubuntu patch for debootstrap, as it came from a @ubuntu user, but unfortunately they are not carrying it yet, so it has not been testet there.
Petter Reinholdtsen, le mer. 14 avril 2021 08:38:00 +0200, a ecrit: In my understanding, "ignore" carries a negative meaning, as if d-b people didn't *want* to have a look at the proposed patches. https://www.dictionary.com/browse/ignore « to refrain from noticing or recognizing: Law. (of a grand jury) to reject (a bill of indictment), as on the grounds of insufficient evidence. » being understaffed doesn't imply refraining or rejecting, but simply not having the time to even just look at it. Not having time to delve more. Samuel
for those not following that bug, this has been fixed in git now and as thus will be applied to future builds.
[Samuel Thibault] OK. Personally, I ignore a lot of things due to lack of capacity, and while it can of course be negative for those being ignored, it has a positive effect on me avoiding overload. To me the difference of not noticing and ignoring is the choice one make about what to look at. I assume the understaffed team decide which issues to look at and which to ignore, even if the decision process is is "pick random issue, leave the rest* or "pick issue that look interesting, leave rest for others* or something else. :) Anyway, I am happy to report that action point 4 was just fixed in git according to <URL: https://bugs.debian.org/986772 >. This should make it simple to fix action point 3 too, enabling eatmydata-udeb in the default installation, before the release of Bullseye. As for action points 1 and 2, patching debootstrap and d-i to use new option, I seriously doubt it can be implemented and tested quickly enough, but would love to be wrong. Perhaps all that is needed is more positive test reports for the patch in <URL: https://bugs.debian.org/700633 >. :)
Hi, I build a custom iso base on Debian 10.9. I only add the following line in my preseed file, no others change. d-i partman/early_command \ string anna-install eatmydata-udeb The time record is come from /var/log/installer/syslog before use eatmydata-udeb: May 8 12:43:27 apt-install: Queueing package e2fsprogs for later installation May 8 13:47:47 finish-install: info: Running /usr/lib/finish-install.d/94save-logs 64min20s after use eatmydata-udeb: May 9 08:36:00 apt-install: Queueing package e2fsprogs for later installation May 9 09:21:46 finish-install: info: Running /usr/lib/finish-install.d/94save-logs 45min46s
Dear everyone,
I just copy eatmydata bins and libs into the prepared debootstrap target prior to running debootstrap,
so that LD_PRELOAD set for debootstrap is inherited by dynamic executables run in chroot,
which then can find and successfully load the required .so :
type eatmydata && emd=eatmydata
# ... then prepare installation folder denoted by $target, then copy needed files into it :
[ "$emd" ] && dpkg -L $(dpkg-query -f='${Package} ' -W '*eatmydata*' ) | egrep 'bin/|\.so' | sudo tar -T - -cS | sudo tar -C $target -xpS
time sudo $emd debootstrap $dist $target $mirror
On a very slow USB drive, Kingston DTXM/64GB , initial debootstrap of Debian Trixie took >30 minutes, repeated debootstrap runs gradually extended their run
time to >100 minutes (all of them done under sudo eatmydata), but this modest hack with copying host-installed eatmydata binaries into the target have reduced
its runtime back to <15 minutes, that is, almost 7-fold. And more important, it does not require any patching of debootstrap scripts.
Please review the solution, customize it for your needs and share your feedback.
Thanks in advance for your responses