Dear Maintainer, The module 'loop.ko' is not shipped with the Debian testing netinstall ISO. It should reside in /lib/modules/3.10-2-amd64/kernel/drivers/block/. Because it is missing, it is impossible to mount ISO images during the install and thus preventing installation from ISO, if the module is not manually imported. Please include the kernel module loop.ko in all installation ISOs. It would be awesome, if you could also add iso-scan to the install ISOs, so that one can use the 'findiso=' boot option. This would make it possible to install from the ISO without having to mount it manually during the installation. Thanks, Andreas
[Now follows a somewhat lengthy text. If you are bored, jump directly to the conclusion.] On 30.09.2013 01:58, wrote ian_bruce@fastmail.net: Hi Ian, thank you for this information on bug #724931. I read your posts and it seems that the developers are aware of the problem, but choose not to act: How can isohybrid replace the findiso option? At least for me it makes a huge difference, whether I can use my 32 GB stick for ONE netinst.iso or ONE HUNDRED. Aside from that it is an unnecessary complexity to download the hd-media initrd, which is why I never did that, but instead only downloaded the loop.ko. By the way, I think that it is reason enough to include findiso on the regular CD, when several persons request it. One always has to balance gain and cost and the only cost that I can see, is that the ISO will be larger. I don't know how big findiso is, but probably less then 100 kB. The loop.ko file is 37 kB, so together this gives 137 kB. Since the netinst.iso is about 270 MB, it would grow by about 0.05 %. Who will be hurt by that? On the other hand according to many post etc. on the subject, which I read in the course of the last two years or so, I imagine that a lot of people would like it. I certainly would have installed Debian earlier, if this option had been included on the netinst.iso. On 30.09.2013 01:58, wrote ian_bruce@fastmail.net: http://live.debian.net/ There the option 'findiso=$iso' works as advertised and indeed: $ zcat initrd.img | strings | grep -i findiso if [ -n "$FINDISO" ] && [ "${TORAM}" ] if is_mountpoint /root/lib/live/mount/findiso umount /root/lib/live/mount/findiso rmdir --ignore-fail-on-non-empty /root/lib/live/mount/findiso \ findiso=*) FINDISO="${_PARAMETER#findiso=}" export FINDISO if [ -n "${FINDISO}" ] if [ -f ${mountpoint}/${FINDISO} ] mkdir -p /live/findiso mount -t ${fstype} -o ro,noatime "${devname}" /live/findiso loopdevname=$(setup_loop "/live/findiso/${FINDISO}" "loop" "/sys/block/loop*" 0 "") replace the old one with the new version (and eventually change the name in the grub.cfg). If I extracted the kernel and initrd, I would have to do this on every update as well. Luckily, most of the ISOs I use come with a grub.cfg, so I can copy the entries from there and adapt them, e.g. with a findiso option or options for my locales. This should just be a compact description of what I did: - I installed Grub2 to the USB stick via grub-install. - Then I copied some ISOs to the $USB/ISO folder. - After that I adapted $USB/boot/grub/grub.cfg to get a Multiboot USB stick, that boots directly from the ISOs. I had tried various tools for this in the past (not yours, though) and was not quite satisfied with them. The above mentioned method seems to be the most efficient one for me. Conclusion: - I see no reason NOT to include findiso and loop.ko on every installer CD. - Hopefully Joey can give us an update of his thoughts on the matter. Best regards, Andreas
On Mon, 30 Sep 2013 13:10:01 +0200 Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:
I said the same thing, in my post to the mailing list:
This is unnecessarily destructive, and makes it hard to install
multiple distributions on the same flashdrive, or to use it for
other purposes. The smallest flashdrive you can currently buy is
8GB; it makes no sense that you would have to have a different one
for every live ISO you might want to use.
Even if you did download the hd-media initrd, it still wouldn't allow
you to use a boot parameter to specify the pathname of the ISO image
file you want to use. It appears that it just grabs anything it can find
that looks like a Debian ISO, and asks the user to confirm it.
Note that "loop.ko" is included on the ISO (but not the initrd), in the
form of /pool/main/l/linux/loop-modules-*.udeb packages.
My patch, which seems to do everything that's necessary, is a few
hundred BYTES (uncompressed). So including that and the "loop.ko" module
in the initrd will increase the size of the ISO image by about one part
in ten thousand, which certainly doesn't seem worth worrying about.
As you point out next, it appears that this functionality is included on
the Debian live CD. What possible rationale could there be for having it
there, but not on the installer image?
I actually didn't think to investigate the live CD. It never occurred to
me that one might have it, and the other not.
The hd-media initrd is no different from the ones on the standard
installer ISO, in this regard:
$ zcat debian-7.1.0-amd64-i386-netinst/install.{386,amd}/{,gtk/}initrd.gz |
strings | grep -i findiso
$
$ zcat debian-7.1.0-amd64-hd-media/{,gtk/}initrd.gz |
strings | grep -i findiso
$
My "loopmount=" patch is again attached to this message, for the sake of
the bug report.
tags 724931 patch thanks Hi, On 30.09.2013 14:52, wrote ian_bruce@fastmail.net: Thank you for mentioning this. I didn't look there. I have applied your patch to the debian-testing-amd64-netinst.iso and also included the loop.ko. This increased the size from 230.7 MB to 232.3 MB, i.e. by 0.7 %. The patch itself is very straightforward, short, and most importantly works very well. Thank you! The only missing thing is 'LOOPFS=' for kfreebsd and hurd. Therefore I would suggest to forget about the findiso option and simply apply this patch to all the Debian isos. (For comptability one could rename 'loopmount=' into 'findiso='.) Best regards, Andreas
This must be because of differences in compression, or something else in the build method. The combined size of the patch code and the loop.ko module is about 37KB (uncompressed), not 1.6MB. In other words, it's completely negligible. I'm glad you found it satisfactory. Yes, I pointed that out in my original post to the mailing list. I will forward that to the bug report, for completeness. I certainly have no objection to that, although I think it would be a good thing if the install ISOs and the live ISOs both used the same loopmount mechanism. It might be better not to do that, in case there are any slight differences in function, which is to say, "incompatibilities". I think that "loopmount" is more descriptive of what is happening than "findiso", anyway.
This message describes the motivation for the "debian-iso-loopmount.diff" patch, which is reproduced above. It was originally posted here: http://lists.debian.org/debian-boot/2013/09/msg00509.html Begin forwarded message: Date: Sat, 28 Sep 2013 03:08:48 -0700 From: <ian_bruce@fastmail.net> To: debian-boot@lists.debian.org Subject: PATCH: specify pathname for ISO loop-mount This patch introduces a boot parameter, "loopmount=", which allows the specification of a full pathname (relative to the root directory of some block device) of an ISO image file which should be loop-mounted as the Debian installer root filesystem. The main purpose of this feature is to facilitate the creation of USB flashdrives which can contain multiple Linux live distributions, selectable at boot time via a Syslinux menu. http://sourceforge.net/projects/multibootusb/ The current method of creating a Debian installer flashdrive overwrites the existing partition table and filesystem: *Warning* The procedures described in this section will destroy anything already on the device! http://www.debian.org/releases/stable/amd64/ch04s03.html.en This is unnecessarily destructive, and makes it hard to install multiple distributions on the same flashdrive, or to use it for other purposes. The smallest flashdrive you can currently buy is 8GB; it makes no sense that you would have to have a different one for every live ISO you might want to use. The effect of this patch is that you can just copy the Debian installer ISO into the existing filesystem on the flashdrive, and boot it with GRUB or Syslinux, using the "loopmount=" boot parameter to specify the ISO pathname. It will be apparent from reading the patch that if this parameter is not specified, everything functions EXACTLY as before; the new code will not be run. The changes amount to no more than forty lines of new code, in one file. I suppose that this patch applies to the "cdrom-detect" udeb, although I just patched the initrd directly. A dependency on "loop-modules" will need to be added, to make sure that the "loop.ko" kernel module gets copied into the initrd. One issue that probably should be addressed is what to do if the specified ISO is not actually present; the boot process should fail gracefully, but I'm not sure how best to accomplish that. Minor changes (see the "LOOPFS" variable) would be needed to make it work for FreeBSD and Hurd; I've only tried it with Linux. Comments are welcome, but apart from the above, I don't think there's much wrong with this patch; testing shows that it seems to work perfectly.
Hi, The size difference in the ISO was due to an additional vmlinuz (2.5 MB) in /install.amd/gtk. If have created a shell script to patch an existing ISO and attached it to this mail. This script has to be executed with root privileges! It expects to find a debian-testing-amd64-netinst.iso mounted to /mnt and the patch file debian-iso-loopmount.diff in the directory where it is executed. It will create a patched ISO named debian-testing-amd64-netinst-loopmount.iso in the directory where it is executed. The size of this patched ISO is 229,8 MB, i.e. less than the original size, which is probably due to better compression. On 30.09.2013 23:16, wrote ian_bruce@fastmail.net: but I agree, that this is bad, because of possible different behaviour. I think the live ISO should switch to 'loopmount=', because it is a very small patch, except, of course, if there are any features 'findiso=' has that are not provided by 'loopmount='. While testing the patched ISO, I noticed a problem very similar to bug #724933: apt-setup is not working correctly. The error message was: main-menu[524]: INFO: Menu item 'apt-setup-udeb' selected apt-setup: umount: can't umount /target/media/cdrom0: Invalid argument apt-setup: Using CD-ROM mount point: /media/cdrom/ apt-setup: Identifying.. apt-setup: [ae8b99d023e59c58db825bbd443912e2-2] apt-setup: Scanning disc for index files.. apt-setup: Found 0 package indexes, 0 source indexes, 0 translation indexes and 0 signatures apt-setup: W: Failed to mount '/dev/sr0' to '/media/cdrom' apt-setup: E: Unable to locate any package files, perhaps this is not a Debian Disc or the wrong architecture? kernel: [1392,717219] ISO 9660 Extensions: Microsoft Joliet Level 3 kernel: [1392,717334] ISO 9660 Extensions: RRIP_1991A After continuing in the menu, the following line is added: apt-setup: warning: /usr/lib/apt-setup/generators/40cdrom returned error code 1; discarding output apt-setup tries to create a list entry in /etc/sources.list for the CD. I think this entry should be removed at the end of installation. Is that correct? (It didn't on my last installation.) Assuming the entry is only valid during installation, apt-setup should simply look in /cdrom (or create a symlink from /cdrom to /target/media/cdrom0) and not try to (u)mount anything. I consider this a bug in apt-setup that should be remedied whether or not the loopmount patch is included. This bug is no big problem for the netinstall ISO, because it does not load any more packages from the ISO, but for a full ISO this is a critical problem. Have you ever tried the patch with a full image? Best regards, Andreas
This is an improved version of the patch I originally posted to this bug
report. It applies mainly to the "cdrom-detect" udeb, although I suggest
that the ISO volume label (such as "Debian 7.1.0 M-A 1") be included
somewhere in the initrd; currently it does not appear to be. As I said
previously, a dependency on "loop-modules" will have to be added, so
that the "loop.ko" kernel module gets copied into the initrd.
This patch adds support for a boot parameter, "loopmount=", which
specifies an ISO image file to be loaded from another filesystem, and
used to boot from. For example:
loopmount=KINGSTON:/linux/debian-7.1.0-amd64-i386-netinst.iso
This directs the initrd to look for a block device filesystem with the
label "KINGSTON" (as reported by /sbin/blkid), and to use the file
"debian-7.1.0-amd64-i386-netinst.iso", in the subdirectory "/linux" (the
leading "/" is not actually necessary) as a boot image.
In addition to the filesystem labels found in /dev/disk/by-label/, the
block device can be identified by UUID, as found in /dev/disk/by-uuid/.
If the filesystem label/UUID (and ":") is not specified, then all
available partitions and CD devices will be searched for the specified
ISO image file, in the specified directory.
If the "loopmount" parameter is not used, then the initrd will search
for a direct ISO filesystem in the same way as previously, although any
block device with the expected volume label ("Debian 7.1.0 M-A 1") will
be tried first.
It is expected that this facility will be most useful with USB
flashdrives, although it is in no way limited to those. It can be
applied to any block device with a vfat, ext{2,3,4}, or iso9660
filesystem; others could easily be added.
Testing seems to show that the patch works well; the relevant portion of
the installation log is reproduced below.
Oct 4 06:02:19 cdrom-detect: Searching for Debian installation media...
Oct 4 06:02:19 cdrom-detect: Devices: '/dev/sr0'
Oct 4 06:02:19 cdrom-detect: LOOPDEV='KINGSTON' LOOPFILE='/linux/debian-7.1.0-amd64-i386-netinst.iso'
Oct 4 06:02:19 cdrom-detect: trying loopmount on (/dev/disk/by-label/KINGSTON)...
Oct 4 06:02:19 kernel: [ 226.383202] FAT-fs (sdf1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct 4 06:02:19 kernel: [ 226.395607] loop: module loaded
Oct 4 06:02:19 cdrom-detect: CD-ROM mount succeeded: device=/loop/linux/debian-7.1.0-amd64-i386-netinst.iso fstype=iso9660
Oct 4 06:02:19 kernel: [ 226.474076] ISO 9660 Extensions: Microsoft Joliet Level 3
Oct 4 06:02:19 kernel: [ 226.474149] ISO 9660 Extensions: RRIP_1991A
Oct 4 06:02:19 cdrom-detect: Detected CD 'Debian GNU/Linux 7.1.0 "Wheezy" - Official Multi-architecture amd64/i386 NETINST #1 20130615-23:44'
Oct 4 06:02:19 cdrom-detect: Detected CD with 'stable' (wheezy) distribution
This patch will increase the size of the (uncompressed) initrd by about
37KB: 36KB for the required "loop.ko" module, and 1KB of extra boot
script. In a 200MB or 300MB ISO image, this is surely immaterial.
Since the patch adds a previously unavailable feature, which will not
operate unless activated by an explicit boot parameter, there seems to
be no reason why it should not be applied. Compare it with the current
procedure for installing Debian from a USB flashdrive:
http://www.debian.org/releases/stable/amd64/ch04s03.html
As an administrative note, I recommend that this bug report be retitled;
the current title no longer reflects what we are discussing, and the
availability of my patch is surely now the most important issue relating
to it.
Hi, @Ian: Thanks for working on this feature. Sadly it only "seems" to work well, but in fact, your previous (and I assume also this) patch break apt-setup trying to add the CD to /etc/sources.list. I am currently working on this problem and hope to finish this soon. Furthermore, for consistency I suggest to rename the boot option to 'findiso=', since the live image already uses this option. They use a implementation fairly similar to your first patch and I am sure, that they would gladly add any additional features implemented in the debian-installer. The iso-scan package on the other hand seems not be in use currently. (I really don't care for the name, but it should be the same for all Debian ISOs. It's bad enough, that this is inconsistent between distributions!) Best regards, Andreas
I'm sorry to hear that. I'll look into it as well; the error log you included in your previous message seemed to indicate that assumptions were being made about what block device represented the ISO, or where it was mounted. Since the "cdrom-detect" script already figures this out, this information just needs to be exported (via a file?) to other scripts. The effect of my second patch is that the ISO no longer has to be "found"; we can specify EXACTLY where (what filesystem, what pathname) it is located, and so that parameter name is now even more inappropriate, and that implementation quite incompatible with mine. I'm not at all sure about that. I've previously found them to be quite unwilling to make obviously beneficial changes, without being able to provide any rational reason for that refusal. http://lists.debian.org/debian-live/2013/08/threads.html#00054 http://lists.debian.org/debian-live/2013/09/threads.html#00000 And notice that so far, loopmount ISN'T implemented in the Debian installer; this discussion doesn't yet involve anybody except you and me. It turns out that it is, but only in the "hd-media" images, not the standard ISOs. http://lists.debian.org/debian-boot/2013/09/msg00097.html Which as you and I have both pointed out, is actually kind of dumb. I agree on both counts, but as I said above, there's not much point in making the names consistent, if the behavior isn't. For example, the iso-scan package in Ubuntu allows you to specify, via a boot parameter, the pathname for the image file, whereas the Debian version apparently doesn't, even if it is included in the initrd. So it would actually be better if the names WERE different, so as not to delude people into thinking that the features are the same. On another issue, I have some advice about your ISO-patching script. (I don't patch ISOs, but only initrds, because I prefer that the official MD5 and SHA256 hashes can still be verified.) You said it had to be executed with root privileges, presumably so that file ownership and device files come out right. You should investigate the "fakeroot" utility, which obviates the need for this. And is it possible to rename the bug report to something more descriptive?
tags 724931 patch
tags 724933 patch
thanks
Hi,
I made some changes/improvements to the patch provided by Ian:
* Support a 'firmware' folder on the same device as the ISO: Due to
being mounted, it was not automatically checked with the old patch.
* Removed "DISTRIB_LABEL" variable: Such things should not be
hard-coded! Furthermore, this is unrelated to the 'loopmount=' option
(and I don't know what the benefit of this should be). Ian, if you want
something like this you probably should open another bug and explain
there the need for this further.
* Added support for kfreebsd/hurd and any filesystem by using blkid to
detect the filesystem of the device to be mounted.
* Added iteration loop over $dev_given (i.e. is it used as
'loopmount=DEVICE:ISO'), "usb-partition", "cd" and "partition". This is
to ensure, that the most likely sources are looked at first.
* Changed/added log messages
* Added comments
* Corrected formatting and removed trailing whitespaces
The new apt-cdrom-setup patch enables the ISOs to be added to
/etc/apt/sources.list:
* Don't (u)mount for ISOs
* Added a if-clause to the chroot_cleanup_localmounts hack, to prevent
it from moving a file, that is possibly not even there (especially since
it get's removed in 40cdrom and 41cdset).
* Added cd-set support: ISOs from the set (i.e. starting with the same
string up to 'CD-' (or 'DVD-' for DVDs resp. 'BD-' for bluerays)) in the
same directory can be loaded. (The entries for the additional files are
removed, when the installation is finished.)
I would also suggest to add a loopback.cfg (similar to the one I
attached) to /boot/grub/. (By the way, I noticed that the /boot folder
is missing from the debian-7.1.0-i386-netinst.iso. It should be added
there.) If it is possible, this should also be done for /isolinux. Then
it can be loaded by grub2 with e.g.:
menuentry "Debian Testing Netinstall (64 bit)" {
set gfxpayload=auto
set iso_path="/ISO/debian-testing-amd64-netinst.iso"
export iso_path
boot_options=locale=de_DE.UTF-8
loopback loop $iso_path
set root=(loop)
configfile /boot/grub/loopback.cfg
loopback -d loop
}
I also updated the script to apply the patches for testing purposes:
This script has to be executed with root privileges!
It's first argument is the path/filename of the Debian install ISO
file to be patched.
The filename should contain amd64 or i386, indicating which
architecture is used.
The ISO will be mounted to /mnt, which will be umounted before.
If a loopback.cfg is in the directory where the script is executed, it
will be included in /boot/grub/.
If the architecture is i386, every '.amd' in the loopback.cfg will be
changed to '.386'.
The script's second argument is the patch file to be applied to initrd
(e.g. initrd_loopmount.patch).
It's third argument is the patch file to be applied to the
apt-cdrom-setup udeb (e.g. apt-cdrom-setup_loopmount.patch).
It will create a patched ISO named ${ISO}-loopmount.iso in the
directory where it is executed.
I tested with the following ISOs:
* debian-testing-amd64-netinst-loopmount.iso
* debian-7.1.0-i386-netinst-loopmount.iso
* debian-7.1.0-amd64-netinst-loopmount.iso
* debian-7.1.0-amd64-CD-1.iso
* debian-7.1.0-amd64-CD-2.iso (+)
* debian-7.1.0-amd64-CD-3.iso (+)
* debian-7.1.0-amd64-DVD-1-loopmount.iso
* debian-7.1.0-amd64-DVD-2-loopmount.iso (+)
(+) These ISOs were not modified, but just copied to the same directory
as the first ISO of the set on the USB stick.
Problems that I noticed with this patch:
* The CD-set installation (CD 1, 2 and 3) works well, until it
actually tries to install the packages. Then tasksel fails, complaining
about unauthenticated packages:
For the installation to continue, one has to change the following
line in /target/usr/bin/tasksel
push @cmd, qw{apt-get -q -y -o APT::Install-Recommends=true -o
APT::Get::AutomaticRemove=true install};
to:
push @cmd, qw{apt-get -q -y --force-yes -o APT::Install-Recommends=true
-o APT::Get::AutomaticRemove=true install};
This seems OK, since I changed a package in the original CD and thus
the Release.gpg is not a good signature. But I did not change anything
on CD-2 or CD-3 that I loaded. Therefore this error should also occur if
only installing from the first CD, but it does not: the installation
works fine (though without print-server and laptop task)!
I'm not sure what the correct behaviour should be, but this is
clearly inconsistent: Either it should reject installation from the
first CD, since that is the 'bad' one, or it should also allow
installation with multiple CDs. Perhaps the later is the case, but I
misconfigured the additional CDs? I had to use 'deb file:' for this,
since when using 'deb cdrom:', the installer stops, when the next CD
needs to be mounted, and I found no way to automate this. Apparently apt
does not support the concept of multiple CD drives.
* When the installation (including gnome desktop) from the CD set (CDs
1, 2 and 3) or the DVD-1 is nearly finished, the error
"Could not build the hash file for en-common"
occurs and it is suggested, that one sends a complaint to the
maintainer of the package that provides 'en'.
The error message in /var/log/syslog are the following two lines:
aspell-autobuildhash: processing: en [en-common]
Error: /dev/null:1: The key "/usr/bin/aspell" is unknown.
I think the error is unrelated to this patch, but I am not sure,
since I cannot test it with the original ISO, because that doesn't
support multiple CDs when booting from USB (and my spare USB for
isohybrid testing is too small for the DVD).
On 04.10.2013 15:02, ian_bruce@fastmail.net wrote:
> The effect of my second patch is that the ISO no longer has to be
> "found"; we can specify EXACTLY where (what filesystem, what pathname)
> it is located, and so that parameter name is now even more
> inappropriate, and that implementation quite incompatible with mine.
I agree that now the functionality is (from the user's point of view)
different to that in the live image, because of the possibility to
specify the device by "loopmount=DEVICE:ISO". Using such a string with
the findiso from the live image surely doesn't work.
So I think the best is to use the name loopmount and when (if?) the new
functionality is added to the live image, they should change their name
accordingly.
>> and I am sure, that they would gladly add any additional features
>> implemented in the debian-installer.
>
> I'm not at all sure about that. I've previously found them to be quite
> unwilling to make obviously beneficial changes, without being able to
> provide any rational reason for that refusal.
>
> http://lists.debian.org/debian-live/2013/08/threads.html#00054
> http://lists.debian.org/debian-live/2013/09/threads.html#00000
I read the discussion and let me add my opinion: I think the Debian live
team wants to avoid any additional work, which is quite understandable,
since they probably have enough to do already (and do quite a good job).
But in this special case I wonder, if it wouldn't have been less work
just to include the two packages than to write all those mails...
> And notice that so far, loopmount ISN'T implemented in the Debian
> installer; this discussion doesn't yet involve anybody except you and
> me.
I think (and very much hope) that a number of people on the debian-boot
mailing list also read this discussion and are therefore 'involved'.
Before, the patch had problems, which are solved now. So I think this
patch is ready for being included in all the Debian installers, since
from the remaining two problems, that I could detect with this patch,
one is probably solved by an official build (Release.gpg issue), and the
other is apparently a bug in a different package (aspell).
When the patch is included, the documentation for installation from USB
needs to be updated.
>> The iso-scan package on the other hand seems not be in use currently.
>
>It turns out that it is, but only in the "hd-media" images, not the
>standard ISOs.
Oh, I forgot the hd-media images, because ... ugh ... well ... they are
not included on any Debian installation ISO! ;)
> (I don't patch ISOs, but only initrds, because I prefer that the
> official MD5 and SHA256 hashes can still be verified.)
I also prefer the official signed hashes, but this script is meant for
testing purposes only, and will be obsolete, as soon as this patch is
included in the Debian installer.
> You said it had to be
> executed with root privileges, presumably so that file ownership and
> device files come out right. You should investigate the "fakeroot"
> utility, which obviates the need for this.
The script attached to this mail needs root privileges to (u)mount the
ISO, so it wouldn't make much difference, if I used fakeroot for the rest.
> And is it possible to rename the bug report to something more
> descriptive?
Feel free to rename it to whatever you think is best by sending a mail
with the following content ('NEW_TITLE' appropriately replaced) to
control@bugs.debian.org:
retitle 724931 NEW_TITLE
thanks
I hope the loopmount option will soon be in the official Debian
installation ISOs.
Best regards,
Andreas
(I see that Andreas has recently posted a set of patches. The patches I have attached below are not based on that work, although they address some of the same problems. At the end of this message, are some comments about how our alternative solutions might be combined.) http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608201 I have therefore adopted the (partial?) solution which was applied in that case, which is that ISO images which are found somewhere other than an actual CD, do not get included in "sources.list". Which seems kind of unfortunate; the same USB flashdrive could be used again, or an actual CD might show up later, but that's how they resolved it, and it's not what we're mainly concerned with right now. If a better solution is developed sometime later for those situations, it should apply to the loopmount case as well. So for now, the solution is to export the fact of the loopmount to the configuration database, where it can later be checked by the "apt-setup" package, and handled the same way as the other non-CD cases. Two patchfiles are attached: a slightly altered one for "cdrom-detect", and a new one for "apt-setup". I haven't actually tested the latter, because I'm working with the "netinst" ISO, which doesn't seem to include that package. Here are the relevant sections of the installer logs: "loopmount=KINGSTON:/linux/debian-7.1.0-amd64-i386-netinst.iso" Oct 6 07:35:06 cdrom-detect: Searching for Debian installation media... Oct 6 07:35:06 cdrom-detect: removable devices: (/dev/sr0) Oct 6 07:35:06 cdrom-detect: LOOPDEV='KINGSTON' LOOPFILE='/linux/debian-7.1.0-amd64-i386-netinst.iso' Oct 6 07:35:06 cdrom-detect: trying loop-mount on (/dev/disk/by-label/KINGSTON)... Oct 6 07:35:06 kernel: [ 322.483186] FAT-fs (sdf1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! Oct 6 07:35:06 kernel: [ 322.495521] loop: module loaded Oct 6 07:35:06 kernel: [ 322.571304] ISO 9660 Extensions: Microsoft Joliet Level 3 Oct 6 07:35:06 cdrom-detect: CD-ROM loop-mount succeeded: device=/dev/disk/by-label/KINGSTON/linux/debian-7.1.0-amd64-i386-netinst.iso fstype=iso9660 Oct 6 07:35:06 kernel: [ 322.578804] ISO 9660 Extensions: RRIP_1991A Oct 6 07:35:06 cdrom-detect: Detected CD 'Debian GNU/Linux 7.1.0 "Wheezy" - Official Multi-architecture amd64/i386 NETINST #1 20130615-23:44' Oct 6 07:35:06 cdrom-detect: Detected CD with 'stable' (wheezy) distribution "loopmount=/linux/debian-7.1.0-amd64-i386-netinst.iso" Oct 6 08:01:19 cdrom-detect: Searching for Debian installation media... Oct 6 08:01:19 cdrom-detect: removable devices: (/dev/sr0) Oct 6 08:01:19 cdrom-detect: LOOPDEV='' LOOPFILE='/linux/debian-7.1.0-amd64-i386-netinst.iso' Oct 6 08:01:19 cdrom-detect: trying loop-mount on (/dev/sda1 /dev/sda10 /dev/sda2 /dev/sda3 /dev/sda4 /dev/sda5 /dev/sda6 /dev/sda7 /dev/sda8 /dev/sda9 /dev/sdb1 /dev/sdb10 /dev/sdb2 /dev/sdb3 /dev/sdb4 /dev/sdb5 /dev/sdb6 /dev/sdb7 /dev/sdb8 /dev/sd Oct 6 08:01:19 kernel: [ 211.616937] FAT-fs (sda1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! Oct 6 08:01:19 kernel: [ 211.664572] FAT-fs (sda10): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! Oct 6 08:01:19 kernel: [ 211.740574] FAT-fs (sda2): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! Oct 6 08:01:19 kernel: [ 211.804567] FAT-fs (sda3): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! ... Oct 6 08:01:22 kernel: [ 214.316771] FAT-fs (sdf1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! Oct 6 08:01:22 kernel: [ 214.329130] loop: module loaded Oct 6 08:01:22 kernel: [ 214.399019] ISO 9660 Extensions: Microsoft Joliet Level 3 Oct 6 08:01:22 cdrom-detect: CD-ROM loop-mount succeeded: device=/dev/sdf1/linux/debian-7.1.0-amd64-i386-netinst.iso fstype=iso9660 Oct 6 08:01:22 kernel: [ 214.406518] ISO 9660 Extensions: RRIP_1991A Oct 6 08:01:22 cdrom-detect: Detected CD 'Debian GNU/Linux 7.1.0 "Wheezy" - Official Multi-architecture amd64/i386 NETINST #1 20130615-23:44' Oct 6 08:01:22 cdrom-detect: Detected CD with 'stable' (wheezy) distribution (Can these "filesystem will be case sensitive" log messages be shut off, somehow? Isn't that kind of the point of VFAT?) ------- Here are my comments on Andreas' latest patches, based on a quick reading: I don't know if you have looked at bug #608201; it seems like our current problem is just another specific case of that general situation, which awaits a general solution. You can see how it was dealt with previously by searching for the strings "hybrid" and "usb-hdd" in the file "apt-setup/generators/40cdrom". It appears that your changes to that file are similar to mine; however, I think mine are preferable, in this regard: I see that you have copied (in this and several other places) this piece of code, for determining if a loop-mount is being used: for arg in $(cat /proc/cmdline); do case $arg in loopmount=*) LOOPMOUNT=${arg#loopmount=} LOOPFILE=${LOOPMOUNT#*:} [ "$LOOPFILE" != "$LOOPMOUNT" ] && LOOPDEV=${LOOPMOUNT%:*} ;; esac done Based on my current patches, this is now unnecessary. Do this instead: loopmount=$(db_getval cdrom-detect/cdrom_loopdev) if [ "$loopmount" ] ; then ... ; else ... ; fi "$loopmount", if non-null, will contain the actual device (not pathname, see "$(db_getval cdrom-detect/cdrom_device)" for that) on which the ISO image file resides, if that is ever useful. I'm not sure if I understand this: if [ "$ROOT" ] && [ "$LOOPMOUNT" ]; then save_label fi Does that mean that the ISO will be specified in "sources.list"? If so, that's valuable. Can that be made to work for the "hybrid" and "usb-hdd" cases as well? (Search for those strings in "cdrom-detect.postinst" to see what they mean.) Your fix for locating firmware images is welcome, with the same comment as above, about detecting loop-mounts. However, I did not see any benefit in your changes to "cdrom-detect.postinst". It's easy enough for somebody to add the proper filesystem types for Hurd and FreeBSD. My current patch produces slightly more informative log entries, if you care about that. (see above for examples) Your work on CD-sets is, of course, important; my patches do not address that issue.
Hi,
I further improved the patch by solving three problems and merging with
the second patch from Ian.
First the solved problems:
* Filesystem support:
On 06.10.2013 04:06, Andreas Cadhalpun wrote:
> * Added support for kfreebsd/hurd and any filesystem by using blkid
> to detect the filesystem of the device to be mounted.
While this is theoretically true, practically it is not, because not
all filesystem drivers are included in the initrd. For example neither
the ntfs nor the ext4 driver are included. For this patch to be useful,
it has to be possible to loopmount from USB or hard disk. Therefore I
suggest to include (in addition to the loop-module) the following
modules (I updated the Apply_Patches.sh script accordingly):
Highly recommended filesystem drivers:
* ext2/ext3/ext4 (currently default Debian file system)
* ntfs (currently default Windows file system)
* udf (probably the future of USB file systems)
These together are about 450 kB.
Optional filesystem drivers:
* btrfs
* jfs
* squashfs
* xfs
These together are about 650 kB.
Adding all of them to both initrd's makes the ISO approximately 2 MB
larger, which should be no problem.
* aspell fails to install when using CD-set: This was due to the
following code in 41cdset:
$logoutput $chroot $ROOT apt-cdrom -d $mount_point add \
-o Dir::Etc::SourceList=/dev/null \
</dev/null;
I used /dev/null as argument for Dir::Etc::SourceList since the returned
list is not used and I thought it would simply do nothing, but
apparently it breaks the system somehow. Changing /dev/null to $tmp
there and afterwards removing the $tmp file resolved the problem.
* CD-set authentication issue: I checked using the ISO as 'deb file:'
repositories in an installed system. There the CD-2 works without
problems, but CD-1 complains, as it should, since it is modified. But
this is no explanation of the inconsistent behaviour of apt during
installation. Perhaps 'deb cdrom:' repositories do generally not get
checked for a correctly signed Release.gpg, but an additional 'deb
file:' triggers a check?
As a workaround for this problem, I changed the 'deb file:' entries to
'deb [ trusted=yes ] file:'. This should not be a security problem,
since, when the checksum of the ISO is identical to the downloaded
signed one, the user can be sure of the integrity of all software on the
ISO. If it is not (or the user does not check) anything could be on the
ISO anyway. Since the [ trusted=yes ] flag does no harm, I included it
in this patch (if it works with an officially signed CD-1 without this
flag, then it can be removed).
Furthermore I noticed, that the loopback.cfg I uploaded utilized
'loopback=' instead of 'loopmount=', which I corrected now. I testet,
that grub2 can load it with e.g.:
menuentry "Debian Testing netinstall loopback (amd64)" {
set gfxpayload=auto
set iso_path="/ISO/debian-testing-amd64-netinst-loopmount.iso"
export iso_path
boot_options=locale=de_DE.UTF-8
export boot_options
loopback loop $iso_path
set root=(loop)
configfile /boot/grub/loopback.cfg
loopback -d loop
}
The Apply_Patches.sh script now uses xorriso and produces isohybrid ISOs
that can be directly dd'ed to an USB stick. I checked, and this still
works, at least for the debian-7.1.0-amd64-CD-1.iso (though the
installer does not use the native resolution of the display).
I tested this patch with all the following ISOs loopmounted using
loopback.cfg from an USB with UDF file system and it seems to work
perfectly:
* debian-7.1.0-amd64-netinst.iso
* debian-7.1.0-i386-netinst.iso
* debian-7.1.0-amd64-CD-1.iso
* debian-7.1.0-amd64-CD-2.iso (+)
* debian-7.1.0-amd64-CD-3.iso (+)
* debian-7.1.0-amd64-DVD-1.iso
* debian-7.1.0-amd64-DVD-2.iso (+)
(+): Not modified, just copied to the folder of the first ISO in the set.
I did not inclued the testing ISOs in that list, although the patch
seems to work there as well, because the testing ISOs are currently in
no good shape (e.g. bug #725714).
On 06.10.2013 15:04, ian_bruce@fastmail.net wrote:
> I think the problem is related to this bug:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608201
I haven't noticed bug #608201, but indeed this is related.
> I have therefore adopted the (partial?) solution which was applied in
> that case, which is that ISO images which are found somewhere other
> than an actual CD, do not get included in "sources.list".
I understand the solution differently: The ISOs get included in the
sources.list, but for some reason I don't know, the default behaviour of
'40cdrom' is to unmount all CDs/ISOs and let 'apt-cdrom add' mount them
again. (This is strange, since CD sets are handled in 41cdset and not in
40cdrom.) Because 'apt-cdrom add' does detect only CDs, not USB-HDD nor
isohybird USBs nor loopmounted ISOs, this default behaviour has to be
changed in those cases.
The choosen method in bug #608201 was to export from 'cdrom-detect',
whether it is an isohybrid or USB/USB-HDD and then from this determine
in '40cdrom', whether or not the device is detectable by 'apt-cdrom
add', i.e. is an actual disk in a drive. This is rather complicated, so
I moved the determination of mountability to 'cdrom-detect' and exported
this information for '40cdrom'. (Since I don't know, if the
isohybrid/USB-HDD information is needed anywhere else, I still export that.)
> If a better solution is developed sometime later for those
> situations, it should apply to the loopmount case as well.
From my point of view, it should work if '40cdrom' just never unmounts
anything and 'apt-cdrom add' is never allowed to mount anything, which
would eliminate the need for exporting this information from
'cdrom-detect' to '40cdrom'.
> Two patchfiles are attached: a slightly altered one for
> "cdrom-detect", and a new one for "apt-setup". I haven't actually
> tested the latter, because I'm working with the "netinst" ISO, which
> doesn't seem to include that package.
The debian-7.1.0-amd64-netinst.iso contains the relevant files in:
/pool/main/a/apt-setup/apt-cdrom-setup_0.79_all.udeb
> It appears that your changes to that file are similar to mine;
> however, I think mine are preferable, in this regard:
>
> I see that you have copied (in this and several other places) this
> piece of code, for determining if a loop-mount is being used:
>
> for arg in $(cat /proc/cmdline); do
> case $arg in
> loopmount=*)
> LOOPMOUNT=${arg#loopmount=}
> LOOPFILE=${LOOPMOUNT#*:}
> [ "$LOOPFILE" != "$LOOPMOUNT" ] && LOOPDEV=${LOOPMOUNT%:*}
> ;;
> esac
> done
>
> Based on my current patches, this is now unnecessary. Do this instead:
>
> loopmount=$(db_getval cdrom-detect/cdrom_loopdev)
I actually thought about using debconf to export the 'loopmount'
variable, but since I didn't know how it works, I decided to use the
'quick and dirty' approach, that I knew to work. Using db_set is surely
better, but your implementation does not work. I checked the manual:
http://manpages.ubuntu.com/manpages/lucid/en/man7/debconf-devel.7.html
One needs to create the appropriate templates in the
cdrom-detect.templates file, which I added now (and also for
cdrom_options and cdrom_mountable). Using loopdev instead of loopmount
to determine, whether an ISO is mounted, is preferable, because even if
the loopmount boot parameter is set, but the ISO could not be loaded and
instead e.g. a CD is loaded, none of the loopmount specific changes in
the code after cdrom-detect will be executed.
> I'm not sure if I understand this:
>
> if [ "$ROOT" ] && [ "$LOOPMOUNT" ]; then
> save_label
> fi
>
> Does that mean that the ISO will be specified in "sources.list"?
No it does not. It just saves the label of the current CD for '41cdset',
which needs to know this. (It probably would be better to use debconf
here as well.)
The entry in sources.list is created with the following command:
$logoutput $chroot $ROOT apt-cdrom add \
-o Dir::Etc::SourceList=$tmp \
</dev/null; then
cat $ROOT$tmp >> $file
Note that these entries only get to the actual sources.list, if the
script exits successfully and otherwise they are ignored.
> However, I did not see any benefit in your changes
> to "cdrom-detect.postinst".
Then let me explain them to you. The remaining differences between my
latest patch and your second patch in cdrom-detect are:
* I did not include the $DISTRIB_LABEL, because I don't know how to do
it best, and it works fine without (although it perhaps has to check
more devices). If it helps you, the actually used $DISTRIB_LABEL is
included on the ISO: In the file .disk/mkisofs is the command, with
which the ISO was created and the option -V 'Debian 7.1.0 amd64 1'
produces the label.
You commented it out with the note:
# this should wait for a proper solution for bug #608201
I see no connection to that bug, since that has nothing to do with
the installer device not being found, but rather it not being included
in the apt database.
In cdrom-detect.postinst:
* In my patch the filesystem of the device to be mounted is determined
automatically. Ian wrote:
> It's easy enough for somebody to add the proper file system types for
> Hurd and FreeBSD.
But it is even easier not having to do so! (And if it is so easy, why
didn't you do it?) Furthermore your implementation suggested, that ext4
was supported, which was not, because it was not in the initrd. With my
patch any filesystem driver in the initrd can be used.
* I export cdrom_options from cdrom-detect, so that the 'loop' option
is used later on, when necessary (in 40cdrom and 41cdset).
* Changed the log message, if the cd mount fails, to give
$loopdev${device#/loop} as device.
* Changed the log message from 'removable devices' to
'CD/maybe-usb-floppy devices', because USB sticks are commonly referred
to as removable devices as well, but not included in the devices here.
* Changed an if statement, so that the CD/USB-HDD/isohybrid detection
is executed, when loop-mounting the ISO fails. A 'break' prevents this,
when the loop-mounting was successful.
* Unmount /loop before trying to mount an ISO to it, because it is
possible that the user selects cdrom-detect a second time, which would
lead to an error, if loop was already mounted in the first time.
* Changed the if-clause to determine which devices to look for to a
for loop over $dev_given "usb-partition" "cd" "partition", which is
better for two reasons:
- The most likely devices (given device, usb devices) are looked at
first.
- If a $LOOPDEV is given, but no such device can be mounted, then
the other devices are tried as well.
* Added log messages with $device and $fstype, before a device is
mounted, since this information is most useful, if the mount fails, so a
message after successful mounting is not enough.
* Do the cd_mountable determination in cdrom-detect and export the
value via debconf.
* Added source code comments, harmonised formatting and removed
trailing whitespaces.
* Added cdrom-detect/cdrom_options (for load-install-cd, 40cdrom,
41cdsest), cdrom-detect/cdrom_loopdev (for 40cdrom, 41cdset) and
cdrom-detect/cdrom_mountable (for 40cdrom) to the cdrom-detect.templates
file.
That being said, I think you agree, that this patch is ready to be
included in the Debian installer, because no known problems exist, and
it is tested quite thoroughly.
Best regards,
Andreas
Hi, the patch for this bug affects the following packages: * apt-setup * cdrom-detect * hw-detect (check-missing-firmware) * mountmedia Since among the maintainers of all these packages is Christian Perrier, I'm sending this to you. A short summary: The patch created by Ian Bruce and myself adds ISO loopback support for the Debian installer using a new boot parameter, to be used as follows: loopmount=[DEVICE:]ISO Here ISO specifies the path to the ISO, i.e. /ISO/debian-7.1.0-amd64-CD-1.iso. DEVICE is for the name or UUID of the device, on which the ISO is located. Giving this is optional and if it is not given, all devices are searched for the ISO. If the boot parameter is not given (or no ISO could be found), everything works essentially as before. For the patch to work, the loop-module is needed in the initrd, so I suggest to make it a dependency of cdrom-detect. I furthermore highly recommend to make the ext2/ext3/ext4, ntfs and udf modules dependencies of cdrom-detect, since these are the most common filesystems and thus being able to loopmount from them would be good. (Fat is somewhat outdated.) The patch makes it possible to boot using USB-HDD from any filesystems with drivers in the initrd. The patch also cleans up the somewhat messy workaround for bug #608201. For ease of use, I propose to add a loopback.cfg similar to the the attached one to the ISO in the folder /boot/grub/. This would allow to easily mount the ISO using grub2. I tested loopmounting with the following ISOs. (They were modified with the attached Apply_Patches.sh.) * debian-7.1.0-amd64-netinst.iso * debian-7.1.0-i386-netinst.iso * debian-7.1.0-amd64-CD-1.iso * debian-7.1.0-amd64-CD-2.iso (+) * debian-7.1.0-amd64-CD-3.iso (+) * debian-7.1.0-amd64-DVD-1.iso * debian-7.1.0-amd64-DVD-2.iso (+) (+): Not modified, just copied to the folder of the first ISO in the set. This worked without problems. To make sure all other boot mechanisms still work, I tested them for the patched debian-7.1.0-amd64-CD-1.iso: * Isohybrid: OK * USB-HDD: OK * CD: I can't open the CD drive with the button the on the drive. I have to change to another TTY and use 'eject'. (This problem exists also with the original ISO image, see #726137.) Since the patch works well and does no harm, I ask you to include it in git. Changes since the last patch: * finish.d/10apt-cdrom-setup: Remove the whole 'deb file:' line. With the last patch, an empty line remained. * The script mountmedia uses 'mount -tauto', but this only tries to mount as fat and nothing else, so I changed this to detect the filesystem with blkid. Now firmware can be loaded from a harddrive/USB with any of the filesystems, for which drivers are in the initrd. * Removed $FATFS from cdrom-detect and instead let the filesystem be automatically detected using blkid. With this it is possible to use USB-HDD for all filesystems, for which drivers are in the initrd. * Removed iso-hybrid and usb-hdd templates, since it works well without. Best regards, Andreas
I'm not done with this yet. I'm working on a more general patch with new features, which will be forthcoming shortly. I would ask that nothing major be done until that is ready. The current version is certainly ready for testing, although Andreas already seems to have done so extensively. "loopback" is the name of a virtual network device; the proper term in this context is in fact "loopmount", hence the name of the boot parameter. Relative to the root directory of some block device, of course. It specifies the filesystem label or UUID, as found in /dev/disk/by-label/ or /dev/disk/by-uuid/, or returned by /sbin/blkid. At least in my version of the patch, if it is not specified, then partitions and CD/DVD drives are searched, but not other devices. As far as I know, if the "loopmount" parameter is not specified, then everything works EXACTLY as before (by design). It appears that the ext4 module would be sufficient to also mount ext2/ext3, whereas the reverse would not be true. https://ext4.wiki.kernel.org/index.php/Frequently_Asked_Questions#What_is_the_difference_between_ext2.2C_ext3.2C_and_ext4.3F https://ext4.wiki.kernel.org/index.php/Frequently_Asked_Questions#Can_I_mount_existing_Ext3_as_Ext4.3F_And_vice_versa.3F_Similarly_from_Ext2_to_Ext4_and_its_reverse.3F https://ext4.wiki.kernel.org/index.php/Ext4_Howto#Compatibility There might be some value in including NTFS, so that you could loop-mount from Windoze partitions. I don't know what usage UDF gets (besides DVDs) that would justify including it in the initrd. VFAT is by no means outdated, since it is used on almost every USB flashdrive in existence. You might expect that it would eventually be replaced for this purpose by F2FS, but that certainly hasn't happened yet. Anyway, it's already in the initrd. Actually, from any supported block device with a supported filesystem. It appears that most standard PATA/SATA/SCSI/CDROM/USB drivers are already included in the initrd, so the only thing that would need to be added is "loop.ko" and a few filesystem drivers. A proper fix would be for all ISO images to be treated the same, regardless of whether they were contained in a CD, a disk partition, or a loop-mounted file. I'm not sure why this shouldn't be possible, but it's not the issue we're mainly concerned with at the moment. I can supply similar config files for Syslinux, allowing the use of the original boot menus with a loop-mounted ISO. As I have suggested previously, you don't actually have to modify the ISOs for testing; you can just patch an external copy of the initrd and boot with that. That way, the official MD5 and SHA256 hashes still verify. Thanks for testing all that. I think I also ran into this at some point. I think this will be a highly useful feature, and may become the primary method of booting Debian. (Does anybody really bother burning install CDs anymore?) However, as I said above, I'm still working on some improvements, so it's not quite done yet. That's what "mount -t auto" is supposed to do. From the manual page: If no -t option is given, or if the auto type is specified, mount will try to guess the desired type. Mount uses the blkid library for guessing the filesystem type; if that does not turn up anything that looks familiar, mount will try to read the file /etc/filesystems, or, if that does not exist, /proc/filesystems. So I don't know why using /sbin/blkid directly would make any difference. Maybe there's some peculiarity about the Busybox version of "mount". For the reasons given above, I'm not very confident about this. I think it's better to specify the relevant filesystem types directly. "mount -t vfat,ext4,iso9660" actually works; I suggest we just add ntfs or whatever seems desirable to that list. I concur. That logic is better handled in another way.
[...] [...] The ext2 and ext3 modules are also being removed from Debian kernel packages, starting with Linux 3.11. Ben.
Hi, I'm curious, what features do you want to add? Yes, loopback is the lo network interface, but for some reason I don't understand, GRUB uses the term loopback for booting from an ISO: https://www.gnu.org/software/grub/manual/grub.html#Loopback-booting This is why I constantly get confused between loopback and (the more descriptive) loopmount. In my latest patch, some changes are effective, even if loopmount is not used, e.g.: * I cleared up the workaround for bug #608201, so that in the future it should not be necessary to change apt-cdrom-setup, if one adds a new booting method. * Since I included more filesystem drivers in the initrd, I changed the code so that USB-HDD also works from filesystems other than FAT. * I also exported the boot options from cdrom-detect and imported them in several places, instead of determining the again. This should not have any effect, if loopmount is not used, but if it is, the 'loop' option is used. If in the future some changes are made with the boot options in cdrom-detect, they will be respected by apt-cdrom-setup. I just tested loopmounting from an ext2 filesystem, while only the ext4 filesystem driver was in the initrd: It worked, blkid identified it as ext4 and mounting as such only gave an info message: kernel: [ 13.170123] EXT4-fs (sdb1): mounted filesystem without journal. Opts: (null) And according to Ben there will be no ext2/ext3 modules in the kernel starting with 3.11. So just add the ext4 module. I have to admit, that I didn't know more than that a month ago. FAT32 is no modern filesystem, since it doesn't support files larger than 4 GB, e.g. the debian-7.1.0-amd64-DVD-2.iso with 4.7 GB. The reason for it to be still widespread is, that it is supported by (nearly?) all operating systems. While F2FS might be modern and optimized for flash drives, it is only supported in Linux and thus cannot replace FAT32. UDF on the other hand is used for DVDs and most operating systems can read (and even write to) it. Furthermore it supports large files and thus is the most probable replacement for FAT32. For further explanation see: http://duncanlock.net/blog/2013/05/13/using-udf-as-an-improved-filesystem-for-usb-flash-drives/ Already, some USB sticks sold are formatted with UDF. So I recommend to add UDF support now, although it will probably not be relevant for the broad audience until several years have passed by. In fact, with my patch, they are treated practically the same, with the one exception of CD set support, which is only available for actual disks in a drive and loopmounted ISOs. It is probably possible to extend this support to isohybrid and USB-HDD, but it is assumed, that one does not want to use multiple USB sticks to install Debian. That would be great. The problem is, that I also have to modify the apt-cdrom-setup.udeb, that is not in the initrd, but gets loaded afterwards from /pool/main/a/apt-setup. Perhaps you could drop this info at bug #726137 for confirmation of the problem. I agree fully. I would be glad for any improvements, though it already works quite well. I also don't know why '-tauto' does not work. With 'mount $1 -tauto $MNT' it gives a lot of the following warnings: UDF-fs: warning (device sdc1): udf_fill_super: No partition found (1) FAT-fs (sdc1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! And it does not actually load the firmware. With 'mount -t $fs_type $1 $MNT' and $fs_type determined with blkid it just works (see attached syslogs). Sorry, I must have missed your reasoning for why specifying the supported filesystems directly is better. I think you only wrote it would be 'easy enough' to give the filesystems by hand. But, as I said, it is easier not having to change the source code for every driver added to or removed from the initrd. And I can see no drawbacks in that. Furthermore, in my opinion, there should be as little as possible differences between the used kernel (Linux/kFreeBSD/Hurd), but these have different names for the filesystems. So I think it is best to let blkid determine the right filesystem and leave the rest independent of the used kernel. Best regards, Andreas
Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com): ../... Ian later asked for more time as he's working on more general fixes. So, should I wait or shouldn't I?
Hi, I don't know why, but I didn't receive your email. I just read it on bugs.debian.org. Using the patched ISOs I noticed an error in the patch: In 41cdset $filename was incorrectly used instead of $ISOname. Therefore too many ISOs were considered as part of the set. The corrected patch is attached. I also removed the unused $ISOend variable. Furthermore I noticed that there are more official ISOs than I had in mind, when I wrote the cdset-patch: If the first ISO is called debian-7.2.0-amd64-CD-1.iso, 41cdset looks for other ISOs with names debian-7.2.0-amd64-CD-*[^1].*[.]iso and thus finds e.g. debian-7.2.0-amd64-CD-2.iso. But there exists also the debian-update-7.2.0-amd64-CD-1.iso, which would not be found. And if you started with a debian-7.2.0-kfreebsd-amd64-CD-1.iso, 41cdset would not find debian-7.2.0-amd64-CD-2.iso. Possible ways to treat this problem: * One could just adapt the script for these cases, but I think that the detailed structure of the ISO names should not be hardcoded. * Alternatively one could just check every *.iso, but this has the disadvantage, that if you have many other ISOs in the same folder, you would have to click through many messages in the installer, stating that this or that ISO is no Debian ISO. * I think the best way would be to mention in the documentation what ISOs 41cdset looks for. Any ISO could easily be renamed to match the assumptions. I have also tried to apply the patch to the kfreebsd ISO to check, whether it works there as well, but I have been unable to extract the initrd. It seems not to be packed as gzipped cpio archive. I haven't heard from Ian since his last mail on 13th October and I have no clue what he has in mind as a more general solution. Perhaps he has been busy otherwise. In general I think, that it would be good to include the patch now and add any additional improvements later. Best regards, Andreas
Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com): I re-added this to my TODO list. I'm just leaving a few more days as an opportunity for Ian to react...
Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com): I just committed the changes to apt-setup. I will probably soon upload the package with them . PLEASE TEST AS SOON AS POSSIBLE.
Hi, thanks for committing the patch to apt-setup [1]. These changes have to be accompanied by the appropriate changes in cdrom-detect (and vice versa), as otherwise the installer will not work anymore. Probably this should be ensured with a versioned dependency. The changes in hw-detect and mountmedia are necessary for loading non-free firmware from filesystems different from FAT. So please commit the changes to the other packages as well. Best regards, Andreas 1: http://anonscm.debian.org/gitweb/?p=d-i/apt-setup.git;a=commit;h=baf5e4aea1d15b0c8a04d9f8c2a186e76c26bb18
Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com): The problem is that, unless I'm mistaken, patches were not sent as patches against the relevant packages, but as patches against the ISO images. So, it makes it fairly hard, when a bit far from the context, to isolate what belongs to which package. The fact that many versions of patches were sent to bug reports doesn't make it easier, too..:-) So, could you isolate a patch that could apply cleanly to cdrom-detect package tree....or point me to it? I think I managed to apply the right changes to apt-setup, but that probably need to be checked, too. TIA,
Hi, I just created patches against cdrom-detect-1.46, hw-detect-1.98 and mountmedia-0.23. I hope that helps. For the patch to work, the loop-module is needed in the initrd, so I suggest to make it a dependency of cdrom-detect. I furthermore highly recommend to make the ext4, ntfs and udf modules dependencies of cdrom-detect, since these are the most common filesystems and thus being able to loopmount from them would be good. Best regards, Andreas
clone 724931 -1 -2 -3 reassign -1 cdrom-detect reassign -2 hw-detect reassign -3 mountmedia thanks Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com): should be added there. The same stands here, but I suspect all these modules are already in D-I kernel image packages. Needs to be carefully checked anyway.
Hi, Why do you think this wouldn't work? The udebs for these modules are on the D-I ISO (pool/main/l/linux), but not installed in the initrd (in <initrd>/lib/modules/3.*-amd64/kernel/fs/), where they are needed. What do you think is the best way to get these modules into the initrd? Best regards, Andreas
Hi,
I have researched how to include the needed modules in the initrd.
If I understand it correctly, it would be possible to add the modules as
dependencies of cdrom-detect.
But that would not be feasible, because the kernel modules have the
version of the kernel in their name and therefore the dependencies would
have to be updated every time a new kernel is released.
Instead I think the best way is to change the config files for the
debian-installer [1] that determine what udebs are included in which
initrd. They reside in installer/build/pkg-lists.
For the modules needed for loopmount I attached a patch against the
debian-installer. In this patch I just added the loop-, ext4-, ntfs- and
udf-modules to all config files that include cdrom-detect. This comes
close to making them dependencies of cdrom-detect, but it is much
easier, since in these config files one can use the variable
${kernel:Version} and doesn't have to update that manually.
Please include that patch in the debian-installer and then release the
new versions of cdrom-detect, apt-setup, hw-detect and mountmedia.
Best regards,
Andreas
1: http://anonscm.debian.org/gitweb/?p=d-i/debian-installer.git;a=tree
clone 724931 -1 reassign -1 debian-installer retitle -1 Please add needed modules for ISO loopmount to work in D-I thanks Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com): Thanks, Andreas. I therefore clone this bug report and assign it to debian-installer itself, so that the needed actions are ataken on time there, too, if we want this feature. I was already suspecting that having Cyril Brulebois (CC'ed) look specifically at all this is really needed as this feature is indeed quite invasive. So, well, let's get his input. I have the feeling that, given the time you invested in this, we should have some cinfidence that your proposal both makes sense....and will not break D-I.....but someone less optimistic and confident than me might want to have a closer look...:-)
Hi, this is now the second time that I did not get your mail. If you have any idea what could cause that, please let me know, because it is not really practical to always check bugs.debian.org. On Topic: It's always better, if more people look at a patch, so I'd be grateful for KiBi's thoughts on it, especially concerning the effects for kFreeBSD, since I could not test these, because I did not manage to unpack the initrd from the kFreeBSD ISO. I don't know really what you mean by invasive, but I assume you mean, that the installer could break, if apt-setup and cdrom-detect are not both updated at the same time. So let me clarify this a bit: The functionality to loopmount is not invasive at all, since the necessary code for this is only executed, if the boot parameter 'loopmount=' is given. What makes this patch more invasive is the fact that it cleans up the somewhat messy workaround for bug #608201. Up to now the situation was, that for every non-CD boot method (usb-hdd/isohybrid) a template was exported from cdrom-detect and imported in apt-setup to check there, whether cdset-detection should automatically (u)mount \cdrom. I moved the code to determine this to cdrom-detect and only exported the result (cdrom_mountable) to apt-setup, so that in the future one does not have to change apt-setup (changes are needed only for cdset support), if one adds a new boot method. Thus if not both patches (cdrom-detect and apt-setup) are applied at the same time, apt-setup will fail, because it looks for a template that does not exist. Best regards, Andreas
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2013-12-02): Gmail, spam folder? It's a royal pain anyway since it believes it knows better than users how many mails one should get… Sorry, will take a while for me to get to it, finalizing the move from ravel to dillon, and setting up a few things on my side to help me work on d-i. Thanks for bearing with me… Mraw, KiBi.
Strangely, these two mails did not end up in the spam folder (nor any other folder), they just seem to have vanishes somewhere on the way. But I get all(?) other mails from Christian (and everybody else). That's no problem for me, since I'm personally using the patch already. Best regards, Andreas
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2013-12-02): The easiest way to get feedback concerning kFreeBSD is contacting debian-bsd@. Given the apt-cdrom regression we're hitting (#740673), I don't feel like shipping this amount of additional modifications in jessie alpha 1 images; on the other hand, not uploading what's in apt-setup's master currently would mean not using updated l10n material, which isn't too nice. I'm tempted to branch "iso-loopback" from current master, so that it's easily found afterwards, and to revert the ISO loopback bits for now. Christian, can you please confirm that it makes sense on an l10n point of view? Mraw, KiBi.
Hi,
use for testing regressions of this patch, but on the other hand using
loopmount to install Debian *works*.
So you might want to reconsider applying this patch now, as it seems to
be currently the only way to install Debian jessie directly. ;)
Yesterday I made a new installation using the patched:
Debian GNU/Linux testing "Jessie" - Official Snapshot amd64 CD Binary-1
20140203-06:21
I'm only half-joking, it really works, otherwise I would have reported a
bug.
I fixed a few things in my previous patch (see attachments):
* Fix typo in finish-install.d/10apt-cdrom-setup: cdrom/$j -> cdrom$j
* Always test for other CDs, except if it is a netinst CD.
Otherwise one can't use CD-2 together with
debian-testing-amd64-kde-CD-1.iso of type full_cd/single. (Maybe I
didn't get how this is supposed to work?)
* Mount further ISOs if at least four parts (seperated by -) of the
name are the same. Example:
a) debian-testing-amd64-CD-1.iso
b) debian-testing-amd64-kde-CD-1.iso
c) debian-testing-amd64-CD-2.iso
d) debian-testing-amd64-CD-2-local-changes.iso
e) debian-testing-i386-CD-2.iso
a) and b) would load c) or d), but not e).
* Fix crash, if the filesystem is not know (e.g. unpartitioned).
Pease add the attached patches on top of current git, even if you don't
want to apply the patch now and instead move it to another branch.
Best regards,
Andreas
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-08): Sorry, but until it's been pushed to the masses, I can't blindly trust that. Many moving parts already, not going to add a critical one when we already have a (known) regression in apt-setup/apt-cdrom. Especially not a few days before a release. Mraw, KiBi.
Hi, someone other than me. I just wanted to express my amazement that loopmount works, while the usual way doesn't. I looked more closely and found that commenting out the following in 40cdrom fixes the problem: # Allow apt-cdrom to manage mounting/unmounting CDs in /target if [ "$cd_mountable" ]; then rm -f $ROOT/etc/apt/apt.conf.d/00NoMountCDROM $logoutput umount /target/media/cdrom* || true $logoutput umount /cdrom || true fi At least I managed to install Debian jessie in a Virtual Machine. So if there are no compelling reasons to have this (Note: I don't know, what this is supposed to do.), I would suggest removing this as a fix for Bug #740673. Best regards, Andreas
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-08): Err, no. We had something that worked well enough for a while, it broke due to a change in apt-cdrom, which was confirmed to be unintentional. So let's fix that for now, and *maybe* rethink stuff after that. I'm not goint to drop random lines, especially not based on “I don't know what this is supposed to do”… Mraw, KiBi.
Cyril Brulebois <kibi@debian.org> (2014-03-08): FWIW I've done that in apt-setup, which is awaiting another patch before an upload (~ in an hour). Mraw, KiBi.
Hi, Given that Bug #740673 is fixed and jessie alpha 1 is released, can you review and upload these patches now? Best regards, Andreas
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-24): sooner (probably the contrary, in fact). Annoyed, KiBi.
Hi all, I really don't want to be pushy (I appreciate that you all spend your free time making Debian a better distribution) but am just curious about the chances to get all the bits and pieces mentioned in this bug report merged before Jessie gets frozen (so that one can take an official Debian Jessie netinst ISO and drop it onto a loopback-boot enabled USB drive next to installers for Ubuntu and other Linux distros that already support it, and it will "just work"). I guess I am not the only person who likes to have one USB drive with all their Linux install and live media present all the time, and currently this means for me that whenever a new d-i comes out (that I want to use), I have to collect patches from bug reports and build my own d-i (and test it on a VM to avoid regressions) so that I can loopback boot it from my USB drive. As this means a considerable amount of effort (multiplied by the number of people who are doing this all for themselves), if the odds are bad to get it into official Jessie installers, perhaps we (who are regularly or occasionally monitoring this bug) can coordinate our efforts and provide an unofficial Debian Jessie netinstaller image somewhere (no idea about how hosting stuff works within the Debian project, but for me even some prebuilt and tested .iso on a Dropbox or similar file hoster would be good enough - YMMV). Just my 2 cents, Michael
El 08/03/14 a las 18:11, Andreas Cadhalpun escribió: Can you please explain why you are using: get_fstype () function which it's based on blkid instead of just using the old method of relying in auto function from the kernel itself? This is used in both cdrom-detect.patch and mountmedia.patch. Please, be aware, that I'm not telling you your approach is incorrect. It seems we are lacking the explanation or rationale on why you made that decision in order to evaluate that change in a fair manner. Thank you very much! adrian15
Hi adrian15, type found with blkid works fine. Doing some archeology reveals the relevant error messages from syslog: With 'mount -tauto': kernel: [ 109.257009] UDF-fs: warning (device sda9): udf_fill_super: No partition found (1) kernel: [ 109.378443] FAT-fs (sda9): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive! main-menu[550]: (process:4539): mount: mounting /dev/sda9 on /media failed: Invalid argument With blkid: [ 80.943104] EXT4-fs (sda9): mounted filesystem with ordered data mode. Opts: (null) These happened during check_missing_firmware, i.e. this comes from mountmedia. I think that convinced me not to use 'mount -t auto' in cdrom-detect. However, that was two years ago. Much could have changed in the meantime. Best regards, Andreas
El 18/08/15 a las 19:28, Andreas Cadhalpun escribió: Ok, I will try to reproduce it and see if it still happens. I'm in debconf15 trying to push forward some improvements that I'm interested of as this one. Having around real people to speed things helps but don't raise your expectations too early. adrian15
Dear Customer, We can not deliver your parcel arrived at February 11. Please check the attachment for details! Thanks, Michael Beasley, UPS Senior Support Manager.
Dear Customer, We can not deliver your parcel arrived at March 11. You can find more details in this e-mail attachment! With sincere appreciation, Walter Whitehead, UPS Mail Delivery Clerk.
Hi, Wanted to use a multiboot USB key with Debian 9 and found this wish. I second this. Maybe for Debian 10? :) A workaround is to use the hd-media initrd, possibly with d-i "shared" options, which is kind of neat for a multiboot (but beware that modules you load from ISOs have to be compatible with you running kernel). Best,
For what it's worth, I have also been bitten by the missing support for loop.ko in the netinst iso many times (since wheezy). Ultimately, support for grub2 iso-boot in some form is exactly what I hope for. Thanks, guys! Best, Julian
Hi! It's the end of 2020 and we still need initrd from hd-media if we wont to loopback-mount ISO file during install. Why not to include iso-scan into netinst and other cd images? Best regards, Aleksei Shargalin