Dear Maintainer,

== Introduction ==

  In Debconf 14 Debian installer and CD BoF talk which it's archived here: http
://meetings-archive.debian.net/pub/debian-
meetings/2014/debconf14/webm/Debian_installer_and_CD_BoF.webm someone in the
public suggested to concatenate 1 GB size images (Skip to time: 28:58 in the
video ) so that you can use them in 4 GB usbs. This will also mean that you
won't have to maintain 1 GB, 2 GB, 4 GB and 8 GB size installer images but
only: 1 GB, 1 GB, 1 GB and 1 GB with most no repeated packages.

So you would only need to do something like:

cat debian-7.6.0-i386-USB-1.iso debian-7.6.0-i386-USB-2.iso
debian-7.6.0-i386-USB-3.iso debian-7.6.0-i386-USB-4.iso > /dev/sdc

where sdc is your hard disk device.

Steve McIntyre replied that was not possible because filesystems don't work
that way. Another reply from the same person that initially suggested the idea
was: "But we can fix it". Finally Steve turned down the idea.

== Overview ==

After thinking about it, although I'm not sure 100%, I think it could work.

The idea is, as you know, how big your initial image size you can mount the
next image.

How would be do that ? Just by offsetting the mount of the partition. Just as
qemu does:

https://wiki.archlinux.org/index.php/QEMU#Mounting_a_partition_inside_a_raw_disk_image

So the suggested:
  mount -o loop,offset=32256 disk_image mountpoint
would have to be rewritten as:

  mount -t auto -o offset=966367673856 /dev/sdc /mnt/apt-2

where the offset is based on an hypotetical 900 MB image as is:
(900*1024*1024*1024)+32256 .


== Possible problem ==

The only doubt I have is that when you initially dd the first 1 GB image you
are defining a 1 GB partition table.

So I'm not sure if this 1 GB partition table will disable the kernel to read
data on usb hard drive beyond the 1 GB limit or not.

I will try to make some tests in the next days to prove if the minimal
functionality needed actually works or if it would need hacks on how Gnu/Linux
kernel reads devices bigger than its partition table suggested size.

== Implementation idea ==

(Loop till you do not find a new valid dd'ed hard disk partition) :Try to mount
the next dd'ed hard disk first partition in the disk. Add the packages
directory found on that partition to /etc/apt/sources.list (or whatever you use
in Debian Installer).

Run apt-get update to add these packages to the available ones.

== Similar bugs ==

This is a very similar bug to: #617853 . The only difference is that in my
implementation the debian-installer sorts out the next images while in the
other bug I suppose the tool is meant to read inside the image, extract its
contents and copy the packages in the usb. Feel free to relate both bugs if
it's worth. Actually Steve McIntyre describes this bug in his talk.

Well, this works if you use mount command from util-linux package.
busybox's mount does not implement offset feature, so it either ignores
it or complains about it.

== Getting Debian images ==

I have downloaded
   * debian-7.6.0-i386-CD-1.iso
   * debian-7.6.0-i386-CD-2.iso
for doing my tests.

== Getting to know Debian CD 1 size ==

# fdisk -l debian-7.6.0-i386-CD-1.iso
GNU Fdisk 1.2.5


Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.


Disk
/mnt/ssd/adrian_sata/conocimiento/distribuciones_isos/debian-installer/debian-7.6.0-i386-CD-1.iso:
679 MB, 679477248 bytes
64 heads, 32 sectors/track, 648 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes


                      Device Boot      Start         End      Blocks
Id  System
/mnt/ssd/adrian_sata/conocimiento/distribuciones_isos/debian-installer/debian-7.6.0-i386-CD-1.iso1
   *           1         648      663536   17  Hidd HPFS/NTFS

So size, in bytes, is: 679477248 .

== Additional notes ==

Although I am booting from a cdrom, the same procedure can be replicated
if you boot from a usb hard disk.

== Virtual machine description ==

Cdrom 1: Debian Live CD
Cdrom 2: Debian CD 1
Cdrom 3: Debian CD 2

Hard disk 1: 8 GB . Blank disk for a new Debian installation.
Hard disk 2: 4 GB . Blank disk. USB that has concatenated images.

== USB preparation ==

I boot the virtual machine from: Cdrom 1: Debian Live CD . Let's see how
GNU/Linux detects the disks:

/dev/sr0: Cdrom 1: Debian Live CD
/dev/sr1: Cdrom 2: Debian CD 1
/dev/sr2: Cdrom 3: Debian CD 2

/dev/sda: Hard disk 1: 8 GB . Blank disk for a new Debian installation.
/dev/sdb: Hard disk 2: 4 GB . Blank disk. USB that has concatenated images.

Now we are going to concatenate the disks. Be aware that final user will
not concatenate cdrom devices but iso files directly. I use cdrom device
because, as I use Virtualbox, it makes it easier for me.

cat /dev/sr1 /dev/sr2 > /dev/sdb
sync

== Testing the idea ==

Now I change a bit cdroms so that I boot from Debian CD 1.
Cdrom 1: Debian CD 1
Cdrom 2: Debian CD 2

I boot the virtual machine from: Cdrom 1:Debian CD 1.

So, first of all install Debian (making sure that Standard tools from
system are installed although I'm not sure if they are absolutely needed
or not) into /dev/sda hard disk up to the point where you are asked if
you want to install bootloader or not. Don't ask the question. Instead
do: ALT+F2 .

This is going to open a shell where sda is mounted in /target and where,
let's remember, sdb is: Hard disk 2: 4 GB . Blank disk. USB that has
concatenated images.

mkdir /target/mnt/apt2
chroot /target /bin/mount /dev/sdb /mnt/apt2 -o offset=679477248 -t iso9660

Now if you run:

more /target/mnt/apt2/README.txt

you will see that you can read Debian CD 2 disk while it's have been
concatenated to the USB !!!

== Target based Implementation ==

This is the implementation you have seen before. You need to have a
basic installation of Debian into the hard disk. That's not problem
because it is provided by Debian CD 1.

Once this basic installation (included the Grub installation to disk)
has finished the user is prompted:

Do you want to search packages into appended images in your usb?

If the user answer yes, then the offset mount is tested, packages path
is added to /etc/apt/sources.list (or whatever), apt-get update is run,
and you somehow let the user add more packages in the installation if he
wants to.

These packages are found in second cdrom.

== Busybox based Implementation ==

This implementation would involve hacking busybox's mount command so
that it supports offset parametre.

Then asking for the user which packages to install would be done once,
because the next Debian CDs would have been found at first stages of the
installation.

== Using Disk 2 from USB everyday ==

Somehow a tool would be needed so that the user can mount its Debian CD
2 from its USB as easier as possible (Maybe a noauto entry in fstab).
This way when running: apt-get update the files will be finally found in
/mnt/apt2 (or whatever) .

== Special tool for making USBs in Windows ==

For Windows people it would be nice to have a GUI where you can select
(always with CD 1 as the first one) all the cds you want and then
concatenate them into a USB device.

== Padding when generating iso images ==

As you might suspect you need to know the iso size in advance so that
this works in a reasonable way.

I think the idea is to embed the disk size in a file somewhere inside of
each one of the disks.

How do you manage to know the final size before generating the disk? By
padding it.

So, if you want your disk to be 900 MB you will build it to have 890 MB,
then you will pad 10 MB at its end so that it is 900 MB.
If the build happens to have: 892 MB, then you only need to add 8 MB. Of
course, the padding needs to be done in bytes not in megabytes.

== Feedback from Debian-CD / Debian-Installer people ==

Do you think it's something that it's easy to implement ?
Will you need help even if it's easy to implement ?

I hope that this new hack (I think nobody has thought about using
mount's offset option in a such a way) lets Debian shrink its Debian CD
archive size.


El 27/08/14 03:08, Adrian Gibanel Lopez escribió:

adrian15

Hi,

i am watching this bug as upstream developer of xorriso,
which packs up Debian ISOs under control of debian-cd.

So i can mainly comment about early boot stages and mounting
of ISO 9660, not much about Debian installer issues which
arise after or before these steps.


From your initial report i got the impression that you
wanted to let the user choose any of the ISOs for booting.
There are descriptions in the web for boot media which
contain multiple ISOs.
Google "boot multiple iso" found:
http://www.pendrivelinux.com/boot-multiple-iso-from-usb-via-grub2-using-linux/

But now it seems that you want to install Debian from
the first ISO onto hard disk and then only want to use the
other ISOs as alternative sources of packages.

So i wonder why you do not dd the first ISO onto the USB stick
(which will install an MBR and declare partition 1). Then you
could use a partition editor to create partition 2 which covers
the remaining blocks of the USB stick.

There you could create a mountable filesystem (e.g. FAT)
and store the other ISOs for loop-mounting.

Originally "offset" is a feature of losetup(8) which creates
a loop device for mount if option "loop" is given.
The Archlinux example shows option "loop".
Did you try busybox mount with "loop" ?

This tells the size of the MBR partition which was wrapped
around the Debian image by xorriso. It may or may not give the
size of the ISO image data file.

Since you concatenate the image files and not the partition
contents, you should rather inquire the file size. Like:

  ls -l debian-7.6.0-i386-CD-1.iso

In the case of a FAT filesystem in /dev/sdb2, which holds
the other ISOs you would first mount the FAT

  mkdir /target/mnt/usb_fat
  chroot /target /bin/mount /dev/sdb2 /mnt/usb_fat

and then mount the desired ISO from inside the FAT:

  mkdir /target/mnt/apt2
  chroot /target /bin/mount -o loop /mnt/usb_fat/apt2.iso /mnt/apt2

Note that no size information is needed to find the ISO but rather
its file name inside the FAT filesystem. I assume that the name of
debian-7.6.0-i386-CD-2.iso copy in the FAT is /apt2.iso.

One could also put the desired ISOs into a ext2/3/4 filesystem
instead of a FAT. This would allow the use of long filenames.
If they all are put into an ISO then they would be write-protected
for normal users.

Would probably not be necessary if the ISOs are presented as
data files of a mountable filesystem.

fstab could contain lines for /target/mnt/fat and /target/mnt/apt*.

The normal MS File Manager could populate the mounted
FAT filesystem.

Not if the ISOs are data files inside a filesystem.

I get the impression that you plan to create a kindof minimal
filesystem. My main objection is that suitable filesystems
already exist and are supported by a wide range of operating
systems and tools.

Outside my personal scope is the motivation for your proposal.
A Debian person might ask:
What are the use cases where it will be of help ?
What benefits would it have for Debian's ISO production and
download sites ?


Have a nice day :)

Thomas

I was the person at that DebConf session who suggested concatenatable
images.  And in particular, if you want to handle media of power-of-two
sizes (1GB, 2GB, 4GB, 8GB, 16GB), you can provide concatenatable images
of power-of-two sizes after the first: 1GB, 1GB, 2GB, 4GB, 8GB.  More
generally, if you want to support sizes a, b, c, d, e, ..., then provide
images of size a, b-a, c-b, d-c, e-d, ...

Offsets were exactly what I had in mind when I suggested that this
seemed easily fixable: you can tell how big the first image is supposed
to be, so just look right after that image for another image, and repeat
until you don't find another image.

Thanks for working on this!

- Josh Triplett

Josh Triplett <josh@joshtriplett.org> (2014-08-31):

I'd strongly suggest using "bts -m show $bugnumber" and reply-all; right
now your replies break threading (which is a minor annoyance), but more
importantly, they don't go to the previous participants to this bug
report (including the submitter).

Mraw,
KiBi.

El 28/08/14 11:35, Thomas Schmitt escribió:
I see.

Yes, but these tools, in theory, won't work perfect with Debian
Installer till #724931 and #759744 are fixed.

Yes, that's the main idea so that you do not have to release 1 GB, 2 GB
and 4 GB size images for each one of the usb media size you want to
provide to.

Yes, I have checked it and it works. I mean even if the ISO's partition
table is ~ 500 GB size initially when you dd. Live CD detected my hard
disk as a 4 GB size one.

And, yes, I could create a new partitions thanks to gparted. So instead
of using gparted from a live cd there could be a new option found at the
same installer: "Repartition free space to store extra ISO images".

The user copies the additional ISO files to the new partition from e.g.
Windows.

Then you would need some of the implementation about detecting new ISOs
from the partition directory and using them that it's already found at:
#724931 .

The other option would be providing a tool for Windows so that it did
the partitioning of the usb after dding the image to it. Not sure if
it's worth the effort.

Probably we should add a new bug suggesting the repartitioning of free
space from installer and close this one. Not sure about it. I personally
like the cat idea because is very new but at the same we are saving data
into an space that could be thought of being empty.

I mean, perhaps someone could imagine to reuse "free" space in the USB
for their own files while loosing their initial "hidden" extra ISOs.

I ask myself what's the opinion of debian-cd people on this subject.

Busybox has loop support but not offset support that I'm aware.
Losetup is not found when running a shell from the installer.
Yes, you are right. Although fdisk -l also shows this size somewhere.

Yes, actually, loop mount (without the offset feature) currently works
from the installer. So you won't need need the previous system
installation for mounting the isos.

Anyways this would need some of the implementation found in bug #724931 .

Yes, the thing is making sure that the path to the isos in the installer
are the same one as the installed system paths. You know like the
installer using /extraisos directory and then finding the usb partition
in the installed system like /media/UUID-1234-... .

I haven't checked it in #bug 724931 yet. They might not have sorted it
out in a proper way.

I agree.
You are right.

When someone want to use CD1, CD2 and CD3 from the same usb. E.g. giving
the usb to someone else who is offline.
I said above: You do not have to release 1 GB, 2 GB and 4 GB size images
for each one of the usb media size you want to provide to.

Thank your for input. Somehow I thought that my 4 GB hard disk test
after dding the ~ 500 GB image would have been detected as a 500 GB size
only hard disk.


adrian15

Hi,

Lack of loop mounting is quite some handicap.

I would rather call it "Claim free space on device as new partition".

We could make a specialized little program for that.
MBR is easy to manipulate. Device size can be inquired
linux-specifically by a ioctl() or portably by binary search.

Question is what to do with GPT. (Is Live-CD prepared for EFI
isohybrid at all ?)
The GPT specs prescribe that a backup table is to be stored
at the very end of the storage device. Debian amd64 has this
backup at the end of the ISO image. This already confuses
some partition editors.

To get a neat partitioning, GPT would either have to be deleted
or the backup table has to be moved. Moving demands patching of
fields in the backup and in the main GPT. This includes computation
of new CRCs for both.

Yes. This is a tricky part after first stages of booting.

One could combine cat with the creation of a new partition
or with expansion of the first partition up to the end of
the storage device.
(Again, GPT will impose extra effort.)

I assume they are waiting whether this discussion yields a
lean and stable proposal.

One could also inquire the size of the ISO image. :))
(Bytes 32768 + 81 to + 83 gives the number of blocks as little endian,
 + 84 to + 87 as big endian)

But actually you want to know where the next ISO will start.
With cat, this depends on the previous image file sizes.
(One could use dd seek= to put the ISOs at predefined
 block addresses regardless of the size of the previous ISOs.)

I now wonder how much effort it would be to merge Debian
installation ISOs into a single ISO filesystem.

The ISO part could be easily done.
But there are files inside the ISOs which would need to be
merged or otherwise adapted.

The benefit would be to get a normal Debian ISO with no
need for any new installer features.


Have a nice day :)

Thomas