- Package:
- fai-setup-storage
- Source:
- fai
- Submitter:
- Antoine Beaupre
- Date:
- 2024-02-17 18:15:03 UTC
- Severity:
- wishlist
I am trying to setup a fairly classic "RAID + LUKS + LVM"
configuration. The idea is to do the following steps:
1. format two disks identically, with a grub part, /boot and the rest
for the RAID/LUKS/LVM array
2. pick the third partition and create a RAID-1 array on it
3. luksFormat that RAID array
4. load that LUKS partition as a PV of a VG
5. create LVs in that VG
From the manpage, it's not immediately obvious how this can be
done. There's a small note in the cryptsetup example about:
With a working RAID+LVM configuration, an encryption layer can
be added between the RAID and LVM device layers by adding the
following cryptsetup configuration. In this case, the encrypted
device will be called 'crypt_format_md1' and will be used as the
underlying physical device (PV) in LVM.
disk_config cryptsetup
luks - /dev/md1 - -
Reading this, I thought I should use this configuration:
disk_config cryptsetup
luks - /dev/md1 - -
luks - /dev/md2 - -
disk_config lvm fstabkey:uuid
vg vg_nvme crypt_format_md1
vg_nvme-root / 30G ext4 rw
vg_nvme-swap swap 1G swap sw
On IRC, MrFai correctly pointed out (thanks!) that the device name
should not be used in the vg directive there, it should instead read
like:
vg vg_nvme md1
But that was far from obvious to me. I would suggest adding the
following example in the manual page to work around that ambiguity:
disk_config disk1
primary - 0- - -
disk_config disk2
primary - 0- - -
disk_config raid fstabkey:uuid
raid1 - disk1,disk2 - -
disk_config cryptsetup
luks - /dev/md0 - -
disk_config lvm fstabkey:uuid
vg vg_nvme md0
vg_nvme-root / 30G ext4 rw
vg_nvme-swap swap 1G swap sw
That way users have a clear, unambiguous example of how to setup the
full stack.
Note that I haven't tested the above configuration, I distilled it
down from another configuration that I know works, but that's more
specific to our situation:
# open questions
# --align=optimal?
# leave keys in /tmp/fai or specify passphrase?
# use sameas: to set all disk names earlier?
# bios_grub flag?
disk_config nvme0n1 disklabel:gpt bootable:2
# bios grub second stage
primary - 8MiB - -
# /boot
primary - 512MiB - -
# rest is RAID+LUKS+LVM
primary - 0- - -
disk_config nvme1n1 disklabel:gpt bootable:2
# same as above
primary - 8MiB - -
primary - 512MiB - -
primary - 0- - -
disk_config sda disklabel:gpt
primary - 0- - -
disk_config sdb disklabel:gpt
primary - 0- - -
disk_config raid fstabkey:uuid
raid1 /boot nvme0n1p2,nvme1n1p2 ext4 rw,noatime,errors=remount-ro
raid1 - nvme0n1p3,nvme1n1p3 - -
raid1 - sda1,sdb1 - -
# FAI defaults to -c aes-xts-plain64 -s 256
disk_config cryptsetup
luks - /dev/md1 - -
luks - /dev/md2 - -
disk_config lvm fstabkey:uuid
# previous convention was "vg_$hostname"
vg vg_nvme md1
vg_nvme-root / 30G ext4 rw
vg_nvme-swap swap 1G swap sw
vg vg_hdd md2
# HDD disks config intentionally left blank
I'll finally note that the device created by cryptsetup is actually
*not* called "crypt_format_md1" as documented in the manpage, but
rather "crypt_dev_md1". This should probably be fixed as well,
although it's unclear if that device name can be used anywhere in the
configuration. That fact could also be made clearer.
Thanks for this tool, it's pretty neat!
Your raid + luks example works when calling setup-storage, but I didn't manage to boot it. Can it work without an unencrypted /boot partition?
Your raid + luks example works when calling setup-storage, but I didn't manage to boot it. Can it work without an unencrypted /boot partition?
I'm using those configurations in production right now: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/disk-config In particular, this is a RAID+LUKS+LVM configuration with a cleartext /boot, on top of NVMe drives, on a bare-metal machine provided by Hetzner: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/disk-config/gnt-fsn-NVMe You should be able to reproduce this by booting a rescue system and running setup-storage, for example on a PX62-NVMe: https://www.hetzner.com/dedicated-rootserver/px62-nvme I've setup a handful of machines with this, but keep in mind we do some post-processing outside of FAI: I only use setup-storage, not the rest of FAI, for our installer. Those are the post-install scripts: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/post-scripts and they are fired from this Python/Fabric installer: https://gitweb.torproject.org/admin/tsa-misc.git/tree/install ... which is mostly a wrapper for: https://gitweb.torproject.org/admin/tsa-misc.git/tree/fabric_tpa/host.py#n507 ... itself a wrapper for grml-debootstrap, which is the bit that's firing the post-install scripts inside the chroot. That may be the bits you're missing for the machine to boot? Sorry, this is all a bit of a mess... I hope that helps! A.
I'm using those configurations in production right now: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/disk-config In particular, this is a RAID+LUKS+LVM configuration with a cleartext /boot, on top of NVMe drives, on a bare-metal machine provided by Hetzner: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/disk-config/gnt-fsn-NVMe You should be able to reproduce this by booting a rescue system and running setup-storage, for example on a PX62-NVMe: https://www.hetzner.com/dedicated-rootserver/px62-nvme I've setup a handful of machines with this, but keep in mind we do some post-processing outside of FAI: I only use setup-storage, not the rest of FAI, for our installer. Those are the post-install scripts: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/post-scripts and they are fired from this Python/Fabric installer: https://gitweb.torproject.org/admin/tsa-misc.git/tree/install ... which is mostly a wrapper for: https://gitweb.torproject.org/admin/tsa-misc.git/tree/fabric_tpa/host.py#n507 ... itself a wrapper for grml-debootstrap, which is the bit that's firing the post-install scripts inside the chroot. That may be the bits you're missing for the machine to boot? Sorry, this is all a bit of a mess... I hope that helps! A.
I'm using those configurations in production right now: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/disk-config In particular, this is a RAID+LUKS+LVM configuration with a cleartext /boot, on top of NVMe drives, on a bare-metal machine provided by Hetzner: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/disk-config/gnt-fsn-NVMe You should be able to reproduce this by booting a rescue system and running setup-storage, for example on a PX62-NVMe: https://www.hetzner.com/dedicated-rootserver/px62-nvme I've setup a handful of machines with this, but keep in mind we do some post-processing outside of FAI: I only use setup-storage, not the rest of FAI, for our installer. Those are the post-install scripts: https://gitweb.torproject.org/admin/tsa-misc.git/tree/installer/post-scripts and they are fired from this Python/Fabric installer: https://gitweb.torproject.org/admin/tsa-misc.git/tree/install ... which is mostly a wrapper for: https://gitweb.torproject.org/admin/tsa-misc.git/tree/fabric_tpa/host.py#n507 ... itself a wrapper for grml-debootstrap, which is the bit that's firing the post-install scripts inside the chroot. That may be the bits you're missing for the machine to boot? Sorry, this is all a bit of a mess... I hope that helps! A.
Hi, currently I think it's not possible to have raid+luks+lvm prepared with setup-storage. I didn't manage to prepare a working example.