Dear Maintainer, I have downloaded a debian 12.2.0 netinst and verified its integrity with sha512sum -c --ignore-missing SHA512SUMS and wrote it to a usb drive with dd bs=2048 if=debian-12.2.0-amd64-netinst.iso of=/dev/sda oflag=sync status=progress I also verified that the contents were copied correctly with these two commands: cmp -n `stat -c '%s' debian-12.2.0-amd64-netinst.iso` debian-12.2.0-amd64-netinst.iso /dev/sda head -c $(stat -c '%s' debian-12.2.0-amd64-netinst.iso) /dev/sda | sha512sum which returned the same checksum. I also safely removed the drive, unplugged it, and plugged it back in, and used those two commands again with the same output. (to make sure the kernel buffer wasn't in use) After booting the Installer from that drive, however, and navigating to the Integrity Check section of the installer from its menu, it said that ./boot/grub/efi.img failed the MD5 checksum verification. Your installation media or this file may have been corrupted. After this, I chose to quit the installer and check the medias integrity again with the earlier two commands. sha512sum showed a different checksum, and cmp showed debian-12.2.0-amd64-netinst.iso /dev/sda differ: byte 2303211, line 21165 showing that the content of the drive changed since it previously passed. I also tried entering the installer, and aborting the installation without using the Integrity Test section. I did this because I thought that maybe mounting the media in the Integrity Test section of the installer somehow changed the data. However, after booting, cmp and sha512sum still showed the same difference. I also tried using ddrescue and verified that there were 0 issues with the blocks on the media, used a different usb drive, and tried 3 laptops. In all of these cases it was always byte 2303211, line 21165 which changed. To my knowledge, I don't think this is supposed to happen. I instead expected an outcome where the media would stay the same throughout the installation process. Lastly, the computer I was using was able to get 10 passes in memtest without any errors. Have a nice day, Ram
Hi, Ram Reddy wrote: I'd say that corruption of ISO image or USB is quite unlikely. It rather looks like some change was made to the FAT filesystem in the EFI partition by the parts of the installer which ran in your experiments. (I expect EFI firmware not to manipulate that filesystem ...) The EFI partition of this particular ISO begins at byte 2291712. The alteration thus happened 11499 bytes inside that partition and its filesystem. In the original ISO i see a sequence of 0-bytes there. It would be interesting to check whether any errors get reported if the ISO is presented on a read-only CD or DVD instead of a writable USB stick. (To be absolutely read-only it would have to be CD-R, unformatted CD-RW, DVD+R, DVD-R, or unformatted DVD-RW. The others can be overwritten without burn program.) Explanation why the file in the read-only ISO filesystem is reported as being altered: The partition shares its block range with the ISO file /boot/grub/efi.img which is mentioned in the ISO file /md5sum.txt . So the integrity check detects the difference between recorded MD5 and the MD5 of the altered FAT filesystem image file /boot/grub/efi.img. Have a nice day :) Thomas
Hello, Thomas Schmitt wrote: Thank you for the suggestion for trying a CD / DVD read only installation media. Unfortunately I don't have one at hand, and there are none in the stores here. The only option seems to be to order one online, which will take a few days to ship. When I get the disks, I'll test using the installer with them. Thanks for the help! Have a great day; Ram
Hello, Thomas Schmitt wrote: Hello, thank you for the help. The DVD+Rs arrived, along with the DVD Drive. I tested the installer on one of my laptops, and found that its contents didn't change. (which is to be expected, as DVD+Rs can't be rewritten) I didn't bother testing that on different computers, because the result would be the same. However, I have tested the USB drive on a few other computers and here are my findings: On the Lenovo Legion 7i Gen 5*, Lenovo Thinkpad X1 Carbon Gen 2*, Lenovo Thinkpad X1 Carbon Gen 5 Intel*, and one Lenovo Thinkpad E14 Gen 5 all showed the corruption error. *These laptops have been tested for installer corruption twice (a week ago, and just now) However, on the Lenovo Yoga C740 and a different Lenovo Thinkpad E14 Gen 5 Intel showed no corruption issues. Both the Thinkpad E14 Gen 5s had the same specifications and type number, differing only in that the one with corruption of the installer has 24GB of memory (16GB installed in the slot, 8GB soldered) and the other only has 8GB soldered. They both have the same BIOS version, R2AET32W(1.07). Again, the ones that had corruption all had it at the same location. (byte 2303211, line 21165) This seems to be really interesting because the corruption only happened on certain computers, and it would stay that way on repeated attempts. Thank you for the help. Have a wonderful day, Ram
Hi, Ram Reddy wrote: Good to know that the software in the ISO still works from read-only DVDs. This seems to indicate that the firmware has a stake in the problem ... ... but the trigger would have to be very subtle. I ponder how to distinguish a change made by the firmware itself from a change made by the software in the ISO image. Maybe we can learn more by comparing the files /boot/grub/efi.img of the original ISO and of an altered USB stick. Mount both and copy out the files, or use dd to extract them from the partition interval that is marked in the ISO (ask by: fdisk -l). With debian-12.2.0-amd64-netinst.iso that would be iso=debian-12.2.0-amd64-netinst.iso or (with the approriate letter instead of "X") iso=/dev/sdX and then dd bs=512 if="$iso" of="$iso".esp skip=4476 count=18976 Both images will have about 9.5 MiB of size. I would be interested in seeing the altered one, as i can extract the original on my own. So if you have a way to upload the altered EFI partition image to a publicly accessible place i would be happy to investigate it. Much smaller and thus more bugreport-friendly would be a binary diff. You would need something more detailed than plain cmp. cmp -l gives the differing byte values one by line (i.e. lengthy if many bytes differ). I made an own willful alteration (subtracting 4476 * 512 from 2303211 yields the new problem byte address 11499): cp debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered echo ABCDE | dd conv=notrunc of=debian-12.2.0-amd64-netinst.iso.esp.altered bs=1 seek=11499 count=5 Then cmp -l debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered yields 11500 0 101 11501 0 102 11502 0 103 11503 0 104 11504 0 105 (Odd enough: Addresses are decimal starting at 1, not at 0. Values are octal.) Less useful is cmp -b: debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered differ: byte 11500, line 278 is 0 ^@ 101 A (Only the "A" is from my change. There seem to be pairs of octal value and character meaning.) I have my own unpublished differ for analyzing ISO entrails. So only the big battleship "diffoscope" comes to my mind as public binary differ. Probably there are leaner programs for that job. Have a nice day :) Thomas
Hi, Thomas Schmitt wrote: I tried doing that for the times the drive was modified, and they are here: https://drive.google.com/file/d/1Zd6iufVRsfIu-qzC-tJx4FEvCOESOz4_/view?usp=sharing Each modification was unique, so I put them all there. The files are for the 3 most recent attempts I did. G = gen, and the number after - is the attempt number. For the ones with no modifications, there's a text file with unmodified in its name which contains "unmodified" in it. unmodified.esp is the unmodified ESP in the installer. However, I found some interesting results. I tried redoing the tests on each laptop, and found that the modifications occurred somewhat randomly. Laptop Model | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 Lenovo Legion 7i gen 5 | modded | modded | modded | modded | modded Thinkpad X1 Carbon gen 2 | modded | modded |unmodded|unmodded|unmodded Thinkpad X1 Carbon gen 5 | modded | modded | modded |unmodded|unmodded Thinkpad E14 Gen 5 Intel | modded |unmodded|unmodded|unmodded|unmodded Thinkpad E14 Gen 5 Intel |unmodded|unmodded|unmodded|unmodded|unmodded Lenovo Yoga C740 |unmodded|unmodded|unmodded|unmodded|unmodded modded = modificationed It did show some patterns, however. Have an awe-inspiring day, Ram
Hi, Ram Reddy wrote: I downloaded the tarball and compared the original FAT filesystem with the various modified filesystem images.-------------------------------------------------------------------------- In Legion7iG5-*_modified.esp the suspect lost its ID card at the crime scene: At byte 39072 (0x98a0) the changes go from 0-bytes to the text "LENOVO". At byte 9711680 (0x943040) i see a change from 0-bytes to "BIOS". Diffing the result of "find" on the mounted unmodified.esp filesystem and Legion7iG5-*_modified.esp shows that a new branch of directoriies with a new file is in each of the modified filesystems: > ./efi/Lenovo > ./efi/Lenovo/BIOS > ./efi/Lenovo/BIOS/SelfHealing.fd The file is empty.-------------------------------------------------------------------------- In ThinkpadX1CarbonG5-0_modified.esp there is no company name to see in the changed bytes. I see UTF-16 strings "mation", "System", and "Volum\000me". ASCII texts "SYSTEM~1", "WPSETT~1DAT". The latter might possibly be "WPSettings.dat", which causes questions in the internet. Most plausible seems an answer in the course of https://answers.microsoft.com/en-us/insider/forum/all/whats-wpsettingsdat-generated-by/e11bca97-8c76-4662-8897-774ea3d5691a "The WPSettings.dat file is generated by the Storage Service (StorSvc). It seems that WPSettings.dat means the data files of Windows Phone's Store Settings saved on the drives, [...]" Diffing the result of "find" on the mounted unmodified.esp filesystem and ThinkpadX1CarbonG5-0_modified.esp shows that a new directory with a new file is in the modified filesystem: ./System Volume Information ./System Volume Information/WPSettings.dat The file has 12 bytes of binary salad: Hex: 0c 00 00 00 2e 42 6b 82 5d 88 0e c5 Char: . B k ] Dec: 12 0 0 0 46 66 107 130 93 136 14 197-------------------------------------------------------------------------- While it makes some sense to me that Lenovo Legion BIOS adds some Lenovo stuff to the EFI System Partition, i really wonder why Lenovo Thinkpad BIOS adds a Microsoft directory and file. Whatever, i'd say that the software in the ISO and especially Debian Installer are not suspicious to create directories with such names. Have a nice day :) Thomas
FWIW check the BIOS L[123] cache settings and consider changing them to more conservative "slower" values if possible. And you have different RAM models and configurations, could there be one DIMM in the mix that is running overclocked? Have a nice day :) Grüß Gott :-)
Hello, Thomas Schmitt wrote:FWIW check the BIOS L[123] cache settings and consider changing them to For changing the BIOS L[1/2/3] cache settings, these are laptop BIOSes which usually have very few features (other than secure boot/tpm options which are required for laptops with Windows), so there was no option for that in the BIOSes. However, when I get home I could try changing the cache settings on the Legion 7i gen 5, as it has a key combination in the BIOS to enter an "Advanced Mode" with overclocking settings and more, which likely has cache settings. running overclocked? None of the memory modules were overclocked, and they were all operating at stock speeds. The Legion 7i gen 5 got 10 passes in memtest with 0 errors, and both of the Thinkpad X1 Carbon laptops, and the Yoga C740, had fully soldered memory. One of the Thinkpad E14 gen 5 laptops also had unchanged memory. The laptops without any changed memory also showed similar results, so I doubt that it has anything to do with memory. Have an exceptionally great day, Ram
Hi, Thomas Schmitt wrote: This seems to make a lot of sense, since for the Legion 7i gen 5 whenever I started the system on the USB drive, it showed text Self Healing BIOS Backup Progressing... __% Or something like that. I probably should have said something about that earlier, but never thought about it. Sorry. I think that this could possibly be explained by flaws in my testing. Some of the computers were using Windows, and for those computers I plugged in the USB drive, shut down the computer, and booted the USB drive from the BIOS. In the time the USB drive was connected to the computer when Windows was in use, it may have written files to it. I realized this could be a problem towards the end of the testing, which is why I started shutting down Windows first before testing the USB. This can be shown in the bottom right results on the table, which are the most recent ones, and the ones where I shut down windows first had no modifications. So it seems to be that the only modifications of the installer caused by firmware is with the Legion 7i gen 5's self healing BIOS backup. I don't think there is any problem with the debian installer itself, and I don't think there's anything to change in it. When I get home, I'll look for an option to disable the bios backup, and then install debian on my computer. Thank you for your help. Maybe this problem could be listed on the Debian Installer FAQ? I think it could save some time for others finding it. Have an awesome day, Ram
Hi, Ram Reddy wrote: This was posted by Nicholas Geovanis <nickgeovanis@gmail.com>, not by me. My suspicion is that the Lenovo Legion EFI BIOS intentionally creates directories and empty file "/efi/Lenovo/BIOS/SelfHealing.fd". I do not see much benefit in investigating this further. More riddling is the directory and file found after using the USB stick on the Lenovo Thinkpad machines. The internet says that "/System Volume Information/WPSettings.dat" is part of Microsoft services. So it might be that the USB stick got in contact with MS-Windows while or before it was plugged into the Thinkpads. If this suspicion can be excluded then i'd say that Thinkpad has some strange BIOS habits. What remains for Debian in this bug report is the observation that the integrity check of the Debian installation ISO says "Your installation media or this file may have been corrupted" when just a harmless change has been made in the filesystem of the EFI partition. Some special handling of /boot/grub/efi.img during the integrity check would cause less surprise to the user. Like: "Possibly some software or firmware made changes in the EFI System Partition. This may be considered harmless unless booting of this Debian ISO encounters difficulties or other files show checksum mismatches, too." Have a nice day :) Thomas
Hi, trying to get our mails in sync again. We seem to have reached consensus in our understanding of the problem. i wrote: Ram Reddy wrote: This explains the MS-Windows file. So the Thinkpad firmwares are exonerated. This suspect can be considered to have confessed.-------------------------------------------------------------------------- As for a better message from Integrity Check: I find in the ISO in file /install.amd/initrd.gz the lines: "The file ${FILE} failed the MD5 checksum verification. Your installation" "media or this file may have been corrupted." But i failed to find these texts in debian-cd or debian-installer Even searching for just "corrupted" yields only a remark in debian-cd. Does any of the bystanders know from what package this message text stems ? Have a nice day :) Thomas
Hey Thomas!
Hi, i wrote: Steve McIntyre wrote: So that would be indirectly put into the code at https://sources.debian.org/src/cdrom-checker/1.62/main.c/?hl=152#L152 by "Template: cdrom-checker/mismatch" in: https://sources.debian.org/src/cdrom-checker/1.62/debian/cdrom-checker.templates/?hl=43#L43 Using my micking instincts i guess my proposal would be to create a new template like: Template: cdrom-checker/mismatch-efi Type: error # :sl3: _Description: Integrity test failed on EFI boot image The EFI boot image file ${FILE} failed the MD5 checksum verification. Possibly some software or firmware made changes in the EFI System Partition. This may be considered harmless unless booting of this Debian ISO encounters difficulties or other files show checksum mismatches, too. and to add a case in main.c, line 151, like (shown with reduced indentation): if(status != 0) { if(strcmp(filename, "./boot/grub/efi.img") == 0) { debconf_subst(debconf, "cdrom-checker/mismatch-efi", "FILE", filename); debconf_input(debconf, "high", "cdrom-checker/mismatch-efi"); } else { debconf_subst(debconf, "cdrom-checker/mismatch", "FILE", filename); debconf_input(debconf, "high", "cdrom-checker/mismatch"); } } else { debconf_input(debconf, "critical", "cdrom-checker/passed"); } A native speaker can probably find a better text and my guessing of the inner code relations may be wrong, of course. Even the path "./boot/grub/efi.img" is guessed from Ram Reddy's reported error message. Have a nice day :) Thomas