Hi, it's been reported to me that debian-testing-amd64-netinst.iso[1] was broken, in that it features d-i using an “old” kernel (3.16.7-ckt4-3), but kernel udebs apparently from sid, as can be seen in the list file[2]: they're at version 3.16.7-ckt7-1. 1. http://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd/debian-testing-amd64-netinst.iso 2. http://cdimage.debian.org/cdimage/weekly-builds/amd64/list-cd/debian-testing-amd64-netinst.list.gz Given the kernel ABI hadn't been bumped, one might argue that the udebs should still work and not break due to some unknown symbols, but that's another topic. I really don't expect us to fetch bits from unstable in testing images. I find the doc in the parent directory[3] quite confusing anyway: | These images are produced every week, normally on a Monday, but this may | vary. They include: | * The latest debian-installer build (currently the "sid" daily builds of the installer) 3. http://cdimage.debian.org/cdimage/weekly-builds/ It's either a daily build (from d-i.debian.org), or d-i from sid. Mraw, KiBi.
Hi KiBi, Hmmm. That's very odd. The weekly builds are currently set up to use daily d-i builds rather than what's in the archive, and they've been that way for a long time. As such, we also use sid udebs for debian-installer, as that should match what we'll be getting from the daily d-i builds which are built using sid. Are you sure that the kernel there is old? If so, that's the bug I think. It's difficult for me to check exactly right now what that kernel version is. We've been using the daily d-i builds for a long time, to guard against exactly this kind of breakage with kernel version mismatches. Has something changed? ACK. That's old wording, and has always been confusing. Given we don't tend to upload to the archive like normal packages anyway, it's best to just remove the "sid" mention altogether I think. Ditto the mentions of sid etc. in the daily-builds area coule do with fixing up I think?
Hi, Steve McIntyre <steve@einval.com> (2015-03-03): I don't think that's a good idea. For one thing, there's no trust chain here (from d-i.d.o because http; and from git, because #746967). I extracted the kernel from the iso, and looked at its version string. The guy who reported that issue to me also mentioned that (even if non publicly, and I still haven't seen him open a proper bug report). Some archs are missing daily builds, because autobuilding in jessie is broken, but AFAICT missing builds are only: - arm64 - ppc64el - sparc (See http://d-i.debian.org/daily-images/daily-build-overview.html) I certainly didn't touch anything on pettersson, or anything remotely involving debian-cd as far as I can remember. We probably should start by figuring out what d-i we want to embed (see first paragraph). Having (some, maybe not the whole set of) images with a daily d-i (if we can fix the trust chain) would be nice. But that really shouldn't be labelled “official testing images” as currently done. Being able to spin some weekly build (possibly manually) with what's in *sid* (i.e. not dailies) is nice to figure out whether we're going to end up with breakages if/when d-i/sid migrates to testing, before we build official release images. Not sure it gains us much compared to first migrating d-i/sid to testing and building images, though. Mraw, KiBi.
[ Writing this off-line again while travelling so I can't check things
directly on pettersson etc. ]
Hi KiBi,
We've had this discussion befire, surely? On pettersson we rsync
things (over ssh) directly from d-i.d.o (dillon), so we don't depend
on http or git. Or am I missing something new? (I can't see #746967
atm...) There's also an explicit warning in debian-cd if you grab
things via plain http or ftp:
echo "WARNING WARNING WARNING WARNING WARNING WARNING"
echo "$0: insecure download for d-i build: $DI_WWW_HOME"
echo "WARNING WARNING WARNING WARNING WARNING WARNING"
OK, thanks for checking that.
OK. Assuming amd64 is working OK, I should check the rsync has not
broken - that seems to be the most likely cause right now. Thinking:
it would also be lovely to verify the versions of vmlinuz and the
kernel udebs in debian-cd at build time, and (optionally?) abort the
build if we know we're going to be making broken images. What do you
think?
Yup.
I'm definitely open to suggestions on what else we should be
providing. I'm not 100% sure the best way to go, though...
In the past, we had the two different sets of daily images:
* use the most recent testing version of d-i in tha archive
* use the daily d-i builds from d-i.d.o (and other hosts before that)
and then we'd switch the weekly images from one source to the other
depending on how working/broken we thought each source was likely to
be. Since the kernel team got much more active a few years back and
kernel ABI changes have been much more common during the life of
testing, using the most recent d-i release from the archive has had a
much shorter working life than we used to have. Hence, we switched
over to using the daily d-i builds as a base.
This can be YA argument for more regular d-i uploads to the archive to
fix that problem, of course. But we know how much work that
involves. At the moment I feel what we have is just about the best
thing we can get without that massive additional effort. It normally
works for people, modulo the odd breakage from time to time like we've
seen here.
Yeah :-/ We've done that in the past, and it's not that difficult to
configure debian-cd to do this. But IIRC it never really gave us much.
Steve McIntyre <steve@einval.com> (2015-03-05): ACK. The rsync part covers the former, but there's still a broken trust chain because of the latter. There's no way for us to know a breakage is going to happen. In this particular case the ABI is the same, say 3.16.0-4-amd64; the ABI is embedded in kernel udeb package names, so if the ABI matches, udebs are supposed to be compatible with the kernel. I'm not sure why that is not the case here (maybe the ABI doesn't cover or not completely udebs?). We really shouldn't enforce using the very same revision for kernel and module udebs, IMHO. I guess we could stick to this once the trust issue is resolved. Daily builds will have to be changed one way or another anyway since it can't be implemented as done previously starting with jessie hosts… OK… Mraw, KiBi.
I think the ABI only guarantees you can load old modules into a newer vmlinuz. In particular I think it doesn't guarantee that you can load newer modules into an older vmlinuz (even if the ABI is the same). The ABI is there to stop your existing modules from breaking when you update the kernel. IOW it is permissible for a module to gain a dependency on a new symbol provided by a newer vmlinuz. This is why local netboot pxe infra sometimes breaks -- they have a vmlinuz downloaded locally but are pulling udebs off the network, which might therefore be newer. Please reread my caveat now though ;-) Ian.
Ian Campbell <ijc@debian.org> (2015-03-05): Thinking a bit more about this: I think having weekly testing with everything from testing makes the most sense. We could have a separate, not-labelled-testing but rather “daily” or bleeding-edge or something in that spirit (but neither experimental nor sid) that picks d-i on d-i.d.o and udebs from sid. I'm not sure it would be reasonable to expect such a build on a daily basis (maybe by reducing the generated images to the bare minimum?), but then we would have a clear winner as far as the name goes… Mraw, KiBi.
Right, I've just had a chance to look at that now. Ugh... :-( ACK, I see your logic. Right, yes. Anyone have any bright ideas yet on how to do that?
Right, that helps explain. Thanks Ian! OK... Are we going to do more frequent d-i uploads to make sure that works, though? It looks like the changes for (daily) builds are going to force quite a lot of work on us here. I'm quite prepared to help with that when I'm back (and have more time!) in a week or so... Right.
Steve McIntyre <steve@einval.com> (2015-03-08): I can't promise anything. Mraw, KiBi.