Dear Maintainers, BACKGROUND I have a cool script that downloads a .deb file using curl, grovels the Version out of the beginning of the file, and aborts the download if it's not a new version. This is useful because Zoom just has a URL zoom_amd64.deb with no version number until you look inside. The script is at https://github.com/barak/zoom-ubuntu-repo/blob/master/zoom-up and the salient bash lines, simplified, are touch tmp/zoom curl https://zoom.us/client/latest/zoom_amd64.deb >> tmp/zoom & ... v=$(dpkg --field <(tail --follow --bytes=+0 tmp/zoom) Version) which gets the new version into $v ASAP so it can be tested and the download aborted as appropriate. This used to work fine, but with a new version of dpkg it gets dpkg-deb: error: archive '/dev/fd/63' is truncated or corrupt, expected more data than available (35724 > 0) ISSUE What's going on is that the new version of dpkg-deb does a seek on its file argument, which fails if it's not a real file. You can see it here: $ dpkg --field /var/cache/apt/archives/task-ssh-server_3.86_all.deb Version 3.86 $ dpkg --field <(cat /var/cache/apt/archives/task-ssh-server_3.86_all.deb) Version dpkg-deb: error: archive '/dev/fd/63' is truncated or corrupt, expected more data than available (608 > 0) $ shasum /var/cache/apt/archives/task-ssh-server_3.86_all.deb 2454b0c8bec4b9f30efc5ef9a28e7dbfc76a3c65 /var/cache/apt/archives/task-ssh-server_3.86_all.deb $ shasum <(cat /var/cache/apt/archives/task-ssh-server_3.86_all.deb) 2454b0c8bec4b9f30efc5ef9a28e7dbfc76a3c65 /dev/fd/63 It would be nice if dpkg-deb would continue to allow a non-seekable argument, so my lazy pipe stream trick would continue to work. Cheers,
Note that this issue is not present in dpkg 1.22.22, so it cropped up 1.22.22 < version(dpkg) ≤ 1.23.7 I just tested with upgrading/downgrading dpkg on a machine with no other changes, so it is not in some library or support package or anything like that.
Hi! I'm wondering whether «wget -N» or the equivalent curl incantation would not be an easier way to handle this? Perhaps you'd need to keep the inode/device or a checksum of the file and compare before and after. If so, that would mean dpkg-deb would stop being involved. :) Of course all these still seems like a workaround for that vendor not providing a proper repository, with properly named binary packages. But I assume getting them to fix that is unfeasible? (But maybe worth trying anyway?) Yes, this was part of a set of changes in 1.23.6 to fix a security issue due to insufficient validation, which is something I've had pending for some time now, also to detect way earlier and be able to give a proper error message when a truncated or partially downloaded package is passed to dpkg (see f.ex. #929100). I didn't expect this to break anything!¹ And while I could change the current check to take into account whether the archive file is a pipe/socket/fifo and then ignore it when the archive size is 0, my plan has been to switch the code to perform two passes anyway. One to analyze the entire ar container for correctness, then perform a seek to extract the requested member data, because at least for dpkg driving dpkg-deb there's already a requirement that it cannot use pipes/fifos/sockets for archives, due to it calling dpkg-deb multiple times (first to extract the control member then the data member). To st correct expectations, my first inclination right now would be to declare that streaming is not supported. But I'll ponder a bit what it would take to add support for such "streaming" mode in that new two pass pre-analysis paradigm, and whether that would make sense, given the potential security risks, where dpkg-deb is the one tool where it is supposed to be able to handle untrusted .deb data (see the SECURITY section in its man page), in contrast to dpkg, where the assumption is that the archives are trusted. ¹ I guess this reminds me a bit of https://xkcd.com/1172/. :D Thanks, Guillem
Well, I suppose there are a couple ways to look at it. One is, should this be Severity: wishlist or Severity: eldrich-horror Another is to note that the file format of a DEB file is deliberately based on cpio and tar formats. And these are deliberately constructed to allow streaming, since they were originally designed to be read and processed from magnetic tapes by machines with very little memory. I'll grant that our machines now have more memory. But streaming access is still enormously faster than random access, due to the way caches and rotating disks and SSD storage works. And Debian variants do run on tiny things like cheap routers and wifi access points. I understand the idea of wanting to validate the file before processing it. But in the unix tradition, maybe that should be a separate tool, called before actually unpacking and installing a package? Things like pulling out the version number seem pretty lightweight compared to full validation of a potentially enormous file. And if things are being validated, where do we stop? Each time we read a property out? Do we cache checksums and recheck them in case the file is changing under us? Are validations cached in a database, recording and using timestamps for invalidation? It's a slippery slope, and sort of violates separation of concerns. I'd suggest that validation is typically unnecessary, because we should be able to recover from an installation that hits an error, and that's the robustness we should be focussing on. Also I really liked my little hack!
xkcd#1172 Workflow or as I remember it: spacebar heating.
Okay, I modified my zoom-up script in https://github.com/barak/zoom-ubuntu-repo to grovel the version field out of the header of the stream using other means if dpkg --field doesn't work. Enjoy!
Hi! I realized later on, that support for streaming was added and documented after a couple of requests some time ago, and this is being used in other places right now. So I've gone with documenting that if an input is non-seekable then certain robustness/security checks will not take place, and then fixing the current regression and adding some regression tests for streaming input, to avoid this in the future. Will be part of my next push to be included in the next release. Thanks, Guillem
Well you'll be pleased to know that my zoom-up script (repo above) already assumes dpkg can handle a streaming input but dynamically falls back to groveling through the stream itself if dpkg cannot. So it will take advantage of your streaming fix, but will be robust should streaming support ever be disabled. Cheers,
Hi! Bug #1132051 in package dpkg reported by you has been fixed in the dpkg/dpkg.git Git repository. You can see the changelog below, and you can check the diff of the fix at: https://git.dpkg.org/cgit/dpkg/dpkg.git/diff/?id=a658782c1 --- dpkg-deb: Do not fail on non-seekable archives We have documented that we support non-seekable archives for some time, and this was a regression introduced in 1.23.7. Disable the robustness check if the input is not seekable. Add regression tests to avoid this happening in the future. Fixes: commit 7d54287521ac4e1b2061d4a92467c634525436d3 Closes: #1132051