Dear Maintainer, I've seen that you're in the past considered and discussed shipping the procps version of pidof at https://lists.debian.org/debian-devel/2013/12/msg00145.html I'm filing this bug because I think the previous reasons against this no longer apply. c.f. "[...] moving pidof around certainly doesn't kill off sysvinit-utils for us (there are still over a dozen other tools in that package in Debian)." The sysvinit-utils package has gone through a heavy diet and now ships alot less tools (in favour of shipping them from other maintained upstreams, notably util-linux). I'm generally interested in shrinking the essential set, but users of pidof seems to be too many to reasonably introduce dependencies for. The currently remaining tools in sysvinit-utils (in stretch/sid): pidof - suggestion to ship it from (new essential) procps(-base) service - might soon be in init-system-helpers (see #805487). killall5 - analysis of the 19 matches on codesearch seems to boil down to: openrc, util-vserver. Adding dependencies seems doable. fstab-decode - codesearch analysis indicates open-iscsi, drbl. Adding dependencies seems doable. Summary: making sysvinit-utils non-essential seems doable. This would also make sysvinit-utils dependencies transitively non-essential. eg. startpar. (Not sure why that dependency exists to start with, but proposed approach seemed simpler than researching that for the same result.) Please also note that sysvinit-utils already have some reverse dependencies despite being essential which works in our favour. I'm attaching a patch for your convenience. Please note that a synchronized sysvinit upload is needed. If you think we should proceed I'll volunteer to NMU sysvinit for this. When doing so you'll need to update the attached patch with the version number of the sysvinit upload and the Closes bug number in the debian/changelog. An announcement on debian-devel about introducing a new essential package is also needed. Regards, Andreas Henriksson
I can re-visit making a procps-base with pidof enabled in it. Upstream is actually maintained (by myself and a few others). It sounds like it is the right time to be doing this. I think it needs a broader discussion than just us two. Is killall5 being actively maintained upstream? If it is not then procps upstream (e.g. me) could recode this and maintain it as part of procps upstream then eventually flow this down to procps-base at a later date. It's basically pkill with different options. OK, then I'm happy for procps-base to exist with only pidof in the package. I'd do it slightly differently to how you had it in the patch but with largely the same result. - Craig
Hello Craig Small. Thanks for your quick reply and for considering. I'm aware (but wasn't aware that you had much help from others). :) I think soo too, from multiple perspectives but most importantly the Debian release schedule. ACK. I'm personally not too keen on posting to debian-devel myself though since just about anything seems to turn into a gigantic shit throwing party. I'd be very happy if you'd be willing to take that on. :P I'm happy to assist with technical matters and do grunt work, possibly provide some background info where people have questions, but mostly stay away from any wider debian discussions which just raises my frustration levels too much. In theory there's a sysvinit upstream, but in practise Debian maintenance of sysvinit is dead (and I don't know the status of the supposed upstream sysvinit project) so any upstream changes probably won't flow into Debian anyway. I don't see killall5 as a blocker for this given very few users (which if pkill provides the same functionality could probably just be ported over to it now that I guess any distribution can pretty much be assumed to have procps toolset... or alternatively just have those users depend on sysvinit-utils directly). Feel free to try establish communications with the sysvinit upstream to check if it's suitable for procps upstream to implement killall5, but I think this is mostly off-topic for this bug report and should be handled as a separate (upstream) discussion. But as already mentioned, not sure it's worth the effort to provide killall5 rather than just porting the few users over to pkill. Feel free to implement it as you wish, but please keep the essential package content/dependencies as small as possible. The patch was just provided as convenience, example and proof-of-concept. I find it sometimes easier to get the details discussed with a patch as example given I'm not a native english speaker. Regards, Andreas Henriksson
Mainly me and Jim for top. Jim usually sends me his patches. Oh you noticed that too? I'll send the email. I think it does that to everyone. Especially anything that has a sniff of systemd about it (anything with init has that sniff, alas). So, to the actual proposal. I need help especially around the why. =================== What: Create a new package procps-base. This uses the existing procps source package and just enable building of pidof. procps-base will be an Essential package and only contain pidof. Why: The aim is to make the sysvinit-utils package non-essential. This was discussed previously in 2013 [1], however there are some important differences between 2013 and now. 1) The previous change was due to possible upstream relocations of pidof. This is about Debian package contents. 2) In 2013 there were a lot of other programs in sysvinit, in 2016 we're down to 4 (pidof, service, fstab-decode, killall5) (why change from one essential to the other?) What about the other binaries in sysvinit-utils? pidof - moves to new Essential package procps-base service - moving to init-system-helpers [2] fstab-decode - 2 packages use this (open-iscsi, drbl) killall5 - 2 packages use this (openrc, util-vserver) The idea would be those 4 packages would depend on sysvinit-utils. pidof is used be far too many packages to have the same treatment. What other packages will procps-base depend on? libc6 and libprocps5. libprocps would be newly pulled in. 1: https://lists.debian.org/debian-devel/2013/12/msg00121.html 2: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=805487 =================== - Craig
Hello Craig Small. Thanks for taking this on and drafting a proposal... I'll throw in some of my own ideas below in case they're helpful so feel free to leave out or reword things as you see fit. Fwiw, in case you want to avoid any potential people out there who might think you're just beating a dead horse for your own personal gain feel free to start out with "I've been asked to..." which should hopefully kill off those ideas from the get-go. On Thu, Jan 07, 2016 at 09:17:41AM +1100, Craig Small wrote: [...] Short and clear. +1 I see what you wrote mainly as a long-term why.... I'd like to add a short-term: Make sure more of the essential utilities in the base of the operating system comes from active and healthy upstreams. Also in general I think it's good to make sure we don't /needlessly/ deviate. This means we can benefit from issues discovered in other distributions and fixed upstream, while at the same time we can be good free software citizens and bring our own fixes upstream for the benefit of the entire free software echosystem. Hopefully pidof is trivial enough that it won't need much fixes though. Why? See above. Otherwise I think this part is good as is. Some minor ideas: - maybe say "longer term plans/ideas..." since I'm not vouching we'll be able to actually pull off makintg sysvinit-utils non-essential before stretch release. - maybe tone down making sysvinit-utils non-essential since I think that should be a separate discussion (and more considerations than just pidof and service, etc, needs to be taken into consideration). Providing procps pidof is a useful idea on its own in my opinion. I've also spent some time on researching the pidof users and the main user seemed to be direct invokations from init scripts. These could benefit from being converted to LSB "pidofproc" helper function (which falls back on executing pidof if available though). I'm not sure the busy work of fixing all init scripts is worth it though. Personally I'd rather spend my time on writing native systemd unit files (which will probably be easier to reuse from other potential future init systems than LSB init scripts), and I doubt the vocal people who shys away from anything systemd related is willing to actually step up and help out with the practical work either. So not sure if it's worth even mentioning LSB pidofproc... maybe in a short paragraph somewhere. for comparison, eg. In comparison sysvinit-utils pulls in: libc6, libselinux1, startpar (Maybe skip mentioning libc6 in both cases.) HTH. Regards, Andreas Henriksson
Hello Craig. On Thu, Jan 07, 2016 at 09:17:41AM +1100, Craig Small wrote: [...] [...] I was thinking that maybe it might be a good idea to possibly avoid this dependency. That would allow us to keep essential out of any upcoming libprocps transition, keep essential minimal, etc.... The idea is simply to build a static version of pidof and strip out the unused parts of libprocps.a, made possible by building the lib with -ffunction-section -fdata-sections and linking with -Wl,--gc-sections. The attached patch does this. (Then just install pidof.static instead of pidof in the base package and discard pidof.) The end result is 34896 bytes for pidof.static (ie. approx 20k growth to avoid the libprocps5 dependency). (While also growing the libprocps.a for the extra sections, but I don't see how that would affect anything.) Things would be even better if we could somehow also avoid libsystemd0 dependency. It's already a dependency of essential packages (via atleast util-linux) but just for the sake of not having to discuss it... Not sure if it's worth implementing a double build-pass (with/without libsystemd enabled) just for pidof though. Maybe there's a simpler way or maybe we just leave it as is.... (Fwiw, the upstream util-linux build system supports building static versions for basically all utilities which has proven handy in a couple of occations. You might want to have a look there if you're interested in an example of a tidier build system example.) Do you have any thoughts on this? HTH. Regards, Andreas Henriksson
Hello Craig Small. Here's a small status update which might be relevant to consider for this bug report regarding procps-base / pidof. The policy-related service management tools moved into into init-system-helpers yesterdays uploads of sysvinit[1] and init-system-helpers[2]. This means service is no longer part of sysvinit-utils and pidof is the only remaining widely used *binary* in that package. I also noticed there's a non-binary which also needs to be considered if sysvinit-utils is to be made non-essential: -> /lib/init/init-d-script This script is sourced from 10 packages according to codesearch.debian.net and random samplings says atleast some of them are unconditional. Introducing dependencies is still doable in my view. Regards, Andreas Henriksson [1]: https://tracker.debian.org/news/741657 [2]: https://tracker.debian.org/news/741508
Hi Andreas, One thing before we move to the procps pid, it doesn't match all processes the pidof one does [1]. I have fixed it upstream but probably worth waiting for that to appear otherwise things like kde and sshd sessions "disappear". - Craig 1: https://gitlab.com/procps-ng/procps/issues/4 -- Craig Small (@smallsees) http://enc.com.au/ csmall at : enc.com.au Debian GNU/Linux http://www.debian.org/ csmall at : debian.org GPG fingerprint: 5D2F B320 B825 D939 04D2 0519 3938 F96B DF50 FEA5
Hello! This information is not directly related to procps, but it is kind of related to the reason why we'd want to introduce procps-base as there's no big point in doing it if we can't also make sysvinit-utils non-essential. Also we don't really have a good tracking bug so posting here to archive the information somewhere before I forget about it.... I've looked closer at the users of /lib/init/init-d-script which was mentioned in a previous post that some packages init script unconditionally sources. Apparently "new school" init scripts as generated by current dh-make and as visible in /etc/init.d/skeleton is the reason for these unconditional sourcing of the file which causes problems if we downgrade sysvinit-utils (which ships /lib/init/init-d-script) to non-essential. Given this style of init script is apparently the currently recommended form, I assume we can see the number of packages using it rise over time... The problem should be (partially?) less problematic if the package using "new school" init script also ships a native systemd unit file, which should avoid the init script being executed at all... (Though some people still manually execute init scripts even under systemd out of old habits and this only works because systemd ships an LSB hook that catches this and issues systemctl seamlessly.) a) packages shipping an init script that uses /lib/init/init-d-script according to codesearch.d.n b) if it's not a big practical issue, because the init script will not be used under systemd. pkg has masking systemd unit file ======================================================================== sddm yes coquelicot yes open-iscsi yes graphite-api yes apache2 no! (only partial snippet to extend autoconverted script) batmand yes netplan no! (but carries own fallback copy of init-d-script!) courier-authdaemon yes uwsgi-emperor no! opendnssec-signer yes kxd yes lvm2 yes waagent yes prometheus yes dbab yes (also carries own fallback copy of init-d-script!) prometheus-pgbouncer-exporter yes shairport-sync yes freeipmi-ipmiseld no! knot-resolver yes prometheus-mysqld-exporter yes hhvm yes prometheus-pushgateway no! courier-mta yes prometheus-node-exporter no! The investigation was done using (so might very well be incomplete): for pkg in .... ; do apt-file show $pkg | grep -e 'init.d' -e 'systemd/system' done TODO: Investigate what happens if executing such an init script directly under systemd where normally systemd-installed LSB hooks would override it and use systemctl... Regards, Andreas Henriksson
If we want pidof to be provided by a new procps-base instead of
sysvinit-utils for buster, maybe now is the time to start uploading
packages with that change to experimental?
(Please see #810018, #851747 for additional context or if you have
forgotten how far this proposal got previously, as I had.)
This appears to have been fixed before procps 3.3.12 (which is in stretch)
if I'm reading changelogs correctly?
Regards,
smcv
I'd say if it is going to happen, it should happen now. This was the revert of the namespace problem? - Craig
"This" referred to <https://gitlab.com/procps-ng/procps/issues/4>
"pidof does not check comm field in /proc/.../stat file like sysvinit did"
which you pointed to from
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810018#40>.
You appear to have fixed it in
<https://gitlab.com/procps-ng/procps/commit/3f5b75035ec4151a5ffe911b0857d0419ffb5a15>
"pidof: check cmd if space in argv0".
I assume by "the namespace problem" you are referring to
<https://gitlab.com/procps-ng/procps/commit/791cb72d32dd963a76fccc4b46facd906a4381fb>
"Revert "Support running with child namespaces""? If so, that looks
unrelated.
smcv
Just wanted to say, that I had a look at the sysvinit-utils package (again) the other day and wondered whether we could move /sbin/killall5 and /sbin/fstab-decode into the (non-essential) initscripts package, as according to codesearch there aren't any users of those two binaries outside of initscripts itself (the notable exception being open-iscsi, which has native service files, so this wouldn't be a concern). The only blocker for moving /sbin/killall5 to initscripts is that /bin/pidof is currently a symlink pointing at /sbin/killall5. So if you proceed with creating such a procps-base package, we could make a follow-up upload for src:sysvinit, dropping /bin/pidof and moving /sbin/killall5 and /sbin/fstab-decode Regards, Michael
package. What would it take to move forward with this?
Hello, You may recall quite some time ago there was this bug #810018 where it was asked can procps ship pidof so that sysvinit-utils could have its Essential flag removed. That was.. back in 2016. Is this still something that would be useful to be done? Michael put some good work in looking into what needed to be done at[1] as we needed to unload some files from sysvinit-utils and put them into procps; mainly the pidof binary and manpage. If we are going to do this, 9+ months from a freeze is a good time to do it. However, does it need to be done? If so now is probably a good time to do it. Otherwise, I can close this bug and keep pidof out of procps. - Craig 1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810018#67
Craig,
Thanks for this.
This dates from before my detailed involvement with this area of Debian. I have
read through the bug report, but apologies if I have missed pertinent detail.
I am happy to work with you to achieve this if it is really worthwhile and time
well spent. However, I am yet to find really compelling reasons. The suggestion
seems to have arisen because:-
1) Upstream sysvinit is dead.
This is not true. Jesse Smith is active[1], extremely helpful and recently
migrated upstream away from savannah[2].
2) Debian sysvinit is dead.
'The reports of my death have been greatly exaggerated.'[3]
3) A desire to reduce the Essential set.
I understand and appreciate the general motivation for this. However, moving
pidof to procps would make procps Essential and it is already about 20 times
bigger than sysvinit-utils, so it does not achieve the aim.
A new Essential procps-base just containing pidof may achieve this objective
depending on how the question is framed. Whilst in many cases the question
and its answer will be obvious, it is somewhat opaque to me exactly which
aspect of the Essential set size we are endeavouring to minimise here:
download size, number of packages, time and cost for debootstrap et al. to
process the set?
The pidof in procps-base option would probably reduce the Essential set
download size by, maybe, a few tens of kB (sysvinit-utils is only 83kB), the
number of packages would be unchanged and I see no reason to expect the
processing time to be significantly different.
In short, to me this seems a lot of work for very marginal gain.
What do you think?
Best wishes
Mark
[1] https://www.patreon.com/posts/62303182
[2] https://github.com/slicer69/
[3] attrib. Mark Twain
Hello Mark, Craig, [....] [...] I don't see why you think pidof (and thus entire procps) must be Essential. That would indeed be counter-productive. (I haven't re-read the discussion, but I'm pretty sure we already covered this.) As I've already proven elsewhere sysvinit-utils (with or without pidof, which AFAIR is the only semi-useful utility left in that binary package) can be made non-essential without any problems already. I think apart from reducing the essential set, it is also useful to eliminate pointless differences. The distributions I've looked at (that are not just following along as a debian derivate) uses procps pidof already. Finally I have to say that I will not object to just closing this bug report (even though I think the original motivation still holds true), because atleast I don't have the energy to fight for forward movement anymore. Regards, Andreas Henriksson
Exactly. And there have already been issues in which pidof's behavior changed in unexpected ways because of the needs of sysvinit (e.g. https://bugs.debian.org/926896 ). Decoupling the two makes such issues less likely, and removes the one way in which systems not otherwise using sysvinit have a dependency on sysvinit components. I wouldn't expect procps to become essential; ideally packages that use pidof in scripts would add an appropriate dependency. (This would help people building minimal containers/chroots.) But orthogonally, I think there's value in migrating pidof to procps.
On Mon, 21 Mar 2022 15:07:21 -0700 Josh Triplett <josh@joshtriplett.org> wrote: Debian. I have pertinent detail. However, moving about 20 times read pidof, package) as a issues use help think Hi Craig, Now that Bookworm has shipped, what about finally finishing this and getting rid of this debianism? There is really no reason to delay it any longer at this point. Thank you!
Hi Luca, I'll need the assistance of the sysvinit-utils maintainers (CC'ed) as well, as pidof will be moving from that package. Doing this will also bring Debian (and Ubuntu) in line with most other distributions out there that use pidof from procps. I'm, not going to cover sysvinit-utils moving out of Essential as it doesn't matter for procps if it is or is not. So I'm looking at https://wiki.debian.org/PackageTransition and assuming procps is 2:4.0.4-2 and sysvinit-utils is 3.08-3 I would create procps 2:4.0.4-3 with pidof and Breaks: sysvinit-utils (<< 3.0.8-4) and Replaces: sysvinit-utils (<< 3.0.8-4) sysvinit-utils maintainers create 3.08-4 without pidof and have Breaks: procps (<< 2:4.0.4-3) If everyone is in agreement with this, I'll update procps this week. - Craig
Thank you, LGTM.
Craig, Thanks for this. IIUC, the proposal[1] was to create a new Essential procps-base just containing pidof. Otherwise bin:procps would have to become Essential itself. Its installed size is about 20 times larger than sysvinit-util and that wouldn't contribute to shrinking the Essential set. I think this approach would also require a debian-devel email announcing the addition to the Essential set and I suppose the new src:procps will need a trip through NEW. The dependencies would then be:- procps-base: Breaks: sysvinit-utils (<< 3.0.8-4) Replaces: sysvinit-utils (<< 3.0.8-4) sysvinit-utils without pidof: Breaks: procps-base (<< 2:4.0.4-3) I hope I have understood the previous discussions correctly . I am not trying to stand in the way at all, just ensure that this transition is worthwhile and done correctly. With best wishes Mark [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810018#10
Good point, I had forgot about that detail, thanks for the reminder
Good catch, I'll write something up on this as it changes a lot. There are probably two questions 1) Does pidof need to be in an Essential package? While a lot of packages do have pidof in them a lot (but not all) of those are in init scripts. 2) Does pidof need its own package then - Craig
I think it's easier and less work for everyone involved to keep it essential for now, and then eventually be scaled back and merged back into the existing package later.
Hello, For quite some time (since 2006!) there has been a discussion at[1] about changing from the sysvinit-utils version of pidof to the procps one. A quick scan of the various distributions shows that only Debian and Ubuntu (and I assume most other downstreams) use the sysvinit-utils version. So to rehash some old drafts, here's the proposal. What: Create a new package procps-base. This uses the existing procps source package and just enable building of pidof. procps-base will be an Essential package and only contain pidof. Why: This would bring the pidof variant in line with other distributions. sysvinit-utils would no longer need to be Essential (though that's a separate issue) and would only have init-d-script, fstab-decode, and killall5. The majority of usage of pidof is in init or pre/post scripts, which really should be using the LSB pidofproc function. That function in turn optionally uses pidof if the pidfile parameter is not given. That's probably a way forward for sometime in the future to not need procps-base Essential, but it is a way off. sysvinit-utils requires only libc6 while procps-base require libproc-2 but this is the same library used for the ps,top,w etc tools which are installed on most systems. 1: https://bugs.debian.org/810018
Hello, I support using procps implementation in Debian, to align us with the rest of the world. I however do not think pidof needs to be part of the Essential set. Instead I think pidof can just be part of procps package. The sysvinit-utils package will then pull in procps via a dependency (once sysvinit-utils stops being Essential), which would smooth the transition for all sysvinit users until LSB pidofproc has been implemented in all init scripts. Additionally most uses of pidof is `if pidof [...]; then` which will expand to false/else when the pidof command is not available (which it should be on all "normal" systems, as procps is already Priority important). A number of years ago I tested booting a regular debootstrapped system (with all priority important packages, etc) with sysvinit-utils excluded and that did not show a single warning about missing pidof. Someone might want to repeat that experiment. Regards, Andreas Henriksson
Hi Craig, I welcome the effort in general. Like Andreas, I question whether having pidof remain essential is useful. A quick codesearch https://codesearch.debian.net/search?q=%5Cbpidof%5Cb&literal=0 suggests that we have less than 500 source packages that even mention it. Many uses are in test suites or documentation, so the final number will be lower still. If we agree that pidof should not be essential, the next question is whether we need that procps vs procps-base split. Andreas suggests "no". I don't have a strong opinion on that one. Let me suggest an alternative transition plan. We extend sysvinit-utils with a new virtual package "pidof". Then we MBF packages using pidof to add a dependency on pidof. Once a significant portion of those bugs is fixed, we move pidof out of sysvinit-utils and have it drop that virtual package. procps or procps-base can then add pidof (with Breaks+Replaces for sysvinit-utils and Provides: pidof) moving it out of the essential set in the process. Any remaining bugs would be bumped to rc-severity at that point. I fear sysvinit-utils being essential is not separate (see below). It really needs to be done together, so additionally there would have to be another MBF for those other tools asking to add dependencies. For as long as sysvinit-utils contains /lib/lsb/init-functions, it'll have to include pidof or depend on it. Therefore the pidof provider can only become non-essential once sysvinit-utils is non-essential. If you see the change in implementation as more urgent than making all of it non-essential, then procps-base is needed indeed. Yeah, please don't increase the essential set. The addition would be very unwelcome to embedded systems. So in essence, you asked for changing the pidof implementation and Andreas and me try to turn this into a much bigger quest of making it non-essential. While these matters are related, they can be done independently in principle and if you do not want to take on the non-essential part, I fear I see little alternatives to that procps-base proposal. Pulling procps-base into the essential set also adds it to the bootstrap set. That also adds numactl to the bootstrap set. I'd rather not have it grown if possible. Both are currently cross buildable, so it's not the end of the world. Helmut
Yeah, let's not make this task impossibly huge and lengthy, please. Using the same implementation as every other distro has immediate benefits for everybody, packagers and users. Rearranging the packaging details and priorities and whatnot is pretty much an internal-only detail - which doesn't mean it's not good or useful or worth doing, just that I don't think it's worth blocking the first part for it, as it can happen just as well later. The procps-base proposal looks good to me.
Hi! I'm all in for shrinking the essential-set. If there is consensus to switch pidof implementations that also seems fine to me in the abstract. But this shuffling around of essential-ness and new tiny packages and stuff seems a bit unnecessary to me, more so when this increases the bootstrapping-set. I'd also rather see instead a _proper_ transition to get sysvinit-utils out of the essential-set, and then at some later point procps can take over pidof. Then there's the following, which I guess complicates things: $ dpkg -S bin/pidof | cut -d: -f2 /bin/pidof Also why is killall5 not a candidate too? In any case the pidof CLI interface does not seem too big, so this does not feels urgent to me, given the trade-offs. I think the status_of_proc function could be switched to use start-stop-daemon (s-s-d) --status instead of pidofproc. To replace pidof inside pidofproc I guess s-s-d could grow some option to print the pid, I'd be happy to implement that. After doing a quick scan over codesearch.debian.org, I notice that it looks like many current uses of pidofproc should instead probably be using status_of_proc, and others seem to just be fetching the pid to then perform some action that could instead all be done directly with s-s-d. Thanks, Guillem
There really is two options then:
1) Switch pidof to new Essential package procps-base THEN update/fix the
dependent packages
2) Update/fix the dependent packages THEN move pidof to standard procps.
Independent? of either: re-work init scripts to use start-stop-daemon
For people that want the standard pidof #1 is preferred, for people
concerned about Essential's size #2 is preferred.
Most of the pidof usage can be broken down into a few sets:
* Used in comments/documentation - can be ignored for "pidof is Essential"
purposes
* Used in init or pre/post scripts - should probably be using pidofproc
* Within their programs use something like system("pidof myprog")
and then there are a few other odd ones that don't fit into those three.
Then there's the following, which I guess complicates things:
I'm not sure what the complication is here.
Also why is killall5 not a candidate too?
There's no other implementation of killall5 though it is probably something
that could be done with one of the other /.*kill.*/ programs.
Significantly, I don't think there is any real dependency of this program
in other programs, codesearch seems to be only for documentation.
I agree that this is a good idea. It will be more about removing the
Essential flag on any package.
- Craig
Probably because it makes no sense outside of sysvinit, except that as a footgun. (Also, is it equivalent to pkill --inverse?)
On Tue, 14 Nov 2023 10:40:33 +0000 Luca Boccassi <bluca@debian.org> wrote: wrote: it procps-base packaging Hello Craig, It's been six months since the last update on this bug - would it be possible to finally take this over the finishing line? Thanks!
On Fri, 24 May 2024 01:24:39 +0100 Luca Boccassi <bluca@debian.org> wrote: and making as good Hello again, It's almost a year since the last ping, and more than 9 years since this bug was originally opened. Is there any blockers left for sorting this out or is it good to go? Thanks!
Hi Luca, The issue is getting from where we are to where we want to be without breaking everything. In other words, it interim steps along the way. Admittedly I haven't thought *too* much about it, but pidof (either one) needs to be installed all the time, without dragging in all of procps. There didn't seem to be a good way of doing it. If you can find this twisty path, I'm happy to hear about it. - Craig
How about, uploaded at the same time: - src:procps with a new procps-pidof binary package that breaks/replaces current sysvinit-utils and with prio: essential - drop pidof and prio:essential from sysvinit-utils and add depends on procps-pidof Shouldn't this work?
There's also the case that packages that have an implicit dependency on sysvinit-utils will have an explicit one. pidof would ideally be built statically, so not to pull in libprocps. There was also the issue about init scripts sourcing init-d-script. systemd unit files don't need this but there are still quite a few like this. pidof is also a symlink to killall5, I assume replaces works with them but not 100% sure. Probably the biggest problem is Trixie freeze, its about 3 weeks away. There was a discussion about this change a while ago, but I'm not sure if we can make it this time or best to wait until after Trixie. - Craig
Hi Debian Release Team, I released probably the best way of knowing if "we have the time" or not is to ask you. So what is this change? It is replacing the pidof in sysv-init-utils with the pidof in procps. This will involve making a new Essential package procps-base which will only have pidof statically linked (to not pull in libproc-2). Then sysv-init-utils would remove pidof and not be marked Essential. There is some talk of in the long-run making packages needing pidof to depend on it, but that is a while off and I'm not sure its possible. - Craig---------- Forwarded message --------- From: Craig Small <csmall@debian.org> Date: Tue, 25 Feb 2025 at 20:24 Subject: Re: Bug#810018: New Essential package procps-base To: Luca Boccassi <bluca@debian.org> Cc: <810018@bugs.debian.org> There's also the case that packages that have an implicit dependency on sysvinit-utils will have an explicit one. pidof would ideally be built statically, so not to pull in libprocps. There was also the issue about init scripts sourcing init-d-script. systemd unit files don't need this but there are still quite a few like this. pidof is also a symlink to killall5, I assume replaces works with them but not 100% sure. Probably the biggest problem is Trixie freeze, its about 3 weeks away. There was a discussion about this change a while ago, but I'm not sure if we can make it this time or best to wait until after Trixie. - Craig
I've now got a git branch on salsa showing the required changes and to see how it went getting built. https://salsa.debian.org/debian/procps/-/tree/new-pidof - Craig
- it will pass NEW (has to happen anyway). The earlier the better, especially with the freeze coming. - it's easier to try out Chris
On Wed, 26 Feb 2025 12:49:29 +0100 Chris Hofstaedtler <zeha@debian.org> wrote: to see Yes an experimental upload is a great idea and removes the first hurdle, passing NEW. Thanks for prepping a branch for this!
I don't think this is a change that should be done this close to the freeze. Please stage this change in experimental if you wish, but let's postpone it until forky. btw, the transition suggested by Helmut on [1] sounds reasonable. Cheers, Emilio [1] https://lists.debian.org/debian-devel/2023/11/msg00105.html
Thanks Emilio, I'll wait until after release. Then get on with it. - Craig
and instead of the one in sysvinit-utils The consensus (and probably the only consensus) was to wait until the release happened, which it has so now is the time to decide. The issue is how to get from here to there? Also what is "there", does anything that has pidof need to be essential? Most of the use-case for pidof is that /usr/lib/lsb/init-functions uses it OR init scripts incorrectly use it. Incorrectly for most because they probably should be using pidofproc() (found in init-functions) instead. From a dependency point of view, you get to the same point; needing pidof directly or indirectly for most init scripts. My reading is that you only need init-functions if you're using the init files instead of the systemd unit files. So there's no impact for a host running systemd with everything using unit files if sysvinit-utils is removed. However if both those criteria are not true then you need init-functions in sysvinit-utils and therefore need a pidof from somewhere. Two paths here. Path A: pidof stays in sysvinit-utils then we can keep going with the pidof most other distros don't use and I'll close #810018. Path B: We decide to use the procps pidof. Then there are two questions. 1) Should the procps pidof package be Essential? 2) Should the procps pidof package be separate to procps and libproc2? My preference is for 1) the answer is no. This package would only be needed because sysvinit-utils needs it, so a dependency should cover it. The "main" procps package would probably need a dependency/recommends on it just so pidof is there for users. For 2) I have no real preference. Keeping pidof in main procps is easier for me, but it does mean anything that needs sysvinit-utils will pull in procps and its libraries. Most people would install procps anyway but there might be a subsection that use sysvinit and don't install procps; I have no idea what this number is. The breaking out of pidof from procps would be for this intersection of users. - Craig
Sounds reasonable Start without adding a new package which is operationally easier (no new queue to clear) and see how it goes? It can always be added later, if it turns out it's needed
I had a look into this and it seems the only valid required user of pidof is /usr/lib/lsb/init-functions Some init scripts call this, but they should be sourcing init-functions (most do) and using pidofproc() (some do). My understanding is the init-functions file is required for sysv init scripts only. So: * a sysv init system uses init scripts, which use init-functions, which needs pidof * systemd init system uses unit files and doesn't need init-functions or pidof I'm not 100% sure of that second bullet point. I'd really like that confirmed. Does sysvinit-utils need to be Essential at all? Is it just merely making sysvinit-core depend on sysvinit-utils (it does already) and then sysvinit-utils requires whatever package has pidof? Looking into it more and adjusting Helmut's suggested migration path[1] Is it a matter of: 1) sysvinit-core depends on sysvinit-utils (already does with a version) 2) sysvinit-utils has its Essential tag removed, its being pulled in by sysvinit-core 3) sysvinit-utils depends on and provides virtual package pidof 4) If there is anything that needs pidof but doesn't need sysvinit-utils it also depends on virtual package pidof 5) At some time sysvinit-utils drops the virtual package, doesn't install pidof, procps picks those up Start without adding a new package which is operationally easier (no Looking again at that small intersection. Having procps pidof by itself would mean the (non systemd) sysctl isn't installed. I'm not sure there are that many systems like that. - Craig 1: https://lists.debian.org/debian-devel/2023/11/msg00105.html
The second point is correct The virtual package could also just be skipped, and a dependency utils -> procps simply added, should provide the same results with fewer steps in between?
Excellent, so a systemd init host doesn't need pidof then That would work too and be less things going on. sysvinit-utils/sysvinit-core maintainers, do you agree with this approach? Do you need some dependent bugs opened, or is 826215 enough? I'll wait a while for some replies. I think it needs to go to debian-devel. Policy says it needs to go there for new Essential packages, but I assume for removing Essential tags it needs that too. - Craig
Craig, Many thanks for the time and effort you have invested in this. Surely pidof remains Essential until all such usages are identified and Depends: whatever-pidof-provider added? I think it is also worth considering that the 2 pidof implementations are not completely identical. A brief comparison of the manpages suggests that the separator option is different (-d vs -S) and some of the other options only exist in 1 or other implementation. If they are not direct drop-in replacements, how do we avoid/handle breakage for callers/users? Please be assured that I am not averse to finding a solution to this, but it is a difficult issue requiring considerable work for pretty marginal gains. At the moment, I don't see just removing Essential: yes from sysvinit-utils, shipping pidof from src:procps' and adding the usual Breaks/Replaces as being a complete and adequate transition plan. With best wishes Mark [1] https://sources.debian.org/src/dbus/1.16.2-2/debian/dbus.postinst#L46
Hi, here is a list of packages using pidof in their maintscripts (last updated in 2024): https://salsa.debian.org/-/snippets/621 Andreas (ah) and I started removing pidof from maintscripts in 2022, but we did not complete the removal. I think pidof can be replaced with something else in most maintscripts. After that there will be perhaps half a dozen of packages left whose maintscripts really really need pidof. Pre-Depends, not Depends. A discussion on d-devel@ will be needed. Regards,
Every affected package has now patches or MRs posted to solve the problem, links in the salsa snippet. As far as I can see nothing is left that needs pre-depends, after the patches linked in the snippet above will be applied.
Dear procps maintainer, dear sysvinit maintainers, while we try to reach consensus in #1131136 on the best way to remove the Essential bit from `sysvinit-utils`, I think we can focus on a simpler task: moving `pidof` from `sysvinit-utils` to `procps`. Do you procps and sysvinit maintainers agree with the following plan? The situation: * No maintainer script uses `pidof` anymore (the 4 low-popcon remaining packages will be NMU'd soon; patches ready). * There are 94 packages/programs that execute `pidof` at runtime and not Depends: on `procps`. (Worst-case number: some of these are probably false positives found in dead-code branches.) * The code snippets where pidof is used in these 94 packages can be inspected at [1]. * Pretty much all these 94 packages use `pidof` without options. * The only options used are `-s`, `-x`, `-q`, and `-o` (respectively, 4, 6, 1, and 2 occurrences; 10 affected packages). * Both implementations of `pidof` support these options, with the same semantics/output. The plan: 1. I will announce a MBF on d-devel@, with a request for the 94 packages that use `pidof` at runtime to add `Depends: procps` (the email template can be seen at [2], the dd-list at [3]). 2. We will wait until the 94 packages (or the vast majority of them) has been fixed. Pings will be made, patches will be offered. 3. The severity of the bugs still open will be raised to RC. 4. Inside a single dinstall window, a modified version of `sysvinit-tools` (without `pidof`) and a modified version of `procps` (with `pidof`) will be uploaded. (WIP patches can be found at [4] and [5].) Does this sound right? Is anything missing? Unless you're strongly against it (or you greenlight it earlier), I'd like to propose the MBF on d-devel@ in one week. Regards, [1] https://people.debian.org/~gioele/pidof-usage/pidof-20260312/mbf-runtime/snippets/ [2] https://people.debian.org/~gioele/pidof-usage/pidof-20260312/mbf-runtime/mbf-text.txt [3] https://people.debian.org/~gioele/pidof-usage/pidof-20260312/mbf-runtime/dd-list.txt [4] https://salsa.debian.org/gioele/sysvinit/-/compare/master...pidof-remove?from_project_id=31168 [5] https://salsa.debian.org/gioele/procps/-/compare/master...pidof-enable?from_project_id=10120
I thought the Essential part had to happen before the switch the pidof part. Looking at this plan, it seems this part could be done first. OK, when I checked a few months ago there were quite a lot. Most of them should have been using pidofproc but they were present. That means a lot of maintainer scripts changed recently. They're reasonably simple calls here. * Both implementations of `pidof` support these options, with the same Agreed, I think -z is the main difference now. I'm not 100% if that option is useful anymore. The plan: That seems ok to me (procps maintainer). It does sound right, but I have a nagging feeling something is missed. - Craig
Hi Craig, Removing Essential first was needed before we got rid of pidof in maintscripts. Now we can deal with that later. The last "wave" of removals was three months ago. There are still occurrences of `pidof` in various maintscripts, but they are all guarded by `command -v` and similar constructs, so they turn into NOPs in case `pidof` is not available. And fortunately `-z` does not seem to be used by any Debian package. Nevertheless local scripts by our users may be impacted. We will need to write a NEWS entry and a release note warning about this change of implementation. (I forgot to put this in the plan.) Thanks! Me too. :) Any transition like this is bound to break something. I think all the preparatory work that has been done will greatly reduce both the chances of this happening as well as the blast radius. Regards,
Hi Guillem, [Addressing only this aspect of the thread in this post - I think it can be considered independently even if its execution helps with other aspects.] Thanks for this offer. I wonder if this might well be worth doing if, as it sounds, it is a low effort, low footprint change that simply allows any patch to swap out pidof usage with a capability that will always be available regardless of all else that happens in this transition? In the pecking order of resolutions to improve an initscript this would not be the first one I would go to but it would be handy to have in the arsenal and above adding a dependency on the 2M procps. Would this still be agreeable? (And these suggestions would be somewhere in the middle of the pecking order, below rewrite with init-d-script (the best), using pidofproc having defined a pidfile, and above using the 'new' pidof feature.) Thanks, Andrew
Hi all,
First I would like to thank Gioele, Andreas, Craig and others who have
put a significant amount of effort into analysing the impact of this
transition, contributing changes to packages over a number of years and
considering the effect on users of init systems in which they themselves
have no (at most) interest. I appreciate that consideration and do not
wish to take it for granted!
Mark has suggested I comment on this thread but do please note that my
views don't necessarily reflect his or anyone else's.
This, I think, is the residual work not covered by the transition plan
and is my main subject below.
While I am not convinced that the benefits of this and the #1131136
transitions, such as they are (and they seem largely aesthetic to me),
outweigh the costs, and I do not find the case that there is a technical
imperative in switching to the same implementation as 'everyone else'
compelling, I would like to help this transition succeed and do agree
that ultimately it would be nice for pidof to come from the same source
as the other p* family of tools.
In general I think using pidof and the other p* tools in a script is
inherently flaky so I hope that those of us invested in the d-i-d
ecosystem will use this opportunity to try to improve some of the
affected initscripts not to rely on such unreliable process tracking
approaches but this should not need to be a blocker to the transition if
some coarser-grained defences are also deployed.
Can I just confirm (as it looks to me) that this covers all runtime
usage *except* for
a) direct usage in an initscript, and
b) indirect usage in an initscript via pidofproc where a PIDFILE is
*not* defined or cannot be reliably guessed?
as numerous as I first thought, how should we mitigate them?
Here are some options:
1. [ sysvinit -> procps (global solution) ]
Make sysvinit-(core|utils) depend on procps. (Does not cover runtime
usage like initless containers.) I would strongly object to this because
it is too large a dependency to bring in for all sysvinit installations
or other inits that fall back to initscripts.
* No thanks *
2. [ <package> -> procps (per-package solution) ]
The next quick fix idea would be introducing a dependency from the
affected packages, similarly to the proposed MBF for the other types of
runtime dependency. This would be a reasonable fallback position if we
can't do better.
* Good fallback solution *
3. [ dpkg (partial global solution) ]
With Guillem's kind suggestion from 2023 to make start-stop-daemon's
internal pid-of logic available as a first class operation, the
pidofproc() implementation in init-functions could use this, dealing
with all the category 'b' cases transparently (although since I wrote
this, I see only three affected packages). This could also help with
category 'a' cases with a change in the package needed, or *possibly*
transparently by overriding pidof in the shell, iff agreeable - this
might be more fragile.
* Yes please, this would be really good, possibly additionally to other
parts of the solution. *
4. [ improve initscripts (per-package solution) ]
In increasing order of preference and work:
i. not to need pidof, by making sure an appropriate
PID file is used.
ii. to use pidofproc instead of pidof directly.
iii. to use status_of_proc instead of pidofproc.
iv. to use aggregate s-s-d operations instead where direct usage is
avoidable.
v. to use init-d-script instead.
* Could be a backlog task for d-i-d team, doesn't need to block
transition if package covered by (1), (2) or (3). *
I think option (2) above, dependencies from affected packages to procps,
is sufficient to give a green light to the transition, and hopefully we
can do better than this with (3) and (4) in parallel.
From my initial attempt at a sweep (could be flawed), these are the
affected packages in sid. This includes some that have been removed from
testing.
AFFECTED PACKAGES
(sorry these are scripts not packages - will need to map back)
* 35 scripts calling pidof directly in initscript:
and
apache2
apcupsd
conman
corosync
corosync-notifyd
dhcp-probe
dns-flood-detector
exim4
fcoe-utils
firebird3.0
firebird4.0
garb
mountall.sh
munge
nagios4
nfs-common
openafs-client
pacemaker
pacemaker_remote
pandorafms-agent
powerman
proftpd
radosgw
samhain
scanbd
shoelaces
slurmctld
slurmd
slurmdbd
swupdate
trousers
virtualbox
vsftpd
yaskkserv
* 3 packages calling pidofproc where pidof fallback seems likely
swupdate
oc2b
cyrus-imapd
(7 other packages call pidofproc without specifying a PID file but the
PID file can be determined by pidofproc from the daemon name.)
Hi Andrew, there are four uses of pidof: 1) at runtime (Depends:) 2) in tests run at build-time (Build-Depends:) 3) in autopkgtests (Depends: in d/tests/control) 4) in initscripts (see #1131136) This MBF will focus on 1. Two follow-up MBFs will take care of 2 and 3. (Although there are just a handful of packages in categories 2 and 3. I believe I'll just send patches instead of doing two proper MBFs.) I see your point, but why shouldn't the package that ships pidofproc (that uses pidof in certain cases), not Depends: on pidof? Given that the use of pidof in pidofproc is behind a `-x /bin/pidof` guard, it should also be OK to accept that pidofproc behaves in a different way when pidof is not available. This would avoid the additional dependency on procps. Another alternative would be changing pidofproc's use of pidof into an invocation of grep and /proc. Regards,
Hi Gioele, #1131136 has converged on specifically the removal of sysvinit-utils from the essential set and, with pidof dependence brought forward to this bug (should we add a blocking relation?), only runtime reliance on the lsb helper scripts remains as an obstacle. With the latest proposal (i-s-h) there I hope something that should be good for everyone, we can hopefully move forward on that bug! I agreed with you (although not everyone was convinced) that in that case making init environments responsible for the runtime environment where such scripts are needed would have been a tolerable compromise at the expense of an unknown number of applications of the initless container use case for initscripts, which could in any case never be a guaranteed mode of operation. But that is in the context of a dependency on the lightweight sysvinit-utils. It would be quite different to require a dependency on 2M procps for all installations of sysvinit-core (and other inits) and directly hit the class of use case for which this reduced Essential set exercise ought surely to be targetting, say a lightweight VM. That would really weight on the cost side of the cost-benefit analysis here. If my analysis is correct in my last e-mail, there are only three packages that would actually be affected by the indirect usage of pidof via pidofproc(), so the other <35 packages that directly invoke it are the ones that will need the dependency (my mitigation #2) except in the cases we are able to do something better for that package (mitigation #4 or your final suggestion below). Andrew
Sub-thread aimed at easing and accelerating the transition:
Symlinks are cheap and dh_installalternatives is easy to use.
How about bringing a modified version of step 4 forward to steps 0a and
0b?
0a) We could immediately upload sysvinit-utils with pidof renamed to
/usr/bin/pidof.sysvinit with an alternative of priority 40.
0b) Then we could upload procps with pidof provided as
/usr/bin/pidof.procps with an alternative of priority 80 and an
appropriate versioned Breaks clause.
Then we could more quickly smoke out possible issues and make it easier
for people to play with both tools in parallel.
I prefer this way myself. Also if people really want sysvinit pidof its still there; e.g if procps pidof breaks some obscure thing horribly, they're one update-alternative away from health. - Craig
Please don't. update-alternatives is an old, crufty and problematic system, that wreaks havoc any time you want to have a self-contained immutable vendor (/usr/) tree, given it _requires_ indirections through a writable /etc/ tree. It has no place as a solution to new problems. If the sysvutils maintainers want to keep their implementation around as pidof.sysvinit they can do that, and then have local aliases in their bash profiles or so.
Craig, Thanks for this. I think there is another route to achieving something similar: procps could employ dpkg-divert(1) to avoid a file conflict on /usr/bin/pidof with sysvinit-utils with sysvinit-utils still continuing to ship its own implementation. I see this approach as having some benefits, in particular for some non-systemd and initless installations. In such cases, having to add a dependency on procps at about 2.2M just to provide /usr/bin/pidof appears materially worse than using sysvinit-utils with an installed size of about 100k. Obviously, if a non-systemd or initless installation requires procps then that pidof implementation would then be used. In my opinion there is little to choose between the implementations and I emphasise that I don't particularly mind which is used. My real concern is how it is packaged and the consequences of that, particularly in terms of installed size. I am grateful for your thoughts on this. Best wishes Mark
No, that is not desirable, as the point is to make procps' the canonical implementation, like it is in the rest of the world. So it's the other way around, sysvutils can use maintainer scripts and diversions to ships its own implementation as an alternative for those who want it.
Hi! I just checked now and it was really trivial. I'm undecided whether to go with a new --print-pid command or an option to pass along the other commands to print the PIDs matche. I'm attaching both patches, and will be mulling over what feels like the better interface, before merging it and after adding proper man page updates, for the next dpkg release. Thanks, Guillem
Hi Guillem, Thank you so much for knocking up these patches! I literally just tried the first one, the new command, along with a pidof() emulation function in /lib/lsb/init-functions, as a transparent solution for all existing initscripts and it appeared to work very well! I see the MBF just went through so we can't refer to this approach in guidance but I think we might have a good way through here with this enhancement. Andrew
Hi Andrew, the MBF did not include the packages that use pidof in initscripts. And there is pretty much no overlap between packages using pidof in their initscripts and at runtime, so it is still possible to do a mini-MBF focused on replacing the direct use of pidof in initscripts. Regards,
I see. Yes, fine! I think if we get this lightweight transparent solution up we can avoid another MBF; I would prefer to suggest considered improvements on a case by case basis for affected packages at leisure. Thanks!
Hi! I've been thinking, and the new command makes most sense to me as it has clearer semantics, so went with that but changed the command name to be the obvious --pidof. :D Attached revised patch with documentation now, which I've queued for my next push. Thanks, Guillem
Hi Guillem, Thank you! :-D I agree this version is more intuitive but there was some good conceptual sense in the 'option' version and in the '--print-pid' name, namely that we confirm we aren't, and don't in full, emulate pidof, but providing access to s-s-d's own determination. Anyway, I think it's not worth expending any thought on this - the solution is great! I have tested your latest patch and attach a patch to src:sysvinit that uses this command to emulate pidof for initscripts that call pidof either directly, or indirectly through pidofproc(). Again, thanks for helping out with this capability - it's really nice to be able to solve this transparently without heavyweight extra dependencies. I hope that I and others, over time, can contribute improvements to specific initscripts that call s-s-d in more idiomatic ways and, preferrably, find that in a simplified form they don't need to do this at all! Andrew