A recent bug#1101304 shows some limitation of version handling of
dh-elpa for packages that are both built-in and packaged separately.
Currently, dh-elpa can mark a package as packaged separately, and then
detect the version from an addon comment header, much like non-built-in
packages.  However, when a built-in package is sufficiently new, an
addon can directly depend on Emacs itself without requiring the
separately packaged version.

As an improvement to avoid requiring the separated packaged version, for
a package `foo' that is built-in, if the packaged version meets the
requirement of an addon, ${elpa:Depends} can just generate "Emacs (>=
<current version>)"; otherwise generate "elpa-foo (>= <required
version>)" as before.

Implementation wise, AFAIK there is no public API to query the version
of a built-in package, so for now one may need to inspect
`package--built-versions' directly.  Not sure whether upstream would
implement an API for querying built-package versions.

Hello,

Right.

There are a couple of options here:

- Emacs adds versioned Provides: and then it satisfies the dependencies
  without the dependencies getting more complicated

- We generate dependencies "elpa-foo (>= ...) | emacs (>= ...)".

I think I prefer the former, because it retains the possibility of not
installing Emacs along with addons, an invariant we've had for a while.
But what's your opinion?

I think they probably would, if you'd like to ask about it.

Sean Whitton <spwhitton@spwhitton.name> writes:

Option 1 also sounds good to me.  This may also be useful to handle
security update for stable version, though I'm not sure whether it will
automatically handle the upgrades.

(Wondering for the situation that Emacs version A provides foo 1.0, and
when foo 1.0 founds a security issue, and Emacs version A+deb12u1 fixes
it by providing foo 1.1, will foo 1.0 automatically upgrade to Emacs
A+deb12u1, or it still requires user manually uninstalling foo 1.0?)

Sounds good.  Will propose to upstream.

control: reassign -1 src:emacs
control: retitle -1 emacs: generate versioned Provides for all :core packages

Hello,

Cool.  I'm not planning to work on it but I can review patches.

No, and in fact the foo with the security hole would override the one
shipped with Emacs.  We'd have to patch both packages.

Sean Whitton <spwhitton@spwhitton.name> writes:

Ack.  Sigh.

Sean Whitton <spwhitton@spwhitton.name> writes:

I have made proof-of-concept patches as a pseudo-MR at [1] (this MR is
against my own fork just for illustration).  Basically, it uses an Emacs
script to generate Debian compatible versions from the builtin package
list, which is integrated with `debian-sync' in debian/rules (similar to
the handling of debian/copyright.in).  I'm currently making the provide
list in emacs-common package which ships the compiled elisp files.

Please review and comment.

[1] https://salsa.debian.org/manphiz/deb-emacs/-/merge_requests/1

Hello,

Very interesting.

Have you considered using debhelper substvars for this?  Like how we
have ${elpa:Depends}.  It avoids needing a control.in.  A control.in
isn't the end of the world but it might be cleaner not to have one.

Don't sharpquote lambdas; unlike in CL, doing so will mean they don't
get compiled (I think).

You can rewrite the cond using cl-case or pcase.

Use cl-incf (just incf in Emacs 31) instead of setq and 1+.

I see you're using package--builtin-versions.  I'm not happy to be using
an internal variable -- it got us in trouble before!

Sean Whitton <spwhitton@spwhitton.name> writes:

Ah didn't think about that.  I think it's better than using control.in
in general, though there are the "unused" warning on other packages
(using "?=" doesn't seem to help).

One advantage of generating d/control that I like is that we can see the
provide list directly and can inspect the diff on regeneration.  To get
something similar, I have each package version pair as one line of
comment in the substvars file before setting emacs:Provides (the
substitute variable I'm using for now).  This way we get something
similar.

Somehow I cannot use debian/substvars or debian/<package>.substvars
(which probably got removed by "clean") and need to use
override_dh_gencontrol to pass the generated substvars file (I'm using
debian/emacs-substvars).  Let me know if there is a better way.

Ack.  TIL.

Done

Done

Current there seems to be no way to get the builtin package info
otherwise.  Fortunately we can easily verify whether it still works by
running "debian/rules debian-sync" and inspect the diff in
debian/emacs-substvars.

Meanwhile I'll try to file a wishlist bug upstream and migrate in
future.

I have updated my branch accordingly.  Also attached the formatted patch
in case it's preferred.

PTAL.

Hello,

I don't follow.

True.

I don't know how to do it off the top of my head, either.  But I'm sure
it can be done.  Look at how dh_elpa does it, I guess?

I'm not up for merging that, I'm afraid.  I think we need a public
function first.

Sean Whitton <spwhitton@spwhitton.name> writes:

Excerpt from build log (basically it shows a warning of substitution
variable unused for all binary packages except emacs-common where it's
used.)

,----
| ...
|    debian/rules override_dh_gencontrol
| make[1]: Entering directory '/build/reproducible-path/emacs-30.1+1'
| dh_gencontrol -- -Tdebian/emacs-substvars
| dpkg-gencontrol: warning: package emacs: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-pgtk: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-pgtk: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-lucid: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-bin-common: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-lucid: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-bin-common: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-nox: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-nox: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-gtk: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-el: substitution variable ${emacs:Provides} unused, but is defined
| dpkg-gencontrol: warning: package emacs-gtk: substitution variable ${emacs:Provides} unused, but is defined
| ...
`----

I think dh_elpa generates the substvars files during the process
(definitely after dh_auto_clean) so that it's not affected.  The current
work around is not too bad, though.

I have filed the bug upstream[1], but that would be for 31.x.  I was
hoping this would be useful for Trixie to handle security updates,
though it may have already been a bit too late for that.

[1] https://debbugs.gnu.org/78844

Hello,

Ah, I see.  Could you define it as the empty string for those packages
and then add a dummy entry to Provides ?

I intended to say, once the interface is settled upstream, I would be
happy to backport it for 30.1 in forky, so we can deploy your change
before the release of 31.1.

Sean Whitton <spwhitton@spwhitton.name> writes:

It turns out debian/substvars and debian/<package>.substvars are indeed
regenerated during build.  So I figured I can just update
debian/emacs-common.substvars with the generated contents right before
dh_gencontrol, and it worked.

Updated in my branch[1].  Hope the builtin package info public
interfaces will be accepted upstream soon.

Ah that's good to know.

[1] https://salsa.debian.org/manphiz/deb-emacs/-/compare/deb%2Femacs%2Fd%2Fsid%2Fmaster...generate-provide-package-list?from_project_id=83361

Xiyue Deng <manphiz@gmail.com> writes:
the direction has consensus, while the details of the functions are
still under discussion.

In the meantime, I wonder whether we can move forward on the Debian side
and have this in Trixie (with an unblock request), which will help users
who upgrade from Bookworm avoid surprises when older versions of addons
are installed.  My branch[2] is updated to the latest version of
upstream bug.  Patches are also attached.

[1] https://debbugs.gnu.org/78844
[2] https://salsa.debian.org/manphiz/deb-emacs/-/compare/deb%2Femacs%2Fd%2Fsid%2Fmaster...generate-provide-package-list?from_project_id=83361

Hello,

I don't think the full change is appropriate for trixie, and I would
like to see the upstream changes committed to Emacs 31 before
backporting them here.

I think though that we could add the necessary dependency relations
manually to deal with potentially broken upgrades.  There is no need to
do every built-in package, right?  We just need to add values for those
built-in packages that were RM'd for trixie?

Sean Whitton <spwhitton@spwhitton.name> writes:

I think there are values for including other packages which helps with
security updates.  For example, we have Org 7.9.11 shipped with Emacs
30.1; later if a security bug is found in 7.9.11, and Emacs 30.2 ships a
newer version of Org to fix that, this gets automatically taken care of
by this.

Hello,

We would have to upload security updates for both packages anyway,
though.

Hi Sean,

Sean Whitton <spwhitton@spwhitton.name> writes:

Yes.  Still, if we get this now, there's less work in future ;)

Xiyue Deng <manphiz@gmail.com> writes:

FYI, upstream merged my proposed patch in bug#78844[1].  I'd still
advocate for its inclusion in Trixie to make the upgrade experience
smoother for dropped packages (e.g. eglot, project, etc.)  Though I do
acknowledge that we are already in the late stage of the Trixie release
cycle, and it's probably safe to maintain the status quo.  If the latter
is chosen, I wonder whether this may still be a good candidate for a
future stable update, or only be suitable for a backport?

[1] https://debbugs.gnu.org/78844

Hello,
Team's freeze policy says we should upload only targeted fixes.  And the
Stable Release Managers only allow fixes for bugs of Severity: important
or higher for stable updates, but this bug is wishlist.

But I think we can do the following:

- Let's start preparing the versioned Provides generation in
  experimental.  Assume I'l backport your patch in #78844 to Emacs 30.
  Can you prepare a patch implementing the Provides generation?

- If you want to work on it, we can propose adding manually generated
  Provides to trixie.

Sean Whitton <spwhitton@spwhitton.name> writes:

Acknowledged.  I thought there was a bug report from Stefano about the
upgrading issue regarding Emacs backports, which turns out to be just an
email to debian-backports[1].  That said, Cyril seemed to support this
idea for a better upgrade experience, though not officially as a RT
member.  As the issue is real, maybe we can file this as an important
bug? (It would sound a bit like tricking the system though but not
really :P)

Still, I'm also OK with postponing this to Forky to avoid causing
additional issues.

The patches I attached to [3] implemented this (as well as in my
branch[4] which could be newer).  It didn't backport bug#78844 in the
Emacs source, but host identify copies of the new functions in the
provides/breaks/replaces generation code (which are guarded by fboundp
so that the Emacs implementations will be used when available in Emacs
31).  I think this has the advantage that we won't hit any conflicts
when upgrading.

I think you meant Forky here?

[1] https://lists.debian.org/debian-backports/2025/06/msg00002.html
[2] https://lists.debian.org/debian-backports/2025/06/msg00012.html
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103033#71
[4] https://salsa.debian.org/manphiz/deb-emacs/-/compare/deb%2Femacs%2Fd%2Fsid%2Fmaster...generate-provide-package-list?from_project_id=83361

Xiyue Deng <manphiz@gmail.com> writes:

I just realized that you have already done the backporting of #78844.
The patches still work (as it's guarded by fboundp), but do you want me
to remove the vendored functions?

Hello,

No -- my suggestion is to add manually generated
Provides/Conflicts/etc. to trixie to fix this problem.

Hello,

Yes, please prepare a patch against current debian/d/sid/master.

Sean Whitton <spwhitton@spwhitton.name> writes:

Updated my branch by dropping the vendored functions[1].  Also attached
the updated patches.

[1] https://salsa.debian.org/manphiz/deb-emacs/-/compare/deb%2Femacs%2Fd%2Fsid%2Fmaster...generate-provide-package-list?from_project_id=83361

Sean Whitton <spwhitton@spwhitton.name> writes:
in my updated patchs in my other email), or were you expecting the
provides/replaces info directly put in d/control (like my earlier
approach based on d/control.in but only keep the generated d/control)?

Hello,

I mean the latter.

Hello Xiyue,

This is great work.  Thank you for the respin.  I'd just like to ask a
few questions about it.

- What is the emacs-provided-package-versions variable for?  I don't
  think it gets set anywhere?

- Why pregenerate emacs-common-substvars?  Why not generate it during
  the build using the version of Emacs we just built?

Sean Whitton <spwhitton@spwhitton.name> writes:

Ack.  Please see the patch attached.

Hello,

I don't think it needs to be a defvar.  It can be a lexical variable
inside that function.

Why not generate it during the build?  That's how other substvars work.

Sean Whitton <spwhitton@spwhitton.name> writes:

It's used later in the `emacs-provided-package-versions` function, which
will populate it with a mapping of package name to version.  This will
then be used later to actually generate the substvars values.

I put the list of package and version provides in comments in this file
so that we can keep track of package addition and version change on each
Emacs update easier.  The actual substvars list everything in one line
so it's harder to check for diff.
the build rule is marked `.PHONY'.  I think this should be run on each
update according to d/README.source.  Maybe this could be added to
dh_auto_clean in case someone (like me) forgets to run it by hand?

Sean Whitton <spwhitton@spwhitton.name> writes:

Ah, it was an optimization so that when we need the mapping multiple
times it doesn't need to be regenerated everything.  It looks like now
we only need to get this mapping once, so I have made it a lexical
variable now.  My branch is now updated (patch also attached).

This makes the development easier so that I don't have to sbuild it
every time to test (which takes ~20min for me).  Also, running the
script needs a working emacs executable (I'm using `/usr/bin/emacs')
which is not available yet during build (or it can some built emacs
executable in some build-path, but I think it's a bit too fragile).

And I realize that because I removed the vendored lisp functions to get
the built-in package information, the script currently cannot work until
Emacs with the backported package.el is released.  So probably we should
upload -6 first (in experimental)?

Hello,

That wouldn't be fragile, and it's what we should be doing.  We don't
want to use /usr/bin/emacs.  We want to be sure we are using the latest
information, from the thing we are building.

This would also be solved by using the Emacs that's part of the build.

Sean Whitton <spwhitton@spwhitton.name> writes:

Well the trial-and-error took a while due to build time (which was why I
preferred external emacs at first :P).

Now it uses the emacs-nox binary in the build tree to generate the
built-in package info.  I have some concerns about this as this now
depends on the upstream source code layout structure to find the built
emacs executable and it could break if that changes, though on the other
hand this layout change may not happen much in the foreseeable future ;)

As a side note, I also tried to use the emacs executable from the
install trees which I think the location should be stable, however the
`override_dh_auto_install' will cleanup the install trees at the
beginning as well as at the end, so the install trees are not available
outside of that build rule - which is probably by design.  Maybe in the
future we can open this up for customization like running this script
before clean up.

Anyway, my branch is now updated and the updated patches are attached.

Hello Xiyue,

Thanks.  I think you sent the wrong patches but I had a look at your
branch on salsa.  This still isn't quite what I mean.  I think that

- you should be using the Emacs built binary (as you are now doing)
- we should not commit the file emacs-common-substvars to git.
  Instead it should be generated during the build.

So the main thing missing is that the generating of the file should be
incorporated into the build.  Probably it should go into the existing
override_dh_auto_install.

Hi Sean,

Sean Whitton <spwhitton@spwhitton.name> writes:

Oops, sorry.

I understand that this file is mostly useful during package building.
Still, I'd like to give another try to propose to have
debian/emacs-common-substvars committed in git: this will help with
debugging in the following scenarios:

* When we want to understand why some packages are (or not) being
  replaces by Emacs, and

* When we want to see what packages are updated between Emacs releases
  (which is also why I put a package+version per line in the comments).

Without this file, we need to get this information from `apt show' or
from https://packages.debian.org/ which would be very cumbersome and
hard to get it right.

Wdyt?

Hello,

I don't mind also committing the information in a text file somewhere
(with a version annotation), but the substvar generation should be
wholly dynamic.

Sean Whitton <spwhitton@spwhitton.name> writes:

Indeed.  I also realized that the file generated during build only
exists during build, so the version information file needs to be
generated outside of building (like in debian-sync), so I'll leave that
for now.

I have now updated the generation script and introduced arguments to
switch output format.  It now supports output as substvars or in JSON as
information, and debian/rules is updated accordingly.  I have also
removed the debian/emacs-common-substvars now that it's only used during
build.

For the JSON information I'll send another set of patches after
1:30.1+1-7 is released (as it requires the backported package
functions), and integrate it in `debian-sync' rule.

My branch is updated accordingly (now 3 commits)[1].  Patches also
attached.

[1] https://salsa.debian.org/manphiz/deb-emacs/-/compare/deb%2Femacs%2Fd%2Fsid%2Fmaster...generate-provide-package-list?from_project_id=83361

We believe that the bug you reported is fixed in the latest version of
emacs, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1103033@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Sean Whitton <spwhitton@spwhitton.name> (supplier of updated emacs package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Fri, 25 Jul 2025 09:37:19 +0100
Source: emacs
Architecture: source
Version: 1:30.1+1-7
Distribution: experimental
Urgency: medium
Maintainer: Rob Browning <rlb@defaultvalue.org>
Changed-By: Sean Whitton <spwhitton@spwhitton.name>
Closes: 1103033
Changes:
 emacs (1:30.1+1-7) experimental; urgency=medium
 .
   * Add & run script to generate Breaks/Replaces/Provides for builtin
     packages during the build (Closes: #1103033).
     Thanks to Xiyue Deng for the script.
 .
   * Backport upstream commit adding builtin packages query interfaces.
Checksums-Sha1:
 f8922ccdba533a3898deb8b475e6a2dc6a5c91f3 3087 emacs_30.1+1-7.dsc
 30b9818caaac68ed4c3ecc916852aedb61458f52 70564 emacs_30.1+1-7.debian.tar.xz
Checksums-Sha256:
 f7d7d5025bcdccb41b3fd75c759f762a4e33309e6fbcf37c43ae9b17942cf289 3087 emacs_30.1+1-7.dsc
 197126da3ca5af8632e443bafc811999066b2350f793e0e6b0eadff8504d9a39 70564 emacs_30.1+1-7.debian.tar.xz
Files:
 7bd65c8ff72538631b48fcc4bdfee7c6 3087 editors optional emacs_30.1+1-7.dsc
 b20ad33209e4acd060d28d8a2540df4c 70564 editors optional emacs_30.1+1-7.debian.tar.xz
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmiDQpAACgkQaVt65L8G
YkBywRAAkKsaHIRZeZpIP/G0kfESfIPUQDMaLxeY1UY8KTyNhJ27JglMzob0eRlq
uzW8H/Zkd6gFyTfzsEI0PluletB1X4ftSOPByvy7buf4ZKOF+/6j7XPn/2Hgugrp
5QR2JJJMXASAmMENAy2uQF/XkYpmn/Z6G32KW11WalEZMuGveGusQlz0bRQwYQ2n
K64YP42qbmnPFAp0zsAS2lzeKsFPOGmtvF/YQoXYIQtZCFjT+s+EOkdKSa9s6Rha
SV1bXaoyTZuB0zQMMpthoacq1SIFKZBZtf+vxSu9kft8w9SN5/CT2w6c4SQQSfVV
vLuCFERP4vwPD1gomvud9ADKVj9Nz+ToW9fV0BeA78GKA0D0IkIfcfhStN5VTPVy
nxG6I1HQd5JKAfFzqUdlvmQDRmkdYF5qc6vdcZedapXZsxEa1HR0zRSdSEOLnY7Z
X2P1rjjTRrmVZMqMBWBcmPfG0jClly4RW7vVSXgV8IsR0CypG7tVZQXcmPLubJvb
AFEIoRIVDMiVcEJTKi8qnsqCr91pfEC43y2n/xqhgy6YmbsZVlUdJZJCpWIgyrNf
zZYnY3O14s4PsHCHlIDgIVFS5hdHtU6wXdGoeD5RCYz+ahqETuxjxFnKbq5PmXgz
ObZahn+Nvj9Ly9AI721nYJo4ERwxRT88m4dcY2xsKlFAUEltlBs=
=rYIX
-----END PGP SIGNATURE-----

We believe that the bug you reported is fixed in the latest version of
emacs, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1103033@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Sean Whitton <spwhitton@spwhitton.name> (supplier of updated emacs package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Sun, 10 Aug 2025 10:21:45 +0100
Source: emacs
Architecture: source
Version: 1:30.1+1-8
Distribution: unstable
Urgency: medium
Maintainer: Rob Browning <rlb@defaultvalue.org>
Changed-By: Sean Whitton <spwhitton@spwhitton.name>
Closes: 1103033
Changes:
 emacs (1:30.1+1-8) unstable; urgency=medium
 .
   * Upload to unstable.
 .
 emacs (1:30.1+1-7) experimental; urgency=medium
 .
   * Add & run script to generate Breaks/Replaces/Provides for builtin
     packages during the build (Closes: #1103033).
     Thanks to Xiyue Deng for the script.
 .
   * Backport upstream commit adding builtin packages query interfaces.
Checksums-Sha1:
 b25373326d885bf327e566f5866d94c650dd3280 3087 emacs_30.1+1-8.dsc
 3fdf7c52e1fd43b11928e0bbef3f47c7287cb4dc 70584 emacs_30.1+1-8.debian.tar.xz
Checksums-Sha256:
 fc1fac3e4f7462d20a82220043684984f890868630d232afdf616d41de813e24 3087 emacs_30.1+1-8.dsc
 ebf0ebf9176de2cfce37827e1aee7cc2888861b59a2d0b96d73df6b5fbb109b5 70584 emacs_30.1+1-8.debian.tar.xz
Files:
 3e432bb6fa140383dcfd09213a1f1e36 3087 editors optional emacs_30.1+1-8.dsc
 27de946fa2af9281951652395275833f 70584 editors optional emacs_30.1+1-8.debian.tar.xz
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmiYZYcACgkQaVt65L8G
YkDG0w/+JujmzIVpLeQ0E5QAMzB3Qb5kZlxROtwiHwNQmQVcrueRTDOez83Ccwlx
+TaVKhZQaBME1TZiRJopFaz+l7h6jURqlqLfVuWE+oC+YmnrI3/W+w5FHHlBDNXU
r5RUsUpb2yC4u9zz/kqZiFOuJVpyNuhp4x0wDtU1xAHggMSN8Q/UpGqjwif07Bxu
UIMX2gk3vJvTwGU/cRZQTHkmDQseCLCy0MJzYFgvJU39N9uCm4ZxEtzIEdMao9fC
DLZb1bPsHtlC5Rt7S0/sc9zyjEPJgORXalbaSEkTF3HJedcKuRwZp9hdxMkMzhAA
+HCn0C2pPU/BUtb224KpdylfgcBUBacR+eAG7mA8wbZBqOWhEOyd0ebcY29Qrrdb
l1XLp9YfxczJclxswzDppfZ2IIVSwf9Y42VskXNrJcJRk9NA4evG5F1SkVpnp67Y
gFDO6h5A8Wo0yzvJayZ6/e8BIK7JohAQH2PT0U0BT7DzvcipRbPH52z4Q1nst+Gm
YhAgsq6rhh5liU2uHSbyEKMe25GdY+caeSXDvETh40/YZzadiOxIcnJtBs3xrwMN
hxnM4zwPhac1vR9rMuaVo0HHqYNXDWZN7Z98ol8kHs1ZJhvncHcrt/oFQ1UNWQP1
meuVFlVmBZLcNfcAcuht1cTjDWaC3d+0novIZesVquqZvMnyrxU=
=M7Qc
-----END PGP SIGNATURE-----