Hi Filing this as a bug for discussion as requested in https://salsa.debian.org/dgit-team/dgit/-/issues/7#note_520035. I would prefer to keep the salsa issue open in parallel, since I can update the proposal in place there to have the final form easily discoverable in case of amendments. I will submit the first version by email once this bug has been accepted. My previous attempt to send (a variant of) the proposal to mailing lists was silently dropped by debian.org (neither the List nor Sean seems to have received it) and vocally by Ian's SMTP server as spam. By splitting the two, I am hoping you will all get the notification and we will have the discussion bug even if the full text gets stuck. Best regards, Niels
As promised, here is the initial version of the content of the salsa issue.
# Content
I am looking into supporting features that involves generating or
enriching certain packaging files automatically. Notably
`debian/control` but in theory also `debian/tests/control`. I noted
that `dgit` generates its `debian/tests/control`, which is under some of
the same constraints as `debian/control` in practice, so I presume
support for this would also have some merit for you as well. At the
moment, my proposal is a high level idea so there is less concrete "how".
While doing the high level design, I remembered that `dgit` might be
adversely affected, so I figured I would reach out to you to hear what
your views are on it and where we can meet between what I want and what
`dgit` expects/wants. To keep things simple, I will only focus on
`debian/control` in the text below.
# Purpose
My motivation for this proposal is to enable us to enable to
tool-assisted packaging. This is not a novel idea; the `debdry`
prototype back in 2014 tried to achieve something similar (though with a
different approach than I what I think we will land on for this proposal).
My go to example is the `Section` field, for which the contributor is
expected to research which section the package should go in and then set
the `Section` field accordingly. Then you run `lintian` or `debputy
lint` and they will tell you "According to the package name, you should
have used `Section: foo` instead" and you get to correct the file.
Similar, for some language specific packaging, we have the tooling to
provide defaults for a number of things like description or
Build-Depends (which was supported by the `debdry` prototype).
I want to flip that on its head and have our tooling provide defaults
where possible. The human contributor should only spend effort on things
the tool cannot provide, or for which the tooling has a bug, or where
the package is too much of a special-case for the auto-detection to work.
That is the conceptual level of what I want to do and why I want to do
it. Note this is very much a "UX" (User eXperience) problem for me and
not a "It is not possible to do X"-problem. Our tooling could be a lot
better at assisting us which packaging (it has the knowledge/data), but
we as a project have painted ourselves into a corner where we made it
hard for us to use our tooling for it. So an alternative phrasing is
that I am looking for ways to paint ourselves out of that corner.
## Practicalities
One practicality that is relevant to understand early is that no matter
how this is sliced or diced, we end up with a (partial) generated
`debian/control` file for any package that would leverage the features
of this proposal.
Personally, I am strongly leaning towards a solution where the
`debian/control` ends up 100% generated for packages leveraging this,
since having `debian/control` being its own template gets complicated
and in all solutions I have found so far, there will be a delta between
the `git` tree and the generated `source package` (we are "just"
discussing how that delta looks and how it materializes) due to the
constraint 3 (see below). As in example, it is possible to generate an
augmented `debian/control` file (like `debian/control.full`), however
the `git` repo would not have `debian/control.full` while the source
package would have (so the delta between git and the source would remain
conceptually; we just move it to a different file).
In the text below, I will use `debian/control` as a generated file for
simplicity in wording.
# Constraints / Design criteria
I have the following constraints/design criteria in mind that I am
hoping to achieve or that I need to satisfy (in no particular order at
the moment):
1. `dpkg` will ideally provide the hook to do the generation, but I do
not think it will do the actual generation itself (that would be
dispatched some how).
Note: This is not aligned with the `dpkg` maintainers yet (I will
have that conversation separately when I am ready to discuss the "how").
Though a minimal prototype of this proposal can be done by
`dpkg-divert`'ing `dpkg-buildpackage` and `dpkg-source` with a small
script the generates `debian/control` if absent and uses `trap` to
remove the file again if generated (add some conditional logic to ensure
it only triggers for relevant sub commands, etc.). Therefore, I do not
see it as infeasible to have `dpkg` provide the conceptual feature. It
is a question of "how early" to generate/runtime patch the file rather
than "is it possible to generate/runtime patch the file?" (and do the
`dpkg` maintainers agree to the concrete way of doing it).
2. Backwards compatibility. I see three major aspects here
a. Anything working with a git checkout that does not involve
calling `dpkg-buildpackage` (or `dpkg-source`) or that might need to
read `debian/control` before invoking the tools. These are tools like
`debputy lsp/lint/reformat`, `cme`, `lintian-brush`, and `apt-get
build-dep`. I think `gbp import-dsc` and `dgit` might also end up here
to cope with the changes for different reasons.
Per definition these would have to be updated to cope. Though
what is really changing is that we formalize the ability to generate (or
runtime patch) the `debian/control`. There are already packages today
that generate their `debian/control` and most of these tools would now
be able to automatically detect this and work with such git checkouts.
Also, the roll out will de facto be done slowly over time in an
opt-in fashion, so it is not like they will break everything immediately.
b. Anything only looking at the generated package. These are tools
like `dak`, `lintian`, archive-qa tools, etc. These should work out of
the box (this is a side-effect of the requirement of contraint 3).
c. Anything that is involved during a package build (think
`debian/rules` and below). The solution here should ideally also be
backwards compatible as it is infeasible to have all those tools
changed. This is a choice of optimizing for adoption and because
`debian/control` must be materialized for `dpkg-source` anyway (see
constraint 4). Though, I am the maintainer of the primary packaging
helper tools, so if needed be, this design constraint can be dropped if
moving the full version to a separate file is necessary.
3. The `debian/control` must remain stable during build and during the
life-time of the package in the archive. This requirement comes out of
the FTP masters reject FAQ as authority but there many other reasons why
this is a desirable property (determinism, debugging, etc.)
This is one of the key reason why I want the source package to
include the full form of `debian/control`. Existing solutions involve
`debian/rules` failing if `debian/control` (which is extra work to write
and maintain that invariant). Including the full generated form and
avoiding generation for uploaded packages will both solve this
constraint but also simplify existing solutions.
4. Certain information must be available at `dpkg-source` time such as
`Priority`, `Section`, `Testsuite` (relevant if auto-generated
`debian/tests/control` is to be in scope), and the synopsis
(`Description`). Therefore, it is a non-option to move this logic / data
into build-time packaging helpers like `debhelper` that add extra fields
via `dh_gencontrol`. That would simply neuter how much assistance our
tooling can do to the point where this proposal would not change the
status quo.
5. The solution should **not** involve committing a generated or
augmented version of the `debian/control` to git.
I know it is possible to write code the validates the
`debian/control` is out of date (etc.), but I would prefer not having
people thinking about this as that becomes its own UX problem, because
then we have to introduce special-case CI/pre-commit logic that people
have to know about and validate works as intended.
**This might be the controversial part for `dgit`.**
My rationale here is that generated artifacts do not be long in
`git` and it would enable the `debian/control` to go out of sync with
the source (which is an error condition I would like to avoid).
6. When we get to the how, I would like the solution to be generic
enough to support package specific generators provided by the source
package (git checkout) itself, to support our existing `debian/control`
generators.
They would need to add themselves to the relevant hook and maybe
change how they are triggered (etc.). Obviously, the template source
(regardless of form) would belong in git in my view.
This implies that I want to support generating `debian/control` not
being its own source/template.
## Solution sketch
To avoid this being **too** abstract, here is a short conceptual
solution that demonstrate how the above solution could be implemented. A
minimal prototype could be done by dpkg-diverting dpkg-buildpackage with
the following pseudo code:
#!/bin/sh
can_generate_dctrl() {
... # Detect if the package opted in to generating `debian/control`
}
generate_dctrl() {
... # Dispatch to relevant generate
}
if [ ! -f debian/control ] && can_generate_dctrl ; then
trap 'rm -f debian/control' EXIT ERR
generate_dctrl > debian/control
fi
/usr/bin/dpkg-buildpackage.real "$@"
exit $?
A similar prototype could be done for `dpkg-source` for consumers that
run `dpkg-source` directly (with a bit more effort).
This prototype works from the assumption that `debian/control` once
generated is the "leading" form. This trivially satisfies Backwards
compats and the stable `d/control` (FTP Master auto-reject) criteria.
Though as stated, we can do alternatives as well if that would enable
`dgit` to support this work flow.
# Your views on this ...?
Thanks for reading through this and I hope you can help me with the
following:
1. Is this problematic for `dgit` to support?
2. Assuming it is problematic, where can we meet to have it palatable
for `dgit` while still achieving the principle goal that I want to solve?
a. The "no generated artifacts in git" is one I feel strongly for.
While I assume it would trivially resolve any concerns from the `dgit`
side, I am hoping we can come to a better solution without that one.
Like, what would it take for `dgit` to support this without waiving that
constraint?
b. The "debian/control must be stable" (FTP-master auto-reject)
constraint is not in my control to change and does not seem feasible to
change either, since the consequences would be unclear for the archive
at large.
Niels Thykier writes ("Bug#1079762: dgit: Supporting generated files (`debian/control` and `debian/tests/control`) not commited to git"):
Thanks. These are intereting ideas. I think I need to digest your
message properly. I certainly think I can provide some helpful input.
I hope you don't mind that it may take a little while for me to get my
thoughts together and write them up. Please LMK if I'm taking too
long.
Ian.
Ian Jackson: Thanks for considering. :) Certainly. Do you have an idea of how long you would use for the digestion or just an `X` for a "ping me after X days if I have not replied, because then I probably forgot about it"? Best regards, Niels
Niels Thykier: Hi, Did you have time to review the proposal in full or would you like a bit more time to consider it? :) Best regards, Niels
Niels Thykier: Ping? :) Best regards, Niels
Hi dgit maintianers, Any news on proposal? Best regards, Niels
Hello, Thanks for working on this. We're certainly interested in packaging UX improvements -- that's much of the point of tag2upload. My first thought is that dgit already has the notion of the maintainer view of a package vs. the dgit view of the package. The former is whatever the maintainer has checked out while preparing their uploads. The latter is what you get with 'dgit clone'. So we could have the maintainer view contain the human-editable file and the generated debian/control would appear in the dgit view. The dgit view is meant to match the archive. I.e., it's got exactly the same contents as the source package. So we would want debian/control committed to it. Is that the sort of committing-of-generated-artifacts you object to? Or is it okay if the maintainer never sees it?
Sean Whitton writes ("Bug#1079762: dgit: Supporting generated files (`debian/control` and `debian/tests/control`) not commited to git"):
(quilt)" .dscs also commit output files, namely debian/patches.
In those workflows the primary representation of the patch queue is in
git commits, and the in-tree version is there for compatibility with
dpkg-source.
git-debrebase does this when used with "3.0 (quilt)". I believe that
some other tools like git-dpm and git-debcherry do too.
ISTM that debian/control could be thought of similarly.
One key question is: what happens if you start with transformed view,
the edit the package? Either the generated files or the inputs.
Ian.
Sean Whitton: Hi :) Somehow I got the understanding that `dgit` had `maintainer view of the package` == `dgit clone view of the package`. Your statement here sounds very promising and like this concept would be compatible with `dgit`. I would be very happy if I had made a misunderstanding. :) My concern are only about the maintainer repo/mainline branch. If it is not committed to the maintainers mainline branches during regular maintenance nor require extra steps/upkeep of them during upload/release flows, then we should be good. Framed differently, having the files committed or requiring extra steps would be a mental load on the maintainer I want to avoid as I see it as diminishing the value of this proposal. The maintainer will meet the generated files when merging back an NMU. That is a possible wart of this idea, but I have not found a way around that without breaking the FTP master requirement or my "no generated files in (maintainer's) mainline branch" requirement. Best regards, Niels
Ian Jackson:
I am perfectly fine if `dgit` creates a commit on the `dgit` branch to
include the generated files for `dgit clone` to match the uploaded
source package. If that is what happens with `git-dpm` / `git-debcherry`
then yes, `debian/control` would be similar.
assumption that you mean the `dpkg-source -x`/`dgit clone` view. This
would be similar to the work flow for NMU'ers and I will frame my answer
in that narrative as it would also interact with merging back the changes.
I will start with the maintainer VCS. For the sake of understanding, I
will introduce `debian/control.in` as a fictive input file that is used
as a part of the generation. The filename keeps the example simpler/more
intuitive for now even if it might work differently in the end.
The maintainer VCS would contain the three files:
debian/changelog [v1]
debian/copyright [v1]
debian/control.in [v1]
With [v1] denoting a version to mark when a file changes. When the
maintainer builds the package, `debian/control` + `debian/rules` will be
generated and injected leading to the source package:
debian/changelog [v1]
debian/copyright [v1]
debian/control.in [v1]
debian/control [v1 (generated from v1)]
debian/rules [v1 (generated from v1)]
If someone downloads the source package and unpacks (`dpkg-source -x` or
`dgit clone`; either way) that is what they will see.
For sake of simplicity, I will keep the changelog unchanged even if it
is unrealistic. But it would be one extra moving part for us to keep
track of and it does not change the cases in any meaningful way.
## Case 1: Modifying `debian/control`
Lets say they modify `debian/control` but not `debian/control.in` and
then rebuild the source package, then you get:
debian/changelog [v1]
debian/copyright [v1]
debian/control.in [v1]
debian/control [v2 (generated from v1 + manual edit on top)]
debian/rules [v1 (generated from v1)]
Only `debian/control` changes in this scenario. This comes from the rule
if `debian/control` exists then auto-generation is disabled.
This is a deliberate strategy for statisfying the FTP master rules but
also to ensure NMUers/NMU tools are not required to know about this
generation mechanism.
This case works the same if it had been `debian/rules` that was changed.
## Case 2: Modifying `debian/control.in`
Lets say they modify `debian/control.in` but not `debian/control` and
then rebuild the source package, then you get:
debian/changelog [v1]
debian/copyright [v1]
debian/control.in [v2]
debian/control [v1 (generated from v1 - v2 was ignored)]
debian/rules [v1 (generated from v1 - v2 was ignored)]
In this scenario, basically nothing changes. This comes from the rule if
`debian/control` exists then auto-generation is disabled. This is a
side-effect of the strategy from Case 1.
Best case, the tooling might detect it and warn about it. But it is
"working as designed" to match the constraints.
The NMUer would have to either manually apply the edits to
`debian/control` as well or use Case 3. Otherwise, 2 is a dead edit and
de-facto a no-op if no further edits occur later.
While this case can happen by mistake, I do not see it a real case. If a
NMU'er uploaded this, they would likely realize their change had not
taken effect and then re-upload with case 3 to finish their change.
## Case 3: Modifying `debian/control.in` + deleting `debian/control`
NMUer modifies `debian/control.in` and deletes `debian/control`, then
rebuilds the source package. This leads to:
debian/changelog [v1]
debian/copyright [v1]
debian/control.in [v2]
debian/control [v2 (generated from v2)]
debian/rules [v2 (generated from v2)]
Note `debian/rules` gets regenerated as well based on the spec I
provided (whether that is the correct approach might be a question -
particular related to merging back part, but for now this is what would
happen). Additionally, new files could be generated if the generator is
of a newer version and picked up more features since the original source
package was built.
In this scenario, files are regenerated from the new input file no
different than if you had started from the maintainer checkout and
edited `debian/control.in` from there.
## Case 4: Modifying an unrelated file
Lets say they modify `debian/copyright`, which is not used in generation:
debian/changelog [v1]
debian/copyright [v2]
debian/control.in [v1]
debian/control [v1 (generated from v1)]
debian/rules [v1 (generated from v1)]
This case is basically the same as it was before this spec from the
NMU'er PoV, but it affects the maintainer on merging back the NMU. Since
`debian/control` exists, the auto-generation does not kick in.
Note: It is quite similar to case 2, but I kept them separate
deliberately to cover them both separate for the sake of clarity.
# Merging back into the maintainer VCS
Assuming the change was an NMU, it will eventually be merged back into
the maintainer view. My line of thinking is that the maintainer will
have to review the generated `debian/control` and `debian/rules` files
for changes.
So for case 1, the maintainer would have to manually extract the delta
from `debian/control [v2 (generated from v1 + manual edit on top)]` and
apply said delta to their current version of `debian/control.in`. With
most `merge NMU` tooling (`dgit` or `gbp import-dsc`), the maintainer
would also have to `git rm debian/control debian/rules` to restore the
auto-generation.
UX improvement here would be if we can get `dgit`/`gbp import-dsc`
to force the `debian/control` into a merge-conflict state since there is
a delta and discard `debian/rules` since `debian/rules` was unchanged.
Not sure about the feasibility of this UX part.
For case 2: I do not see that as a real long term case (I think an NMUer
would redo their upload, so it follows case 1 or case 3), so I am
leaving this one unanswered.
For case 3: The merge would include `debian/control.in` (real value)
plus `debian/control` + `debian/rules` (noise; to be deleted).
Maintainer would merge and delete the two noise files.
UX improvement here would be if the merging tool (`dgit`) could catch
the noise files as auto-generated from the updated `debian/control.in`
with no delta and discard them automatically on merge. Not sure about
the feasibility of this UX part.
For case 4: The merge would include `debian/copyright` (real value) plus
`debian/control` + `debian/rules` (noise; to be deleted). Maintainer
would merge and delete the two noise files.
UX improvement here would be if the merging tool (`dgit`) could catch
the noise files as generated with no delta and discard them
automatically on merge. Not sure about the feasibility of this UX part
(but it would likely be easier than case 3, since the files are
unchanged since the maintainer upload).
That is how I see what would happen with this proposal if you work from
the `dgit clone` / `dpkg-source -x` with the files generated. Hope that
answered your question.
Best regards,
Niels
Hello, Yes, I think you did misunderstand. Currently dgit updates the dgit view from the maintainer view before it invokes the package build. It could learn to invoke your new thing to update the generated d/control as part of doing that update/conversion. To make this smooth, it would be best if it was always unambiguous just when such transformation is required. For example, the presence or absence of a particular file. That way, tag2upload gets support for your new workflow with no extra work. We have similar issues with merging NMUs when the dgit and maintainer views diverge. It's just the same problem. Anyway, glad to hear everything seems to be compatible :)
Es gibt eine Familienspende in Höhe von 1.850.000,00 USD von Cheng Charlie Saephan. Bitte antworten Sie für weitere Informationen. Denken Sie daran, Ihrer Familie und den Bedürftigen in Ihrer Umgebung Gutes zu tun. Dies ist bereits der zweite Versuch, Sie zu erreichen. Bitte antworten Sie für weitere Details.