Currently, Debian Policy is silent on when it's appropriate to use a
native package, but there may be a project consensus aganist using
native packages when the software has an existence outside of Debian.

Even if that consensus does not exist, there is probably consensus
that native packages are a poor match for large packages (because of
the inefficiency of making small updates to the packaging of native
packages), and there may be other cases where we can give stronger
guidance.

Probe the project consensus here and see if Policy should say something
stronger.

(See #542288 for some of this discussion.)

Russ Allbery <rra@debian.org> writes:

Personally I don't think Debian policy should be concerned with what
happens outside Debian. As long as a given process serves the users and
contributors of/to Debian well, that seems sufficient to me. I
understand that others disagree, particularly on how much we should
consider the needs of derivative distributions.

That seems more like something that could be documented in devref as a
best practice, rather than something needing normative language.

David Bremner <david@tethera.net> writes:

To be clear, my understanding of the advocacy of using non-native packages
is primarily about their impact on *Debian* workflows (being able to base
multiple packages on the same tarball, not introducing confusion between
upstream version numbers and Debian version numbers and thus making it
easier for other people in Debian to track the package to an upstream
version, triggering various package checks that ignore native packages but
care about non-native packages such as uscan, etc.).

This is a good additional point for discussion, though, since I vaguely
recall that using native packages had some oddities for derivative
distributions, which is a significant Debian user use case.

Yes, that's a good point; it's possible that some or all of this belongs
in devref rather than Policy.

I do not support advising against using native packages with our current
tooling.

My issue is that for some work flows  generating and keeping up with the
upstream tarballs significantly increases the frustration of packaging
and and brings doing a package update across a pain threshhold that
psychologically matters.
The idea is that if I can accomplish a task in a minute or two without
much frustration, I'll do it more and be happier doing it.

If the task takes twice as long, especially when I can't see the value
in the steps, resentment builds up, happiness decreases, and the task is
done less often.

Being able to update packages frequently without frustration is more
important to me than the legitimate concerns raised about native
packages.
I was actually surprised how many legitimate things to think about we
came up with while discussing native packages during my DPL term.
(There's a pointer from the consensus summary I wrote up to d-d-a.)

But at least for me, when I'm trying to closely track a fast moving
upstream  from a git repo, native packages are just easier.
Generating new upstream tarballs for each upstream change, trying to
keep track of when I needed to do that,  and keeping track of all the
upstream tarballs crosses a frustration threshhold for me.

Various suggestions were made during previous discussions to make
workflows easier.
Unfortunately, these suggestions were generally made while dismissing
the needs of people favoring native packages.
I'll admit that I was frustrated when people were dismissing my concerns
enough that   it was hard to engage with the suggestions.
It's possible there are better work flows I don't know about, but I
think there are still gaps.

I'd propose the following way forward:

1) Capture the discussion thread we had during my DPL term and the
things to think about that were brought up in the native packages part
of that discussion.
I'm not sure policy is the right place for those; I think some of those
might better belong in a wiki page or dev ref.

2) Figure out  what the work flow gaps are that cause people to find
native packages easier to deal with.
I suspect we'll find that something in our git workflows is not great
especially for closely tracking upstream git and especially when
upstream itself doesn't make releases.

3) Fix these work flow gaps.

4) Then add advice to policy.

Sam Hartman <hartmans@debian.org> writes:

Thank you for this perspective!  I had entirely forgotten about that.

Realistically, I probably won't be able to drive this myself (at least
soon), since I want to try to dig through the Policy backlog for places
where we're blocking progress (which I don't think is true here).  But
even if we don't tackle this soon, I think this is a great statement of
the issues.

Hello,

I believe that we need to distinguish between version numbers without
Debian revisions and native source package formats.  At one point an
experienced contributor convinced me that there are cases where it is
good to use a version number with a Debian revision but a native source
package format.

Perhaps we can already write something useful in Policy about packages
which don't use Debian revisions, even though there is a lack of
consensus about source package formats?  Something like: you should
always include a Debian revision unless the package has a release
process which is tightly coupled with making uploads to the Debian
archive (and we would not want to include the converse, that having such
a tight coupling implies you shouldn't include a Debian revision).

    Sean> Hello,
    Sean> On Sat 03 Apr 2021 at 09:25AM -07, Russ Allbery wrote:

    >> To be clear, my understanding of the advocacy of using non-native
    >> packages is primarily about their impact on *Debian* workflows
    >> (being able to base multiple packages on the same tarball, not
    >> introducing confusion between upstream version numbers and Debian
    >> version numbers and thus making it easier for other people in
    >> Debian to track the package to an upstream version, triggering
    >> various package checks that ignore native packages but care about
    >> non-native packages such as uscan, etc.).

    Sean> I believe that we need to distinguish between version numbers
    Sean> without Debian revisions and native source package formats.
    Sean> At one point an experienced contributor convinced me that
    Sean> there are cases where it is good to use a version number with
    Sean> a Debian revision but a native source package format.

I agree with this advice.
Does the current tooling support it?

    Sean> Perhaps we can already write something useful in Policy about
    Sean> packages which don't use Debian revisions, even though there
    Sean> is a lack of consensus about source package formats?
    Sean> Something like: you should always include a Debian revision
    Sean> unless the package has a release process which is tightly
    Sean> coupled with making uploads to the Debian archive (and we
    Sean> would not want to include the converse, that having such a
    Sean> tight coupling implies you shouldn't include a Debian
    Sean> revision).

Assuming that dpkg-source supports this, I would generally support the
advice you're working toward and would be likely to second specific
text.

Hi,

Russ Allbery wrote:

I agree about this (modulo the bits discussed elsewhere in this bug
about using native packages as a workaround to issues with the format
of non-native packages).

Do you mean large packages with a separate upstream existence, or
large packages in general?  I don't think there's such a consensus for
large packages in general: if Debian is the canonical place for a
particular package to be released (e.g., as is true for dpkg), then it
doesn't seem like it would be worth the overhead of making two
releases, one upstream and one for packaging, whenever updating that
package.

[...]
particular comment in it in mind?

Thanks,
Jonathan

speaking as the debian-edu-artwork maintainer, which at one point in time
was a >50mb sized native package: it's pretty annoying to upload 50 or more
megabytes for a 2 byte fix of a maintainer script.

    Holger> On Mon, May 09, 2022 at 07:16:59PM -0700, Jonathan Nieder wrote:
    >> > Even if that consensus does not exist, there is probably
    >> consensus > that native packages are a poor match for large
    >> packages (because of > the inefficiency of making small updates
    >> to the packaging of native > packages), Do you mean large
    >> packages with a separate upstream existence, or large packages in
    >> general?  I don't think there's such a consensus for large
    >> packages in general: if Debian is the canonical place for a
    >> particular package to be released (e.g., as is true for dpkg),
    >> then it doesn't seem like it would be worth the overhead of
    >> making two releases, one upstream and one for packaging, whenever
    >> updating that package.

    Holger> speaking as the debian-edu-artwork maintainer, which at one
    Holger> point in time was a >50mb sized native package: it's pretty
    Holger> annoying to upload 50 or more megabytes for a 2 byte fix of
    Holger> a maintainer script.

I'd much rather upload  a 50M package regularly than deal with the vcs
complexity of separate maintainer and upstream releases in a lot of
cases.
10 years ago sure, that would have been annoying for me, but these days
with modern networks 50M is not a big deal.

Granted not everyone has a fast network.

If there is a consensus that the upstream source split is important for
large packages, it is fairly rough.
I'd definitely like to call that consensus into question and suggest
that it's unclear whether it exists.
It may; my opinion on this issue is definitiely on one side, and that
would make me a poor choice to judge the consensus here.
However, I don't want us to move forward assuming that consensus exists
without actually exploring it.

There cannot be any consensus because "native pakages" cover widely
different packages and situation, and anyone has in mind their own
situation, and there are no general rules.

I expect most developers do not maintain any "native pakages",
and for a minority, this is the opposite, almost all their
packages are native.

For me it is a mixed bag, with packages that "obviously" should be
native, other that should not and some other it could go both way.

tl;dr: natives packages are too diverse to be covered by a single rule.
Sometime we should trust maintainer good judgement since they are the
one that are the most impacted.

Cheers,