Hi Sylvestre

I'm filling this as bug now.  Please discuss this issue there.

I'm setting it to serious as several ftp team members told you not to do
that.

Hi Sylvestre

No.  But is not common to do that automatically, on thousands of source
packages.  So much you even asked the ftp team to drop part of their
checks on new binary packages.

Adding new binary packages automatically.  This whole thread was about
the problems of adding new binary packages all the time and you wanted
to way around binary-NEW for them.  Not making the scope clear was my
bad and I want to apologize for it.

Okay.  Who does?  Please remove the wontfix from the bug about the
Provides, to declare it needs to be fixed.

Regards,
Bastian

Le 26/11/2019 à 20:02, Bastian Blank a écrit :
Yeah, I did ask for this and I still think that we (Debian) should evolve to follow
the evolution of software but I understand that you aren't happy with this change.

I know I am biased (Mozilla employee here) but Rust is too me the best thing that
happened to system programming for a very long time.
I agree that the way we package Rust crates is unorthodox. Just like we
deal with other "uncommon" languages like ocaml.

Maybe there is a better way. I honestly don't know. However, I think that as a
packager of a few rust-based binary, it is really a pleasure to package rust software.
Cargo and the dependencies mechanisms are working very well (esp when you come from the C/C++ world)
even if we have some corner cases which we indeed need to address.

Also, I would like to highlight that nobody told me "not to do that". Such changes have been approved
by ftpmasters for a month without, to my knowledge, any complaint. So, your message was a surprise to me.
Thanks for that!
Josh Triplett (added as cc) or Ximin Luo are probably the most experienced folks.
For context, Bastian mentioned in private the bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942898

And I am not the one who set the wontfix tag and for now, despite the unorthodox packaging method,
I haven't seen any blocking argument (besides more work for ftpmaters), only potential future issues which don't
happen for now. The CPU/bandwidth overhead is currently minimal.

Cheers,
Sylvestre

Bastian Blank:

The more precise reason, as I have explained many times already, is because the cargo package manager supports crates having optional dependencies. It is not feasible to automatically merge optional-dependency-sets together because it results in dependency loops that would not otherwise exist. It is not economically feasible to manually merge these sets together either, because it is boring and time-consuming work, error-prone (hard to manually tell if you did or did not introduce a cycle) and of questionable benefit.

I do not see any users complaining about this behaviour of our automatic tooling. We would be happy to work towards a patch on any Debian infrastructure to make these processes smoother. There is no reason why adding and removing empty metadata-only packages should require manual oversight, and if one is (and one should be) interested in automating the amount of manual work involved in maintaining Debian infrastructure, this is one obvious tedious task to automate away.

We are all volunteers, there is no "job security" here, why are we manually reviewing empty packages and we are we trying to conserve a process that involves manually reviewing empty packages?

X

Control: severity -1 serious

We told you several times that cycles in Debian packages are supported.

So is the ftp team.

Regards,
Bastian

Hi Ximin

Please stop fiddling with severities.

Yes, I got that.  And it seems cargo does not support recursive
dependencies.

However Debian does not use cargo, it uses dpkg and apt.  apt and dpkg
actually support recursive dependencies.  Due to some downsides in
regards of the handling of maintainer scripts they are discuraged.  But
as long as you don't have any of those, which all those packages don't
have, that's not much of a problem.

Sylvestre as rust team member asked the ftp team, which is responsible
for the archive content, to change their handling of binary-NEW.  So you
expect the ftp team to do the work you don't want to do.

This bug is about members of the ftp team asking you to change your
solution to that problem.  Re-iterating why it's not possible does not
help.

Because the ftp team is responsible for the content of the archive,
including package names etc.

Regards,
Bastian

Bastian Blank:

The maintainer of a package decides the severities and whether things are bugs or not. Neither have you provided a justification for "serious", it is not breaking anything.

Manual cost, nobody is going to want to do this work. Do you want to do this work?

Your proposed solution involves us doing more manual work, our suggested solution involves you doing less manual work. So, no thanks.

X

Hi Ximin

The output of debcargo breaks the Debian archive by increasing the
Package file much more then it needs to do.  Plus it can create
oversized Provides lines.
debcargo and rip the code out to create the surplus packages.

I would have preferred it if you would do that yourself, but if you ask:
yes, I will delete that code from debcargo.

Regards,
Bastian

Le 29/11/2019 à 14:07, Bastian Blank a écrit :
Breaks seems a strong word, no?

We have at least 680 rust source packages in the archive currently, we shipped buster with more than 500 Rust packages.
For now, I only seen a few ftpmasters complaining about the current packaging strategy (but maybe I missed users complaining about this).
Despite this, Thorsten and a few other (I saw some trainees) did an amazing job keeping up with the incoming flow of NEW rust packages.

What solution would make you happy here? (or at least acceptable).
To start with, we could trim the description in the library crates.
Source:
$ grep-dctrl -FProvides -e '.{5000}' -sPackage < /var/lib/apt/lists/ftp.fr.debian.org_debian_dists_unstable_main_binary-amd64_Packages

Cheers,
Sylvestre

Bastian Blank:

I don't understand what point you are making here. If you delete code from dpkg without thinking about the consequences, stuff is going to break.

Likewise, deleting code from debcargo like you're saying here, is going to break things. Fixing these breakages without simply undoing your deletion, will involve much more manual work.

X

It adds bloat to the Packages file in a volume that is not longer
considered acceptable.

I assume you know the difference between doing something once or doing
something a thousand times.

So would setting a limit to 500 rust source packages in the archive
at the same time help?

Stick to one binary package per source package.

The only point that shows up again and again are cycles, which don't
pose a problem in the dpkg world.

Sure, but the 256k example that showed up shows that the algorithm used
to generate it can produce problematic output.

Regards,
Bastian

Bastian Blank <waldi@debian.org> writes:

Procedurally in Debian neither of these are justifications for setting the
severity to serious.  This is not a Policy requirement that has reached
consensus, and the release team is the team in Debian that has the
delegated responsibility to determine which bugs are release-critical for
the next release.

I realize that you're frustrated, and I suspect you're feeling like the
Rust package maintainers are not respecting something that you think is
obvious, but I don't think that fighting over the bug severity is the
right way to approach this disagreement in Debian.  If you feel that
you've reached an impasse in this discussion, this is what the Technical
Committee is for.

Bastian Blank:

Whilst dpkg can install cycles, they only work well in special circumstances that don't apply to rust packages. For example where the same source package builds everything in the cycle, or where the source packages in question are very stable and essential e.g. glibc and gcc.

The reason is when building the source package you have to install all the Build-Depends. So if source package A and B depend on each other's binary packages, and A and B are not yet in Debian, you have no easy automated way of building them. You have to do the hacky manual bootstrap way where you sudo cp stuff into /usr. Doing this for 600 rust packages, and continuing to do this in the future, is not a standard I want to adopt for the Debian Rust team, and I don't think anyone would enjoy making this part of the normal process of making rust packages. There is also the issue that rust crate often change API compatibility, so occasionally you will have to rebootstrap cycles even if one of the packages exists already in the archive but at an older version.

X

Hi Russ

My plan was to get the problem acknowledged and then accept the
remaining waiting Rust packages from NEW.

This problem is already listed since a long time in
https://ftp-master.debian.org/REJECT-FAQ.html.

The problem is not about severities, but about the Rust team to
acknowledge that there is a problem.

Bastian

Bastian Blank <waldi@debian.org> writes:

I assume you mean the "Package split" point there?  If so, I think it's
clear the Rust packaging team did think about this split and is doing it
for specific technical reasons, and the disagreement is over the merits of
those reasons.

(If you mean some other point of the REJECT FAQ, I'm not sure which one
you're referring to.)

Right, but it's not in the power of anyone in Debian to require some other
person in Debian acknowledge there is a problem (and this is good).

I don't think there's a disagreement here over the technical impact of the
packaging model used for Rust crates.  The disagreement is about tradeoffs
between ease of packaging, dependency resolution, and the size of the
Packages file, and different people are arriving at different conclusions
because of their different weighing of those concerns.

In that situation, if that discussion reaches an impasse and no one can
come up with a breakthrough technical solution to resolve the problem, it
seems appropriate to me to invoke the Technical Committee to help evaluate
the tradeoffs.

Hi all--

I've been trying to make sense of the current struggles around rust
packaging in Debian, and this bug report (#945542) seems to be where
they're crystallzing.  (it's not clear to me how this bug report is
supposed to be distinct from #942898, which already captures some of
these details.  if the reporter or maintainer wants to merge them, i
think that'd be reasonable)

Full disclosure: i *want* rust software to work in a well-established
way in Debian. i am involved in helping with some of the rust packaging.
and at the same time, i am also frustrated at the immaturity and flux
that i see in some parts of the Rust ecosystem.  And i also find myself
frustrated with some of the frameworks involved with NEW queue
processing and unstable-to-testing migration, which i think are
occasionally counter-productive to debian's goals as a whole, and are
wasteful of the limited time that all of us (on the Rust team and the
FTP team) have to spend on Debian.

So i acknowledge that there is legitimate friction here, and i hope we
can work through it in a way that makes both Debian and the Rust ecosystem
healthier in the long run.

Identifying problems
--------------------

The claims that have been listed here (but actually seem distinct to me)
are:

 a) A small number of Rust packages provide very long Provides lines,
    which break some other infrastructure (e.g. reprepro)

 b) Many Rust source packages produce a handful of additional binary
    packages that are empty but have different dependencies, as a way of
    capturing crate "features" without introducing a large mass of
    interlocked circular dependencies.

 c) The subject of this bug report is "Randomly adds and removes binary
    packages".  I but haven't been able to find evidence of randomness
    here -- the binary packages that do trigger additional passes
    through NEW are not random -- they represent evolution of the
    features offered by upstream software.  Surely this is similar to
    evolution of other upstream software that is packaged for Debian
    (though perhaps the scale is different).

 d) There is an assertion that binary packages are specifically
    problematic for the FTP team.  For example:

    But reading this page, the closest thing i can find is its
    description of "Package Split", which reads:

    Bastian, if you meant something else, please correct me here!


Evaluating solutions
--------------------

(a) appears to be quite real, as lengthy Provides: lines have caused
failures in other pieces of software.  However, I'm a bit surprised that
i didn't easily find a bug report that either references or affects
reprepro, one of the packages known to have line-length limitations.
Someone who has encountered this bug in the wild and has documented it
should make sure that it is adequately represented on its own as a
specific bug report (or more than one), so we can peel this particular
situation off from the other issues.

(b) appears to be important to make it possible to avoid significant
bootstrapping difficulties with rust packages in the debian
ecosystem. Ximin has already explained some of the issues related to
re-bootstrapping a bunch of indpendent packages that have circular
dependencies so i won't go into it in more detail, but the goal here is
to facilitate package maintenance and require less manual intervention.
These are good goals.

There are potential performance concerns about what these extra binary
packages do to consumers of the archive -- both for mirrors and for
typical debian installations.  I asked for some clear profiling of these
performance issues over on #942898, but no one concerned about the
performance loss has stepped up to provide measurements.  I believe
these risks are real, but i also believe that there are real advantages
(to both Rust and Debian) to making sure that packaging rust software
for debian is easy and straightforward.

I'm not convincd that (c) represents a real problem, or at least it's
not one unique to the Rust ecosystem.  While shifts in upstream designs
may be annoying/frustrating to the debian maintainers of this software
and to the FTP team, the changes in stated feature sets by each version
of a rust crate upstream is no different than a C library package that
pushes changes to the SONAME more rapidly.  As packagers, we should be
applying pressure on our upstreams to adhere to some reasonable measures
of stability.  we can help to explain to them why the costs of
instability and complexity are probably not in their project's (or their
users) best interests in many cases.  For example, I do that already in
rust projects like https://gitlab.com/sequoia-pgp/sequoia/issues/356.
But, there will inevitably be some legitimate instability or complexity.
this is software.  and as packagers and distributors, some of our work
is to absorb and systematize that instability and flexibility.  At least
Rust is trying to explicitly grapple with these variations via feature
flags and semantic versioning.  Some other language ecosystems don't
even have an understanding of these issues or a way to talk about them,
and they just get swept under the rug.  We shouldn't be penalizing a
language that tries to grapple with them explicitly.

As for (d), reasonable people might disagree about whether the quoted
text describes the Rust crate packaging produced by debcargo.  I don't
think it describes the situation (except perhaps for the distinct
concern raised in (a)).  In particular, these package splits are not
made for any unfounded arbitrary reason.  they have clearly articulated
goals, and they are situated in the broader context of debian package
management.  The main concern is even explicitly called out as a
legitimate justification for package splits -- "big dependency chains
can be a reason".

There's clearly a problem, because Rust-based software isn't getting
into Debian at the rate that it probably should.  But it doesn't seem to
me that the problem is on the Rust team alone to solve, or solely the
fault of the current debcargo packaging infrastructure.  And it's
certainly not a problem that would warrant removal of decargo from a
future debian release.

We have varied, competing interests in Debian, and it works best when we
all understand where each other is coming from, and we work together to
find a balancing point, along with exerting pressure on our upstreams to
provide sensible, maintainable software.

I'd love to see some concrete proposals put forward to move this out of
the current stalemate.  Perhaps evidence of the claimed performance
problems would help that work progress?

Regards,

Control: merge 945542 942898

In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945542#94
is this text
| it's not clear to me how this bug report is supposed to be distinct
| from #942898, which already captures some of these details.  if the
| reporter or maintainer wants to merge them, i think that'd be reasonable

That merge is now done.  No, I'm not the maintainer of debcargo,
I'm a DD that wants Rust packages in Debian. Infact I want a
smooth way from Rusts dependency resolve and builder 'cargo'
to the Debian archive.  I'm fully aware it will take effort
both from Rust team and FTP team


Links for clicking to the bug reports
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945542
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942898


Regards
Geert Stappers

Control: affects 945542 ftp.debian.org
Control: merge 945542 942898