Hi. This just happened:

  dima@shorty:/tmp/opencv-4.6.0+dfsg$ sbuild -c sid-amd64 -d unstable -s -A --anything-failed-commands '%s'

  dh clean
  dh: error: unable to load addon maven-repo-helper: Can't locate Debian/Debhelper/Sequence/maven_repo_helper.pm in @INC (you may need to install the Debian::Debhelper::Sequence::maven_repo_helper module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at (eval 14) line 1.
  BEGIN failed--compilation aborted at (eval 14) line 1.

  make: *** [debian/rules:126: clean] Error 255
  E: Failed to clean source directory /tmp/opencv-4.6.0+dfsg (/tmp/opencv_4.6.0+dfsg-11.dsc)

The user expectation is that sbuild takes care of all the Build-Depends
(by installing them in the chroot), but this apparently isn't 100% true:
it runs "dh clean" outside the chroot, so any extra debhelper bits must
be installed outside.

Can we fix this by not doing anything outside the chroot that sbuild
itself doesn't Depend on? The simplest way to do that is to make
--no-clean the default. Of we can run "dh clean" inside the chroot. Can
we do something like that?

Thanks

Quoting Dima Kogan (2023-03-28 23:13:16)

no. This is a feature not a bug. From your invocation I assume that you ran
sbuild inside and unpacked source directory? The input for sbuild is a .dsc. To
make it easier for the user, sbuild contains some code that automatically
builds the .dsc for you if you run sbuild inside an unpacked source directory.
To build the .dsc you need a clean source directory. Since sbuild cannot know
whether or not your source directory is clean, it runs the clean target by
default. If you know it's clean, then you can force sbuild to not do that by
passing --no-clean. If you do not want to pass --no-clean, then you can instead
create the .dsc in any other way you like and then pass the .dsc to sbuild. If
you pass a .dsc to sbuild, then sbuild will not attempt to run the lean target.

Does that make sense?

Thanks!

cheers, josch

Hi. Thanks for the explanation.

I have never once in my life ran sbuild from a .dsc file. In fact I
don't think I've ever done anything with .dsc files directly. I'm always
sitting on the sources, with a ../whatever.orig.tar.gz on disk.

If I've been using it wrong this whole time, I guess that's on me. But
starting from sources feels like the natural flow to me, so can we make
this work a bit better?

If we wanted to make the clean step optional, sbuild can check for
source differences, and barf if any are detected. It mostly does that
already. I believe it doesn't complain if there are extra files on disk,
but we could make it do that too.

Other than that, can we run "dh clean" inside the chroot?

Thanks!

Quoting Dima Kogan (2023-03-28 23:27:55)

Yes, since most people want to use it like that, the convenience wrapper was
added. That way, you do not have to manually build the .dsc before calling
sbuild. But building the .dsc is mandatory one way or another because that's
the input for sbuild. It's how the source package is transferred into the
chroot.

You have not been using it wrongly. And you are not clear in your terminology.
The source is the .dsc. You could also call the unpacked .dsc source but the
disadvantage of that kind of source is, that in contrast to the .dsc, the
unpacked source can be tainted and thus has to be cleaned first.

I don't think you've been doing anything wrong. You've just taken advantage of
a convenience feature sbuild offers to be able to run from within an unpacked
source package.

No it does not. Maybe what you observed is, that dpkg dies if it finds
unexpected changes but sbuild does no such checks.

That's dpkg complaining, not sbuild.

I fear I do not quite understand what kind of feature you are asking for. Do
you really think it would be a good idea if sbuild, every time you run it,
first locates a .dsc, unpacks the .dsc, compares the unpacked .dsc to your
current directory and only invokes the clean target if it finds differences?
Would there not almost always be differences because you only invoke sbuild
*after* you've made some changes to the unpacked source directory? And what
should sbuild do if it has detected changes? It would still need to run the
clean target before it can create the new source package.

What would that accomplish? At the point where the .dsc is unpacked inside the
chroot, it already is clean. You need a clean unpacked source directory, so
that you can build a .dsc so that it can be copied into the chroot. So this
cleaning has to happen on the outside.

Thanks!

cheers, josch

Johannes Schauer Marin Rodrigues <josch@debian.org> writes:

Those questions are all valid, of course, if you think of the .dsc as
the input to sbuild. Up until today I was not even aware that this is
how it works.

The feature I'm asking for is that on a brand-new Debian install I
think I should be able to

  1. apt install sbuild

  2. create schroot for sbuild in whatever way

  3. apt source package

  4. cd package

  5. sbuild

Today this doesn't always work, because sbuild wants to "dh clean"
outside the chroot. Omitting the "dh clean" (by relying on dpkg
complaining) would be one way to get this working. Doing the "dh clean"
inside the chroot after the Build-Depends have been installed is
another.

Maybe the above sequence shouldn't be expected to work, but that makes
sbuild less useful in my view. I can make --no-clean the default in my
config, I suppose. Probably others use sbuild in this way too? I guess I
have no way of knowing.

Hi,

Quoting Dima Kogan (2023-03-29 03:33:46)

that is correct. But it could be said that this is still not a bug because you
are using sbuild wrong. ;)

Yes. The downside would be, that this would mean you've this check that figures
out whether you made any changes every time you run it. I'd argue that the most
common way to run sbuild is *after* you've made some changes and not on a
source package without modifications. If you want to build a source package
without having modified it, then your above list of steps are wasting CPU
cycles. Instead of unpacking the source package and then repacking it again,
you could just:

apt-get source --download-only package
sbuild -d unstable package_*.dsc

Or could could leave even the downloading of the .dsc to sbuild:

sbuild -d unstable package

Or instead of using apt to download and unpack, you could get the source
package with debcheckout and build with gbp buildpackage and then use the
--git-cleaner option to run something like "git clean -fdx && git reset --hard"
before running sbuild. Cleaning like that also doesn't require any additional
dependencies on the outside being installed.

Would either of this solve your use-case?

No. Doing the dh clean on the inside does not solve your problem because before
anything can be run inside the chroot, a .dsc has to be made availabe on the
outside so that something can be copied into the chroot. To make the .dsc
available on an unpackaged source package, sbuild runs the clean target first.
I've explained this already in past mails. The input to sbuild is the .dsc. If
you want to put anything in, it has to be made a .dsc first. If your input is
an unpacked directory, it has to be made into a .dsc first. In that case, your
directory needs to be cleaned before a .dsc can be made.

I'm still trying to figure out what you are trying to do. If you want to build
a source package without modifications, don't unpack it but feed sbuild the
.dsc directly. If you want to make changes to it, unpack it. But then, after
you've made changes, the directory needs to be clean before it can be made into
a .dsc again.

Beware that this will lead to broken packages if you did make the wrong
changes. That's why it's not the default.

Me neither. I can find two other mails from people who didn't know that the
input to sbuild is a .dsc (and thus the clean target needs to be run if you
start from an unpacked source directory) in my inbox from the past eight years
i've maintained sbuild.

At this point I don't know anything that can be done from the sbuild side to
act on this bug (hence the "wontfix" tag):

 - cleaning on the inside is pointless because it would not avoid having to
   clean on the outside
 - detecting changes would be pointless in most cases and a waste of CPU cycles
   because most people run sbuild *after* having made changes. If you don't
   want to change anything, then don't run from an unpacked source directory.
   Is there any reason to unpack a .dsc other than to make changes to it?

Thanks!

cheers, josch

Hi. Thanks for all the explanations. I just re-read this whole sequence
of emails, and I'm mostly clear on this now.

First off, I think the last email confused things a little bit. I run
sbuild on modified source, as you expect. The sequence in the previous
email was just a simple example of something that surprisingly to me
doesn't work in sbuild.

All packages I ever work on live in git. So the "clean" state is defined
as the result of "git clean -fdx && git reset --hard". I know that
sbuild can't assume git

So my workflow is usually something like

- git clean -fdx && git reset --hard
- sbuild

If I forgot the clean step above, dpkg usually yells at me. That feels
like enough, without needing a "dh clean" also.

If there's no way to make this always work in all cases, can we make the
error message better somehow? How about

  "dh clean" failed. Note: this runs outside the chroot, so the required
  Build-Depends may not be installed

Or something like that. This bit of info would have saved me some time:
I spent time looking around before filing this report. Is that
reasonable? Can we do even better?

Thanks much

Hi,

Quoting Dima Kogan (2023-03-31 08:59:01)

if you are not using gbp which has the --git-cleaner option, how about adding a
setting you can put into your ~/.sbuildrc, which, instead of running
"debian/rules clean" will run an arbitrary command of your choosing like "git
clean -fdx && git reset --hard"?

Yes, we can absolutely improve the error message.

Note though, that in the sbuild.conf man page it already says:

Does that paragraph say everything you would've liked to know or is there
anything you'd add there?

Thanks!

cheers, josch

Hi

That paragraph says what I would have liked to know, yes. But I never
went looking for it in the docs. If one thinks of sbuild as handling all
of the Build-Depends for you, then those failures just look like weird
bugs, and I wouldn't expect the manpage to say anything about it. Maybe
it's all fine. I don't know.

Quoting Dima Kogan (2023-04-04 07:03:09)

Okay, then what would be your preferred solution that would close this bug?

That the clean target needs to be executed is nothing special for sbuild. The
pdebuild utility from pbuilder also runs debian/rules clean first so that it
can generate a source package.

How would a resolution to this bug look like from your point of view?

Thanks!

cheers, josch

An extra line in the error message that reiterates that "dh clean" runs
outside the chroot, and needs manual Build-Depends would be sufficient I
think. Then the user knows it's not a bug, and can go read the manpage
for more detail.

Even better (but more work) would be to identify the missing package.
It's almost always dh-SOMETHING. Is it easy to grep the Build-Depends
for all packages that match ^dh-.*, and say "try installing THIS and
THAT"?

Hello everyone, just became aware of this bug due to
https://salsa.debian.org/debian/sbuild/-/merge_requests/71

I would like to explain why I would prefer sbuild to default to --no-clean and
ask if I'm missing something.

Running dh clean inside the chroot will remove the need to run it outside.

I don't know why it has to happen outside, why can't sbuild run dh clean even
when the input is a .dsc?

I agree that it's not a bug, but I think it would greatly improve the usability
of sbuild by making it simpler.

Today, a person using sbuild needs to either A) Default to --no-clean-source,
B) Install build-dependencies whenever there are errors when building a
package, cluttering the host system.

Both of these things require the person to also know about this detail.

WIth --no-clean-source by default, there's no need to ever bother with it, and
that's what we recommend to mentees in the Debian Brasil community.

If you clean it on the inside, you allow sbuild to have --no-clean-source by
default.

I guess this is where I'm likely missing something, considering the following
things:
1) sbuild calls dh clean inside the chroot even when a .dsc is provided
2) sbuild defaults to --no-clean-source

What type of change can one make to a package which will lead to any type of
issue?

As far as I know, running dh clean twice can't cause any issues. Changes
outside of the scope of dh_clean are not related to this issue as it won't
matter whether dh_clean is even run.

I can't think of any scenario where this can lead to issues.

I do acknowledge that this might lead to dh_clean being called twice, if the
user calls dh_clean when creating the dsc, but that doesn't seem to be an
issue. If really needed, sbuild could provide a parameter that lets the user
control whether dh clean is run inside (defaulting to true).

This way sbuild becomes much more "standalone", while still allowing users to
choose the other behavior.

Cheers,

Hi,

Quoting Samuel Henrique (2024-11-30 23:26:03)

I am not fundamentally opposed to switching the default. But if it is switched,
then there should be good reasons which should be explained in an entry to the
NEWS file.

What surprises me is that there is now an influx of people who have a problem
with cleaning being the default. Before 0.87.1, sbuild just ignored whether the
clean target is run on a system without the dependencies for that installed.
One of the "resolutions" might also be to go back to this behaviour which I
call buggy. But apparently now people are calling checking build dependencies
before running dpkg-buildpackage a bug, so I guess my view of the world is
skewed.

No, it would not. The clean target inside the chroot is run *after* the source
package is unpacked inside the chroot and that happens *after* the source
package is packed up outside the chroot which in itself happens *after* the
clean target is run on the outside.

The input to a run of sbuild is a dsc. If what you have is an unpacked source
directory, then sbuild has to create a dsc first. To create a dsc you first
want to run the clean target first. If you don't your source package might
contain content that you don't want. This is why the clean target is run by
default.

The above is not correct. With --no-clean-source you might run into other
problems (files you don't want are now part of your source). So there *is* a
need to bother with it.

There are also more than the options that you show above. There is at least
also:

   C) don't default to --no-clean-source but tell "gbp buildpackage" to run
   sbuild with --no-clean-source because gbp will only run if the git tree is
   clean

   D) move build dependencies from Build-Depends to Build-Depends-Arch and
   Build-Depends-Indep

   E) pack up the source package into a dsc yourself and then call sbuild on
   the dsc

Errrr... no. Why?

$ apt-get source hello
$ cd hello-2.10
$ dch -i
$ touch debian/foobar
$ dpkg-source -b .
$ diffoscope ../hello_2.10-3.dsc ../hello_2.10-3.1.dsc
[...]
 -rw-r--r--   0        0        0      964 2022-12-26 15:30:00.000000 debian/control
 -rw-r--r--   0        0        0     2264 2022-12-26 15:30:00.000000 debian/copyright
+-rw-r--r--   0        0        0        0 2024-11-30 22:46:47.000000 debian/foobar
 -rwxr-xr-x   0        0        0      141 2022-12-26 15:30:00.000000 debian/rules
 -rwxr-xr-x   0        0        0      798 2022-12-26 15:30:00.000000 debian/rules-old

That. The hypothetical debian/foobar could be something created by running the
full build or parts of the build locally. Those things are usually (if there is
no bug) removed by the clean target. These days, our packages are mostly in git
repos, so we can use "git clean -fdx" to even remove files that a buggy clean
target would miss. Which is why it's pointless to let sbuild clean the source
if you are working with git-buildpackage. But not everybody is. So if you
switch the default then you break some user's setups.

I have shown one above. Do you need a more real-world and less hypothetical
issue? Then I'll find a source package for you that exposes this.

I don't understand what this has to do with the clean target (it's not always
dh clean) being run inside the chroot. That run of the clean target is
completely irrelevant for this change. The clean target on the outside is
necessary to produce a good dsc. That's what you will potentially upload to the
archive. That's the thing that must not be buggy.

I would like to improve the situation but this is not it.

Thanks!

cheers, josch

Hi Johannes,

* Johannes Schauer Marin Rodrigues <josch@debian.org> [2024-11-30 23:55]:

I am not sure if the reason needs to be in NEWS but I would be happy if
that would unstuck sbuild!71.

$ dch -i fake

This is just as well an example why running clean is not enough as it
only works for files that are actually cleaned and assumes that the
developer actually uses sbuild to generate the upload. I think we should
advice people to rather use gbp or dgit for that.

Would adding a warning similar to the chroot-mode one so people can
prepare for the default switch work for you?

Cheers Jochen

Hello Johannes,

I don't understand what "sbuild just ignored whether the clean target is run on
a system without the dependencies for that installed." exactly means, I always
had this issue with sbuild (even pre 0.87.1), I've been using --no-clean-source
by default for years now. I've been meaning to request this feature and I
always postponed, up until I noticed the discussion was already happening.

I fully understand if at some point further you say it's a matter of personal
taste and you don't want to change (and I would be ok with that), but I feel
like we have not agreed on what specifically is being asked and the possible
solutions for this.

This characterization of the issue is not correct or fair, it's not what I'm
asking, so let me try to clarify it down below.

This would indeed solve the issue for users of gbp, but I think there's
potential to solve it for non-gbp users as well.

At this point it becomes easier to just install the build-dependencies in the
host machine, so I didn't consider these options.

Because there will be no difference whatsoever to the built package, but I now
understand your concern is what a crufty .dsc file, more below...

I believe I understand your concern now, so it would make sense to double check
that we're on the same page, please let me know if I'm missing something below.

* Considering a proposal to run the clean target inside the chroot instead of
  outside by default.
* The risk of this issue is that the build-generated .dsc is created outside of
  the chroot without "clean", and it would result in uploading a crufty .dsc to
  the archive.
  - This can only happen for files under "debian/", given dpkg-source will
    error out if there are any upstream changes.
  - This can only happen for files which are taken care of by the clean target,
    as with cruft files not listed in the clean target will not be cleaned even
    if we run it outside of the chroot.
* There's a "hidden contract" that ".dsc"s are always clean, and sbuild assumes
  that's the case today by not running clean.

If this is not exactly your concern, feel free to ignore my proposal below and
let me know what I got wrong.

And just to avoid the (small) possibility that we are not in agreement on why I
would like this change, for clarity's sake:

I would like sbuild to build a package without requiring build-dependencies to
be installed in the host system. I don't think the current behavior is a bug,
this would be a usability improvement.

Now about proposals to address your concern, if we agreed on the definition of
the issue with the change:

Proposal 1) Bail-out on cruft:

* Default to --no-clean-source
* Add a parameter which lets users control whether "clean" is run inside the
  chroot.
* Default to running "clean" inside the chroot.
* If there were any changes on debian/*, or the debian tarball differs from the
  .dsc, bail out and tell the user there was cruft in the .dsc.

The failure scenario is so unlikely that this will be rare, so bailing out is
not problematic. This also enforces that ".dsc"s uploaded to the archive are
clean, as today sbuild will happily take that.

Proposal 2) Re-generate .dsc in chroot and use that instead:

* Default to --no-clean-source
* Add a parameter which lets users control whether the dsc file is re-generated
  inside of the chroot.
* Default to re-generating the .dsc inside the chroot.
* If the "dsc" is re-generated inside the chroot, then run clean there.
* Generate a temp ".dsc" to serve as input to sbuild, then re-generate the
  ".dsc" inside the chroot.
* Use the re-generated .dsc as the final build artifact.

What do you think?

Thank you,

I've spoken to Johannes and I have a clearer understanding of his concerns.

The main thing I was missing in my suggestion was to clearly distinguish the
behavior from two use cases:

1) sbuild is called with a .dsc as input file
2) sbuild is called with an unpacked source tree as input.

On case number 2, sbuild will create the .dsc (with the d/rules clean target)
and feed that into the workflow used for use case #1.

The people in this bug report, and I assume most people outside of it, mostly
used sbuild with the second use case, but some people (and buildd itself) falls
under scenario #1.

The solution needs to make sure not to fiddle with the .dsc used as input, for
example: it should not re-generate a .dsc when inside the chroot if the input
was a .dsc already.

For the second use case, Johannes would accept a solution, but he believes it's
not going to be straightforward due to the way sbuild works today.

For the second use case, I've mentioned the possibility of generating a
temporary .dsc, feeding that into the chroot, regenerating the .dsc there, and
using that one as the build artifact. Johannes said there might be issues
regarding sbuild hooks used for when the .dsc is first fed into it. Those hooks
could result in values from the initial .dsc to be used instead of the final
.dsc.

This was a bit of a convoluted proposal anyway, so what I believe it's the most
clean solution is:

Add logic to send the unpacked source tree straight into the chroot, generate
the .dsc there and use it as the build artifact, but this should ONLY happen if
the input is not a .dsc.

There's a parameter which already lets users get a .dsc from the chroot as the
build artifact: --source. So most of the complexity relies on implementing the
"move unpacked source into the chroot" and "only behave as --source if the
input was an unpacked source tree" features.

Johannes, feel free to correct me if I got anything wrong based on our
conversation.

And if anybody reading this is interested in implementing it, please go ahead.
I want to give it a try myself but I don't know when exactly I will have the
time.


Cheers,

Hi,

you might want to consider bringing up what you think are problems with sbuild
on on the Debian BTS of sbuild instead of d-devel unless you think that this
topic deserves wider discussion. I put the relevant bug in CC. Maybe drop
debian-devel@lists.debian.org unless you think that this issue deserves being
discussed by all DDs.

Quoting Scott Talbert (2025-02-14 03:01:29)

this is not what is happening. Not *all* the build dependencies need to be
installed. Those needed to run the clean target need to be installed. The
change in behaviour you see happened due to this commit in November 2024:

https://salsa.debian.org/debian/sbuild/-/commit/4da64ffbb800230ad7c84ab0c701d84ea35794cd

This commit fixed two issues:

 - the clean target was run without making sure that the packages required to
   run the clean target are installed. In the past this led to situations in
   which you ran sbuild but then you got an error from d/rules at the point
   where it tried to run a tool as part of the clean target that was not
   installed
 - the clean target was run inside fakeroot and unconditionally. By letting
   dpkg-buildpackage run the clean target, we let dpkg figure out if that is
   really needed or not.

If you want sbuild to run the clean target without checking whether the
packages required to run the clean target according to d/control are installed
then no, this is not going to happen. You can see this wontfix bug for details:
#1033626. Essentially your options are:

 a) add $clean_source=0; to your ~/.config/sbuild/config.pl

 b) install those dependencies required to run the clean target on your machine

 c) if you use gbp buildpackage to run sbuild, configure it such that it passes
 --no-clean-source to sbuild (gbp will only work in a clean state anyways)

 d) move move build dependencies from Build-Depends to Build-Depends-Arch and
 Build-Depends-Indep leaving in Build-Depends only those dependencies which are
 required to run the clean target (see Debian Policy §7.7)

 e) pack up the dsc yourself manually and call sbuild with the .dsc as input

 f) teach sbuild to accept a source *directory* as input instead of the dsc
 (patches welcome) and then it can run the clean target inside the chroot

 g) teach gbp to pass the --no-clean-source to sbuild if the directory is
 already clean (i'm sure guido would accept patches too)

The fact that sbuild creates the dsc for you is a convenience option. The input
to sbuild is the source package not the unpacked directory. So before sbuild
runs it needs to create the source package. For that to happen without messing
up, the unpacked source directory must be clean. You can achieve this without
running the clean target if you do your packaging with git but then you might
want to use "gbp buildpackage" and let it do the right thing. sbuild is also
used for packages that are not maintained in git, so it cannot know that your
unpacked source directory is probably clean already.

Thanks!

cheers, josch