Hello,

Since debhelper compat 10, "The dh command will no longer use log files
to track which commands have been run.  The dh command still keeps track
of whether it already ran the "build" sequence and skip it if it did."

How does it keep track now? In some file or only live? Is there a way to
get back to a file-tracking? Only by creating timestamps ourself?

My concern is that it seems to be thus running ./configure each time
I invoke dpkg-buildpackage, and thus the packaging system basically
rebuilds everything. The problem comes when there is a build failure at
some point, and I want to run dpkg-buildpackage -nc to resume building
the package after fixing the build failure ; since ./configure is
invoked again, the build system rebuilds everything! This can be
extremely time-consuming for very big packages, basically making
debhelper compat 10 unusable for packages with big build time.

I have attached a test case:


$ dpkg-buildpackage -b
[...]
   dh_auto_configure -O--no-parallel
	./configure --build=x86_64-linux-gnu --prefix=/usr [...]
[...]
   dh_auto_build -O--no-parallel
[...]
sleep 2s
touch build
false
make[1]: *** [Makefile:7: build2] Error 1


$ dpkg-buildpackage -b -nc
[...]
   dh_auto_configure -O--no-parallel
	./configure --build=x86_64-linux-gnu --prefix=/usr [...]
[...]
sleep 2s
touch build
false
make[1]: *** [Makefile:7: build2] Error 1


where "sleep 2s" represents the long build time before the actual build
failure. Because ./configure is run again, the Makefile is regenerated
and the build runs from start again.

Run again with 10 put in debian/compat:


$ dpkg-buildpackage -b
[...]
   dh_auto_configure -O--no-parallel
	./configure --build=x86_64-linux-gnu --prefix=/usr [...]
[...]
   dh_auto_build -O--no-parallel
[...]
sleep 2s
touch build
false
make[1]: *** [Makefile:7: build2] Error 1


$ dpkg-buildpackage -b -nc
[...]
[ no dh_auto_configure ]
[...]
[ no sleep 2s ]
false
make[1]: *** [Makefile:7: build2] Error 1


We directly get to the build point we were at previously.


Yes, ideally upstream's ./configure would be idempotent and be cautious
with timestamps of identically-generated files, and thus not actually
trigger a rebuild, but I believe it's far from being a general case
for upstream source, and getting this "rebuild-from-start" behavior by
default is painful.

I for instance got hit by this while fixing build issues of the samba
package, the complete rebuild helped warming up my room.

Note: in the particular example case shown above it is the dh_auto_build
step that fails. We could run make to re-run the build without being
afraid of a spurious ./configure, and be fine. But when the build
failure is within some dh_install step or such, we fall down again in
the same issue. In the example demo-1.0, replace false with true in
configure, and the problem will hit again the same with the dh_install
issue introduced in debian/rules.

Samuel

Samuel Thibault:

Hi,

It no longer tracks this via files except for being able to tell that
the entire build target has already run.  There is no way to get the
previous behaviour back.

Interestingly, I have had people asking me to *remove* the logging
because they were tired of it getting it their way with -nc rebuilds.
Presumably they worked with idempotent "configure" steps.

It occurs to me that you need this for debugging.  Would it help you if
debhelper provided a way for you to intercept the failure and resume the
build from there?

Thanks,
~Niels

Hello,

Niels Thykier, le sam. 12 sept. 2020 14:12:55 +0200, a ecrit:

That can be a start yes, indeed.

It happens that another thing I often do is build the package, install
it, try what I get, change the source, quickly rebuild the package with
-nc, reinstall, etc.  With dh < 10 I used a homemade dh_rebuild script
that clears all steps after dh_auto_build in the log. Is there a way to
achieve the same with dh >= 10?

Samuel

Samuel Thibault:

By default, debhelper tracks that it has run the build target and will
resume from there on a "-nc".  The devil in the detail here is that I
suspect that you also want to re-run dh_auto_test and that is on the
"wrong" side of that "restart point".

~Niels

Niels Thykier, le sam. 12 sept. 2020 16:49:17 +0200, a ecrit:

Ah, right.

That may happen, yes, although way less often.

Could there perhaps be a way to specify at which point we want to make
-nc restart from?

Samuel

Samuel Thibault:

It occurs to me that this can be done via a custom dh addon using
something like:

"""
mkdir -p /some/where/Debian/Debhelper/Sequence/
cat <<'EOF' > /some/where/Debian/Debhelper/Sequence/skip_commands.pm

my @cmds = qw(
   dh_autoreconf
   dh_auto_configure
   dh_auto_build
);
for my $cmd (@cmds) {
  remove_command($cmd);
}

1;
EOF

PERL5LIB=/some/where DH_EXTRA_ADDONS=skip-commands \
   dpkg-buildpackage .. -nc
"""

Except you have to manually list the commands to skip.


It is not entirely simple, but it should work already today but assumes
you inject the perl library and the two ENV variables smoothly.

You can generate the @cmd list by using:

  dh build --no-act

plus a bit of selective copy/paste.

Thanks,
~Niels

Niels Thykier, le sam. 12 sept. 2020 17:46:03 +0200, a ecrit:

That's not exactly simple :)

My general concern is about making free software contribution as simple
as possible. When discussing with people about bug fixing etc. I would
tell them that I like the Debian way of hacking software: apt-get source
... ; apt-get build-dep ... ; modify source ; dpkg-buildpackage, and
voilà, you have a modified package in your usual testing environment
(unlike getting source from upstream, with different dependencies
compatibilities etc.).

But the "modify source ; dpkg-buildpackage" part needs to be as simple
as possible for people to consider this as an accessible thing to
do. That's why I'm looking for a solution that people would be able to
use easily, e.g. an environment variable that we could document them to
use, such as

DH_REBUILD=dh_auto_build dpkg-buildpackage -B -nc
or
DH_REBUILD=dh_auto_configure dpkg-buildpackage -B -nc

depending on the need.

Samuel

Samuel Thibault:

True.

I am sympathetic to the cause, but I am not sure what the right
interface should be.  However, since it can be done via a debhelper
add-on, then anyone can write a prototype for it an experiment with the
right API for the users that need it.

If/when such a prototype has proven successful, I am happy to consider
pulling it into debhelper itself as a built-in feature.

~Niels

Hi,

Samuel is not alone in this issue. As I frequently rebuild packages with
small changes, I fully recognise the reported problem occurring since
compat 10. I would very welcome a solution to get the behaviour of
compat<10 back as waiting half an hour per build, just to find an error
in some final step is not fun...

Or using a command-line parameter to specify the step at which dh should
restart:

dpkg-buildpackage -B --no-pre-clean=dh_auto_build

Or if reusing --no-pre-clean is a no-go:

dpkg-buildpackage -B --continue=dh_auto_build

As I said, I understand the issue and can definitely see why you would
like a change in dh.
  For now, there is a work around.  In its raw format, it is not very
pretty but someone could clean it up and provide a short-hand for it.

Looks like as good a starting point as any, but it would require support
in dpkg (which has a separate set of maintainers).  Feel free to suggest
it to them, but be advised that that the dpkg maintainers almost
certainly will want a general solution (i.e. one that is not tied to
debhelper).


If dpkg-buildpackage provides an interface that debhelper can rely on
and easily use, then that I happy to consider adding support for that
interface in debhelper.  That said, I am not planning to drive the
discussion/design - "someone else" will have to volunteer to take that role.

~Niels