It would be nice if it were easier to use sbuild with a gitish
downstream workflow which does not produce "3.0 (quilt)" source
packages.

For example, consider this scenario:

   dgit clone foo stretch
   cd foo
   hack hack hack
   git commit
   git merge upstream/stable
   hack hack hack
   git commit
   gbp dch
   sbuild -A
   dpkg -iGOEB ../*.deb

This is similar to the recommendatioms in dgit-user(7), except that we
are trying to use sbuild.

The above attempt will probably fail.

This is because the package "foo" is probably "3.0 (quilt)", according
to debian/source/format.  sbuild will try to build a source package,
and dpkg-source will want the user's changes to be "committed" as
patches in debian/patches.  The user won't have done that and probably
doesn't want to.  Worse, in the general case, changes might be made to
the git tree which dpkg-source cannot represent.

The solution is to not build a source package.  The user who adopts a
gitish workflow probably doesn't want one.  However, sbuild needs a
source package because that's how it transfers the package into the
chroot.

I suggest that sbuild grow an option which generates a dummy source
package in some hidden or temporary place.  I think the actual source
package generation can be done with something like this rune (and that
this will always succeed):
  dpkg-source -Zgzip -z0 --format=1.0 -sn
(but I have not yet tested this myself).

I don't have much of an opinion what the sbuild option should be
called.  --no-source is a possibility.  Or maybe it should be inferred
from the other options and the fact that sbuild is working from a
directory tree - although that would be an incompatible change.

In the meantime, I think users can use
  sbuild -A --dpkg-source-opts='-Zgzip -z0 --format=1.0 -sn'
However, that sets up a hazard: it produces a dummy 1.0 native tarball
package as if it were a build product.  But that native tarball
package is broken: it can be extracted and built, but it cannot be
re-packed, without again providing special optionsn to dpkg-source.

Thanks,
Ian.

Quoting Ian Jackson (2017-07-16 14:32:17)

Indeed, the task to solve is how to transfer the source into the chroot.

But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?
Would it not be more fitting to use some git-based command to exchange the
data? For example using a git bundle? This would then make sure that everything
git knows about is transferred into the chroot. Sbuild can make sure that git
gets installed together with build-essential and is thus able to git unbundle
the file.

What do you think?

Thanks!

cheers, josch

Hi,

No, it's clearly a hack and one that might have side effects due to
differences (for example in set of ignored files) between format 1.0
and format 3.0 (quilt).

To me the right approach is to extend sbuild to support alternative ways
to copy the unpacked source tree into the build chroot. Your suggestion
to rely on a git bundle is a possibility. A plain tar of the build
directory is another approach.

IMO this goes in the right direction of entirely eliminating source
packages in the long term.

Cheers,

Johannes Schauer writes ("Re: [buildd-tools-devel] Bug#868527: want sbuild --no-source or something"):

That would be a perfectly fine answer from my point of view.  However,
it has a few different behaviours.  I'm not sure what is best.

Things I thought of:

1. If there are files which are ignored, or uncommitted, this would
   use the committed files.  I think this is a benefit.

2. Using git this way means git is definitely in the chroot.
   But a package might build differently.  For example, it might
   have a stealth b-dep on git, which this approach wouldn't detect.
   I think this is a downside of using git.

3. The git approach cannot be easily used with a different VCS.

4. Building and unpacking a git bundle is pointless work when we know
   that the destination completely trusts the source, and does not
   have any objects already.  A tarball would do.

From the point of view of a dgit user, item 1 is quite important and
IMO outweighs the other considerations.

Regards,
Ian.

Quoting Ian Jackson (2017-07-17 11:31:07)

The only assumption sbuild should make about the chroot is, that all
Essential:yes packages are installed and that apt is installed and able to
install more packages. Starting from this situation sbuild should be able to
install anything that's needed to carry out the build like build-essential. It
is not impossible to only temporarily install git for unpacking the bundle and
to then remove and autoremove it before dpkg-buildpackage is run.

The switch could be git-specific in which case it's up to the user to make sure
that the underlying VCS is really git.

If the new switch is not git-specific then sbuild could auto-detect whether
it's git (and then do git-ish stuff) or not (and then just make a tarball).

Indeed a git bundle is overkill because dpkg-buildpackage inside the chroot is
not interested in the history stored in the bundle. But just using "tar" would
not have the advantages of item 1. So "git archive" should be used together
with its disadvantages (and workaround) from item 3.

Agreed.

I wonder what the best interface for this feature would be. Even more if we
consider this a long term feature, existing git workflows as well as dgit...

Thanks!

cheers, josch

Johannes Schauer writes ("Re: [buildd-tools-devel] Bug#868527: want sbuild --no-source or something"):

Quoting your 3:
other vcs's tarball export functions.

In this context this project came to mind:
http://www.greenend.org.uk/rjk/vcs/

Yes.  I didn't want to make some wrapper in dgit for this because
1. we already have too many wrappers in our build tools 2. my doctrine
is that dgit should be necessary only if source packages are involved.

Regards,
Ian.

Raphael Hertzog writes ("Re: [buildd-tools-devel] Bug#868527: Bug#868527: want sbuild --no-source or something"):

Indeed.  I think it's the best recommendation, though, with the
current tooling.  Unless you can think of a better option, I think I
should recommend this hack to stretch users.

You are right that it has differences in the ignored files.
Specifically, this rune transfers the complete contents of the working
tree, including the .git directory, into the build environment.  I
doubt that is likely to cause trouble in practice.

Ian.

Hi,

It has been more than two years since the last message in this bugreport. As
dgit becomes more and more mature, maybe we should revisit what sbuild could do
to improve on the current situation?

For example I would not be opposed to a --dgit switch which would then do the
right thing as it knows about the state of a source tree that dgit is able to
work with and how it can (and should) be transported into an sbuild chroot.

Personally, I never needed the dpkg-source workaround for any of my packages
with dgit but if anybody wants to propose a patch then I'd like to review and
include it into sbuild.

Thanks!

cheers, josch

Hello Johannes,

Thank you for raising this topic again.

dgit can be used with several different branch layouts.  We probably
don't want to copy that knowledge from dgit into sbuild, so perhaps what
we want is for dgit to pass to sbuild the name of a git tree, which
sbuild will then try to build?

An issue with this is passing uncommitted changes to sbuild.  But
perhaps someone who wants to do that is probably best using the existing
sbuild wrapper where you just type 'sbuild' from the root of the source
tree.

Hi,

Quoting Sean Whitton (2019-09-02 01:00:12)

I just had another idea. Since the issue at hand is, that you need a way to
transfer the source tree into sbuild and sbuild currently only supports source
packages, why can dgit not be amended to build a temporary source package from
the git tree it has in a temporary location which it then hands to sbuild or
any other builder?

I mean, a dgit users will call "dgit sbuild" instead of just "sbuild" anyways,
no?

Thanks!

cheers, josch

Hello,

This is precisely what it already does -- I thought your interest here
was in removing the intermediate dpkg-source step.

Hi,

Quoting Sean Whitton (2019-09-02 20:15:23)

okay now I'm confused. I read [1] and it made me understand that users somehow
have to remember this horribly complicated sbuild command in certain
circumstances.

[1] https://lists.debian.org/23912.64795.250863.812460@chiark.greenend.org.uk

I came back to this bug because I find it horrible for users having to remember
this rune. But now you say that dgit is already taking care of doing the right
thing?

Thanks!

cheers, josch

Johannes Schauer writes ("Re: Bug#868527: [buildd-tools-devel] Bug#868527: Bug#868527: want sbuild --no-source or something"):

Thanks for looking at this bug again.  I have been meaning to reply to
your messages for some days.  Sorry for the delay, which is partly
caused by family stuff and partly by me traveling a lot (right now I
am in Madrid for the GNU Hackers Meeting...)

I'm afraid this is going to be a long mail.


I don't think Sean is right about the existence of functionality in
dgit which addresses this issue.

dgit is a bidirectional geteway between a dsc-style archive (and .dscs
in general) and git.  When it is generating a .dsc, it tries to
generate a sane .dsc.  dgit (at least in its existing operation modes)
cannot be used with git trees which are not representable as sane
.dscs.

It is *possible* to convert any git tree into a .dsc which will unpack
into that tree.  But in the general case that conversion to a .dsc
requires several additional options to dpkg-source - the hideous rune
which is the subject of this bug report and which we are trying to get
rid of.

The resulting .dsc is insane.  I don't want to try to completely
define "sane" in this context, but ISTM that a necessary condition for
sanity of a .dsc is that it is a fixed point under dpkg-source -x &&
dpkg-source -b.  With one of these insane .dscs, dpkg-source -x works
but the subsequent dpkg-source -b fails (because it lacks the hideous
override options).


So let me step back a bit.  I think it would be worth starting with an
investigation of what the problems are that this bug is about, and
then think about the best solution approach, intended new behaviour
and implementation strategy, and the names of the new stuff.

There are three things that seem to me like they are a significant
nuisance to a dgit user using a dgit view, with the current
arrangements.  I fact they affect all users who are using pure git
workflow based on what Debian publishes, unless those users take
special care to maintain quilt patch queues up to date etc. - even if
those users don't ever run dgit (ie they got a git branch from
somewhere).  I want users to be able to just use git and not have to
care about quilt etc.

The issues I see using sbuild in this situation are:

1. The need to pass a long, silly-looking and unergonomic option
   to sbuild.

2. The fact that a .dsc is left lying around.

3. The fact that the .dsc left lying around is a beartrap: it cannot
   be unpacked and repacked without again using weird options, and
   must not be uploaded to anything like a Debian-style archive.

There are other possible concerns which I think are far less important
and possibly not important at all, but which I mention here only
because I thought of them (so that if someone else thinks they are
important they can be prompted to say so):

4. I'm not sure if making a 1.0 native gzip -1 dsc is as fast as other
   possible approaches for copying the source into the chroot.

5. Possibly unclarity and infelicity surrounding untracked working
   tree files and/or uncommitted changes.

6. Whether the .git directory should be passed into the chroot and
   thus be present during the build.  (I think yes - see below.)


Of these concerns, (3) follows inevitably from (2), in the sense that
if we want to deal with git trees generated from arbitrary packages,
and then modified in aribtrary ways (as envisaged in dgit-user(7)),
the .dsc might be insane, causing (3).  Avoiding (2) avoids (3).

So we need to tackle (1) and (2).


It would of course be possible to write some wrapper program which:
 - contains the horrid rune
 - makes a .dsc somwhere private
 - passes that .dsc to sbuild

There are a number of ways in which that would seem suboptimal to me:

 - Such a wrapper program might have to interact with sbuild's option
   parser (eg try to filter or understand options to sbuild).  At the
   very least, certain sbuild options don't make sense but maybe we
   can just tell users who pass silly combinations and get strange
   behaviours to please not do that.

 - This would still do the work of building a source package, which
   is needless copying etc.   This is consideration (4) above.
   Generating a .dsc makes us vulnerable to bugs and infelicities
   in dpkg-source.

 - The generated .dsc would presumably have to be in a directory in
   /tmp or a temp directory under .git which is kind of inconvenient
   from an implementation point of view (and we might leave it behind
   if we don't manage to run our cleanup - ie we become less
   crash-only).  More generally the overall data flow is more
   complicated than it needs to be.

 - Thin wrapper programs are confusing to users and should be avoided
   where possible.  I know Debian packaging workflows are full of thin
   wrappers - but this is indeed confusing.  For example, any time you
   see someone give a talk or a demo, or try to help them with some
   packaging problem, you have some good chance that they will use
   some wrapper or small utility you haven't heard of.  This makes
   everything hard to follow.  I used to think that this was just me
   being ignorant but since incidentally having to become an expert on
   Debian git practices I have become convinced that there is so much
   of this stuff no-one could be expected to know it all.

So I think it would be best to teach sbuild another way of
transferring source into the chroot, ideally without making an
unnecessary additional copy of it.

To my mind the obvious approach would be to make a new git tree in the
chroot and "git push" to it.  (We want to copy the git objects because
some gittish workflows want a working .git directory, for example to
encode the built commitid somewhere in the output. This is obviously
not done in Debian but it would make sense for some downstream or
ad-hoc workflows.  This does imply that git would have to be installed
in the chroot, by sbuild presumably.)

I haven't looked at the internals of sbuild at all.  Do you think this
would be fairly straightforward ?  Is it desirable ?

For the new option name(s) I suggest:
  --git-ref=<somthing git-rev-parse understands>
  --git               equivalent to --git-ref=HEAD


As for naming etc.:

I appreciate that you are seeing this through a dgit lens.  I can see
why, and dgit adoption is certainly a reason to think about this again
now.  But I don't think this is actually "dgit stuff":

dgit's purpose is to help bridge the gap between source packages and
git.  If you do not have an actual desire for source packages then you
should not get involved with dgit.  (And if your involvement with
source packages is solely that your upstream publishes dscs, then you
use dgit clone/fetch to access it but then work solely in git.)

Ie, you can start with a pure git tree containing a debian/rules and
so forth, and modify it and build it, and so on, without involving
dgit.  dgit is complicated and users who don't need to deal with dscs
shouldn't need to wrestle with it.  (Ideally eventually dgit would
render itself obsolete...)

So for this reason I think any new sbuild option should *not* be named
after dgit.  Likewise, if we decide that the best approach is the
wrapper script, I think the sbuild wrapper is probably best not being
function of dgit.


If we go for a sbuild wrapper I'm not sure where it should live.  I
have been putting various things that aren't to do with dgit.deb or
/usr/bin/dgit in src:dgit because of the test suite I have there.  But
testing such a wrapper script doesn't really seem to involve most of
the stuff that the src:dgit test suite is good at.  Mostly it would
probably involve stuff that the sbuild test suite probably knows about
already.  So maybe it should live in src:sbuild...


HTH.

Regards,
Ian.

Hello,

I'm sorry, I think I confused two dgit+sbuild topics.  Let's continue
the discussion in response to Ian's message.  Thanks again for your
interest.