Dear Maintainer,

Thanks for adding pristine-tar support to tag2upload; it's nice to be
able to maintain compatiblity with the workflows of others and also use
tag2upload.

In uploading python-bumps 1.0.5-1 yesterday, the builder failed to
recreate the tarball using pristine-tar. The output doesn't give me any
clues as to why it failed:

builder:work$ tag2upload-obtain-origs p=python-bumps v=1.0.5-1 s=unstable u=c93c353afa3bf91c2f9e14052413bb2e2c24daea pristine_tar=d65b577f8d41cd09eb7f357690593e945b26cfd9
+ dgit --build-products-dir=../bpd -ppython-bumps download-unfetched-origs --write-sha256sums=.git/dgit/t2uoo/origs.sha256sums
canonical suite name for unstable is sid
package in target suite is different upstream version, 1.0.3
no .origs for package python-bumps upstream version 1.0.5
# no orig(s) in archive, generating
+ pristine-tar checkout -- ../bpd/python-bumps_1.0.5.orig.tar.gz
xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
pristine-tar: Failed to reproduce original tarball. Please file a bug report.
pristine-tar: failed to generate tarball
t2u processor [dgit-repos-server]: failed command: ./ssh-builder 'tag2upload-builder@tag2upload-builder-01.debian.org' 'env -C /srv/builder.tag2upload.debian.org/tmp/autopkgtest-virt-docker.shared.0b093mpr/downtmp/work tag2upload-obtain-origs p=python-bumps v=1.0.5-1 s=unstable u=c93c353afa3bf91c2f9e14052413bb2e2c24daea pristine_tar=d65b577f8d41cd09eb7f357690593e945b26cfd9'
tag2upload-obtain-origs failed, failed with error exit status 1


Running locally, pristine-tar checkout works just fine:

$ pristine-tar checkout -- ../python-bumps_1.0.5.orig.tar.gz
pristine-tar: successfully generated ../python-bumps_1.0.5.orig.tar.gz


I'm not sure if it's something about this particular tarball, this
particular git repo, or one of the odd little corners of pristine-tar
that has led to this failure.

regards
Stuart

Hi.  Thanks for the report.

Stuart Prescott writes ("Bug#1143449: git-debpush: pristine-tar checkout failed on builder (but works locally)"):

I'm afraid I have no idea what this means.

These all seem plausible explanations.

I suspect different versions behaviour differently.  The tag2upload
source package construction currently runs largely on stable.

I'm afraid I don't have time to try a local repro and experiment with
different versions right now.  I might have some time for that
tomorrow.  But if you felt like telling what it does for you on stable
(if that's not what you're running) that would be very helpful.

Thanks,
Ian.

Hi Ian
itself from sid so that I could try out the new pristine-tar workflow
but the machine is otherwise all trixie)

Versions of packages pristine-tar depends on:
ii  bzip2                    1.0.8-6
ii  libbz2-1.0               1.0.8-6
ii  libc6                    2.41-12+deb13u4
ii  libsys-cpuaffinity-perl  1.13~03-2+b4
ii  pbzip2                   1.1.13-1
ii  perl                     5.40.1-6
ii  pixz                     1.0.7-4
ii  tar                      1.35+dfsg-3.1
ii  xdelta                   1.1.3-10.7
ii  xdelta3                  3.0.11-dfsg-1.2
ii  xz-utils                 5.8.1-1+deb13u1
ii  zlib1g                   1:1.3.dfsg+really1.3.1-1+b1

pristine-tar and xdelta3 seem most relevant and have the same versions
in trixie and sid.

Looking at this particular repo - there is a .gitattributes file - how
is the pristine-tar orig-generating path interacting with .gitattributes?

regards
Stuart

Hi, gbp maintainers.

It seems that gbp import-orig doesn't currently arrange that
transforming .gitattributes are disabled.  This can cause disrepancies
if upstrem origs contain files with crlf with `text` .gitattributes.

Discussion of a particular case is below and earlier in this bug.
This relates to tag2upload job 5586 and python-bumps:
https://salsa.debian.org/science-team/python-bumps.git

Stuart Prescott writes ("Bug#1143449: git-debpush: pristine-tar checkout failed on builder (but works locally)"):
works for me iff one doesn't suppress gitattributes.

In this package, considering line endings in doc/make.bat:

Line endings are CRLF in:
 * python-bumps_1.0.5.orig.tar.gz in the Debian archive.
 * the extracted source package for 1.0.5-1 (which Stuart presumably
   made and uploaded with a traditional gbp and dput based workflow).

Line endings are newlines in
 * upstream/1.0.5 tag, presumably generated by gbp import-orig.
   (Stuart, can yhou confirm, and tell us the version?)
 * salsa master branch and debian/1.0.5-1 tag.

So the source package, as built on the buildds, contains carriage
returns, but the working git branch, as the maintainers work with
locally, contains newlines!  I think this is unintended and
undesirable.  (Although, here, it's probably harmless.)

(AFAICT actual upstream git is not being used here so isn't relevant,
but presuambly it has linefeeds.  Four other files are affected too.)

I don't fully understand how pristine-tar works but I think the
overall mechanism of this failure was something like this:

 1. gbp import-orig does not suppress gitattributes.  AIUI nowadays
    gbp mostly does suppress gitattributes, so this is surprising.

    As a result, the imported git tree has newlines for this file,
    even though the tarball had carriage returns.  (Due to the `text`
    attribute which "normalises" line endings on checkin.)

    So pristine-tar makes a "baseline" .tar.gz file based on a
    git-archive, with carriage returns in, and generates an xdelta
    between that and the actual orig.

 2. When trying to regenerate the tarball, tag2upload *does*
    suppress gitattributes.  It does this right at the beginning,
    before running pristine-tar or checking anything out.

    pristine-tar (which is influenced by the local tree's settings)
    invoked by t2u generates a "baseline" .tar.gz file with newlines.

    Then pristine-tar runs xdelta to apply the .delta (a binary patch)
    but the binary patch doesn't apply because the baseline is wrong:
    the tarball with newlines doesn't match the one with crlf.

Although my view is that defusing .gitattributes is essential for
reliability, it can be controversial.  (See #1092800, #1079434.)

I think the gbp maintainers will agree with me that the output of gbp
import-orig was wrong in this case.  I guess they will probably want
to change gbp import-orig.  So I'm reassigning this bug there.

IMO pristine-tar ought not to be influenced in this way by
.gitattributes nor by local git configuration.  That just makes an
already-fragile tool more exposed to strsngeness and lossage.  I don't
know if the pristine-tar maintainers will agree; CCing them.  But a
change to pristine-tar wouldn't be suffiicent, since we need the
Debian `upstream/` git branch to contain un-transformed files.  (Eg,
if instead of tag2upload, Stuart had used `dgit push-source`, dgit
would have detected and reported this discrepancy, and failed.)

The error message Stuart got from the t2u service here is very bad.
I think we could probably do some hinting.  I'll clone this bug for
that.  See also #1115853.

Thanks to Stuart for the report and to everyone else for your
attention.

Ian.

Hi all

Yes, I was in the middle of getting a set of cooperating packages all
uploaded at the same time and so dputted the changes file I had from my
test build so that I could crack on with that work.

Yes, upstream/1.0.5 was created by gbp import-orig --uscan,
git-buildpackage 0.9.38 from trixie was used. The tarball was downlaoded
from the upstream github release, meaning that it's a github-generated
tarball, meaning git-archive or something closely related is at work; I
know that file exclusion git-attributes apply to them but I'm not sure
about transformation ones). Looking at that tarball directly from
github, doc/make.bat has CRLF in it.

That means that the source data that was fed to gbp import-orig for this
file had CRLF in it; gbp import-orig has faithfully imported all the
files and git has transformed them back into LFs when importing them due
to the gitattributes but t2u has not undone that transformation? (Yes...
too many transformations.)

As for what is in the git repositories, I'm not sure how to answer that
question when git-attributions like this are active; at some sort of
philosophical level, I'm not sure that's a well posed question, given
that these transformations are intended not accidental.

FWIW though, I do not see those as newlines in a git checkout, so what
I'm seeing in front of me matches the tarball in the archive. I don't
see a contradiction in that.

$ git clone https://salsa.debian.org/science-team/python-bumps.git
$ cd python-bumps
$ file doc/make.bat
doc/make.bat: DOS batch file, ASCII text, with CRLF line terminators

That's not what I'm seeing but maybe that's not what you're meaning. Is
it possible you have some local config in place that ignores or
overrides the .gitattributes in the repository? (I don't think I have
any config here that is relevant?). Or when you talk of the git repo on
salsa, is there something other than the 3 commands above you're looking
at to check?

Upstream git also has CRLF for that particular file when I clone (but
the .gitattributes ensure that).

Interestingly, when downloading this individual file from both the salsa
and github interfaces, the file I end up with is LF. I'm not sure what
to make of that little factoid in amongst the mess of attributes, but
thought it might be useful.


regards
Stuart

Stuart Prescott writes ("Re: Bug#1143449: Please defuse gitattributes in gbp import-orig"):

I meant the contents of the blob object referenced in the tree object
referenced by the commit.  What git cat-file blob produces.

In your message you seem to take the view that what is "in git" is the
version as transformed during checkout by by the gitattributes - ie,
what you see in your working tree.  But in the general case, the
transformed-by-gitattributes version of the file can depend not only
on the contents of the original blob object, but on extraneous things
like git config settings and the set of tags available in your refs
namespace (`ident` gitattribute).  Also many of these transformations
are lossy (albeit only for data the transformation considers
anomalous).

In my view this means that this is not sufficiently stable to be
regarded as the meaning of a git tree object: the meaning in terms of
file contents has to be the same everywhere.

Nowadays, in Debian, whether you see the transformed file depends on
whether your git working tree has had the gitattributes "defused".
This is nowadayus done by gbp clone and perhaps other gbp operations
(as well as by dgit clone).  The tag2upload builder defuses
gitattributes at the very start, because it also requires that its
output be predicteable and stable.

tl;dr: IMO .gitattributes in its current form is an ill-conceived
feature that should not have been added to git.  Using it is a
mistake.  (AIUI it metastasized by generalisation from a system which
would only do transformations on Windows.)

There's previous discussion in the context of dgit here
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079434#20
and some debate in the context of gbp in #1092800.

This is all quite annoying.  I'm quite aware that the current
behaviour of tag2upload and dgit and gbp (defusing the gitattributes)
is an unfortuante compromise, but as I say I think it's a necessary
one.

I reassigned this bug report to gbp because I think gbp import-orig
should align with gbp clone.  If gbp import-orig had defused the
gitattributes then its tarball-to-git import would have contained crlf
and everything would be consistent within Debian (although
inconsistent with upstream git, but you don't seem to be using that).

So, yes, you are very likely right about this.  Some other users will
see something different.

I think probably neither of these git web uis apply the gitattributes.
In the general case, how to apply the gitattributes in such a context
is not clear, since it can depend on the ref namespace, should perhaps
depend on the intended operating system (HTTP User-Agent?!), and so
on, but it is also obvious that downloading the same file from the
same commit ought not to depend on which fork of the repo we're
looking at.

Ian.

Hi again


Ahh... TIL gbp clone did that.

Indeed with gbp clone I end up with LF EOLs.

When I push gbp import-orig to import the tarball, that file switches to
CRLF.

$ gbp clone https://salsa.debian.org/science-team/python-bumps.git
$ cd python-bumps
$ file doc/make.bat
doc/make.bat: DOS batch file, ASCII text
$ git tag -d upstream/1.0.5
$ gbp import-orig --uscan --upstream-version 1.0.5
$ file doc/make.bat
doc/make.bat: DOS batch file, ASCII text, with CRLF line terminators


It would be quite unfortunate if in an effort to be able to make Debian
workflows more standard and less dependent on special Debian tools, we
ended up not being able to just "git clone" from salsa but need a
special tool to do so (such as gbp clone).

(FWIW I concur that it would be lovely for these .gitattributes to
simply go away. I've never had anything but trouble with them and
equivalent ideas in svn, but my experience is that is messy to get rid
of them across a distributed developer base, and upstreams don't always
want to go there.)


regards
Stuart