breezy FTBFS on the s390x buildds:

https://buildd.debian.org/status/logs.php?pkg=breezy&ver=3.3.11-1&arch=s390x
https://buildd.debian.org/status/fetch.php?pkg=breezy&arch=s390x&ver=3.3.11-1&stamp=1744775171&raw=0

======================================================================
ERROR: breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)
----------------------------------------------------------------------
testtools.testresult.real._StringException: log: {{{
793.512  creating repository in file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/.bzr/.
793.513  creating branch <breezy.bzr.branch.BzrBranchFormat7 object at 0x3ff17a48b90> in file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/
793.516  trying to create missing lock '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch/.bzr/checkout/dirstate'
793.516  opening working tree '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch'
793.520  preparing to commit
    INFO  Committing to: /tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch/
793.521  Selecting files for commit with filter None
    INFO  Committed revision 1.
793.524  Committed revid b'jrandom@example.com-20250416020608-wtigrtiemex6q4g4' as revno 1.
793.529  preparing to commit
    INFO  Committing to: /tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch/
793.530  Selecting files for commit with filter None
    INFO  added hello
    INFO  Committed revision 2.
793.534  Committed revid b'jrandom@example.com-20250416020608-yk262zf8uf070i3p' as revno 2.
793.535  run brz: ['push', 'sftp://foo:bar@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch', '--directory', 'branch']
793.535  breezy version: 3.3.11
793.535  brz arguments: ['push', 'sftp://foo:bar@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch', '--directory', 'branch']
793.537  opening working tree '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch'
793.540  opening working tree '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch'
793.547  creating repository in sftp://foo@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch/.bzr/.
793.552  Using fetch logic to copy between CHKInventoryRepository('file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/.bzr/repository/')(RepositoryFormat2a()) and CHKInventoryRepository('sftp://foo@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch/.bzr/repository/')(RepositoryFormat2a())
793.552  fetching: <PendingAncestryResult heads:frozenset({b'jrandom@example.com-20250416020608-yk262zf8uf070i3p'}) repo:CHKInventoryRepository('file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/.bzr/repository/')>
793.562  creating branch <breezy.bzr.branch.BzrBranchFormat7 object at 0x3ff17cfcb90> in sftp://foo@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch/
    INFO  Created new branch.
793.574  errors:
'Created new branch.\n'
}}}

Traceback (most recent call last):
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 182, in _read_bytes
    return _get_cache()[key]
           ~~~~~~~~~~~~^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/lru_cache.py", line 61, in __getitem__
    node = cache[key]
           ~~~~~^^^^^
KeyError: (b'sha1:e1e466803f6dd86ee55fe44890ac4a470bb44f07',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1525, in _get_blocks
    yield read_memo, cached[read_memo]
                     ~~~~~~^^^^^^^^^^^
KeyError: (<breezy.bzr.btree_index.BTreeGraphIndex object at 0x3ff17ace890>, 926, 131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/tests/test_upload.py", line 790, in test_upload_without_working_tree
    self.do_full_upload(directory=self.remote_branch_url)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/tests/test_upload.py", line 208, in do_full_upload
    upload.run(up_url, *args, **kwargs)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/commands.py", line 816, in run
    return class_run(*args, **kwargs)
  File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/cmds.py", line 587, in run
    uploader.upload_full_tree()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/cmds.py", line 324, in upload_full_tree
    for relpath, ie in self.tree.iter_entries_by_dir():
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/inventorytree.py", line 323, in iter_entries
    for p, e in inv.iter_entries_by_dir(
                ~~~~~~~~~~~~~~~~~~~~~~~^
        specific_file_ids=inventory_file_ids
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/inventory.py", line 805, in iter_entries_by_dir
    self._preload_cache()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/inventory.py", line 2195, in _preload_cache
    pid_items = self.parent_id_basename_to_file_id.iteritems()
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 566, in iteritems
    self._ensure_root()
    ~~~~~~~~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 162, in _ensure_root
    self._root_node = self._get_node(self._root_node)
                      ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 175, in _get_node
    bytes = self._read_bytes(node)
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 185, in _read_bytes
    bytes = next(stream).get_bytes_as("fulltext")
            ~~~~^^^^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1570, in get_record_stream
    for content_factory in self._get_remaining_record_stream(
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        keys, orig_keys, ordering, include_delta_closure
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1736, in _get_remaining_record_stream
    yield from batcher.yield_factories(full_flush=True)
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1244, in yield_factories
    block_read_memo, block = next(blocks)
                             ~~~~^^^^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1528, in _get_blocks
    zdata = next(raw_records)
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack_repo.py", line 2277, in get_raw_records
    for _names, read_func in reader.iter_records():
                             ~~~~~~~~~~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 320, in iter_records
    self._read_format()
    ~~~~~~~~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 364, in _read_format
    format = self._read_line()
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 288, in _read_line
    line = self._source.readline()
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 252, in readline
    self._next()
    ~~~~~~~~~~^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 236, in _next
    offset, data = next(self.readv_result)
                   ~~~~^^^^^^^^^^^^^^^^^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/transport/sftp.py", line 186, in request_and_yield_offsets
    for (start, length), data in zip(requests, data_stream):
                                 ~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/paramiko/sftp_file.py", line 527, in readv
    yield self.read(x[1])
          ~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/paramiko/file.py", line 203, in read
    new_data = self._read(read_size)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_file.py", line 182, in _read
    data = self._read_prefetch(size)
  File "/usr/lib/python3/dist-packages/paramiko/sftp_file.py", line 162, in _read_prefetch
    self.sftp._read_response()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 887, in _read_response
    t, data = self._read_packet()
              ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/paramiko/sftp.py", line 212, in _read_packet
    x = self._read_all(4)
  File "/usr/lib/python3/dist-packages/paramiko/sftp.py", line 196, in _read_all
    x = self.sock.recv(n)
  File "/build/reproducible-path/breezy-3.3.11/breezy/transport/ssh.py", line 193, in recv
    return self.__socket.recv(n)
           ~~~~~~~~~~~~~~~~~~^^^
  File "/build/reproducible-path/breezy-3.3.11/breezy/tests/fixtures.py", line 162, in signal_handler
    raise TimeoutException(self.timeout_secs)
breezy.tests.fixtures.TimeoutException: 6000
----------------------------------------------------------------------
Ran 33064 tests in 6793.519s

FAILED (errors=1, known_failure_count=49)
2136 tests skipped
Missing feature 'Filesystem treats '\' as a directory separator.' skipped 5 tests.
Missing feature 'Missing tdb' skipped 6 tests.
Missing feature '_winreg' skipped 3 tests.
Missing feature 'apport.report' skipped 1 tests.
Missing feature 'attrib Windows command-line tool' skipped 1 tests.
Missing feature 'breezy._annotator_pyx' skipped 1 tests.
Missing feature 'breezy._chunks_to_lines_pyx' skipped 1 tests.
Missing feature 'breezy._readdir_pyx' skipped 2 tests.
Missing feature 'breezy.bzr._btree_serializer_pyx' skipped 23 tests.
Missing feature 'breezy.bzr._chk_map_pyx' skipped 1 tests.
Missing feature 'breezy.bzr._dirstate_helpers_pyx' skipped 104 tests.
Missing feature 'breezy.bzr._groupcompress_pyx' skipped 15 tests.
Missing feature 'breezy.bzr._knit_load_data_pyx' skipped 27 tests.
Missing feature 'breezy.bzr._rio_pyx' skipped 1 tests.
Missing feature 'case-insensitive case-preserving filesystem' skipped 20 tests.
Missing feature 'case-insensitive filesystem' skipped 1 tests.
Missing feature 'git executable' skipped 34 tests.
Missing feature 'gpg' skipped 3 tests.
Missing feature 'meliae.scanner' skipped 1 tests.
Missing feature 'pyinotify' skipped 16 tests.
Missing feature 'pywintypes' skipped 1 tests.
Missing feature 'quilt executable' skipped 21 tests.
Missing feature 'win32 platform' skipped 9 tests.
breezy.bzr.tests.test_smart_transport.ReadOnlyEndToEndTests.test_mkdir_error_readonly is leaking threads among 6 leaking tests.
1 non-main threads were left active in the end.
brz: warning: some compiled extensions could not be loaded; see ``brz help missing-extensions``
make[1]: *** [debian/rules:23: override_dh_auto_test] Error 1
make[1]: Leaving directory '/build/reproducible-path/breezy-3.3.11'
make: *** [debian/rules:15: binary-arch] Error 2
dpkg-buildpackage: error: debian/rules binary-arch subprocess returned exit status 2

...

Hi Chris,

Is this repeatable or perhaps a flaky test?

Cheers,

Jelmer

Hi Chris,

Is this repeatable or perhaps a flaky test?

Cheers,

Jelmer

Hi Jelmer,

* Jelmer Vernooij <jelmer@jelmer.uk> [250419 18:40]:

It failed three out of three attempts on the buildd. So, not just
flaky I think?

Chris

Hello. In my setup (amd64), this package FTBFS...

100% of the time on machines with 1 CPU
50% of the time on machines with 2 CPUs

So, it does not seem to be architecture-specific (s390x).

I suggest trying GRUB_CMDLINE_LINUX="nr_cpus=1"
since that's what apparently makes the failure rate to be higher.

If that does not work, please contact me privately and
I will gladly provide a VM to debug.

Thanks.

Hi Jelmer,

Any progress on this bug? The FTBFS is hindering the migration to
testing of the fix for RC bug 1099121. We're running out of time for trixie.

Mind you, I very much prefer a targeted for both issues (reverting to
the version in testing) because the diff between unstable and testing is
huge (and thus most likely inappropriate at this stage):
  47 files changed, 835 insertions(+), 3572 deletions(-)

Paul

Unfortunately not - I'm swamped by other stuff (upstream and otherwise), and
skeptical this is a actually a new regression. I haven't had the time to
actually go back and see whether this is reproducible with older versions
though.

Most of the recent delta between unstable and testing was actually for addressing 1099121,
so I'm not sure how much you could reduce that (unless you were to prevent compilation
of some cython files rather than deleting them).

Jelmer

The problematic test is skipped in this version as a workaround.