#1151066 neutron: FTBFS: testtools.matchers._impl.MismatchError

Package:
src:neutron
Source:
src:neutron
Submitter:
Santiago Vila
Date:
2026-10-17 23:07:01 UTC
Severity:
normal
Tags:
#1151066#5
Date:
2026-10-16 16:03:35 UTC
From:
To:
Dear maintainer:

During a rebuild of all packages in unstable, this package failed to build.


Note: My build environment is made of machines with 1 and 2 CPUs.
On machines with 1 CPU, I have not seen this package to fail yet.
On machines with 2 CPUs, I'm getting a failure rate around 100%.
I hope this is meaningful for the authors.


Below you will find the last part of the build log (probably the most
relevant part, but not necessarily). If required, the full build log
is available here:

https://people.debian.org/~sanvila/build-logs/202610/

About the archive rebuild: The build was made on virtual machines from AWS,
using sbuild and a reduced chroot with only build-essential packages.

If you cannot reproduce the bug please contact me privately, as I
am willing to provide ssh access to a virtual machine where the bug is
fully reproducible.

If this is really a bug in one of the build-depends, please use
reassign and add an affects on src:neutron, so that this is still
visible in the BTS web page for this package.

Thanks.
--------------------------------------------------------------------------------
[...]
======================================================================
FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test__enter__
neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test__enter__
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/neutron/tests/base.py", line 176, in func
    return f(self, *args, **kwargs)
  File "/<<PKGBUILDDIR>>/neutron/tests/unit/agent/common/test_resource_processing_queue.py", line 62, in test__enter__
    self.assertNotIn(FAKE_ID, queue.ExclusiveResourceProcessor._primaries)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 562, in assertNotIn
    self.assertThat(haystack, matcher, message)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 704, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: {'7b37687e-a5ea-431b-93da-1aa3a152acee': <neutron.agent.common.resource_processing_queue.ExclusiveResourceProcessor object at 0x7f63f028c550>, '1392e4c0-6fb5-498c-a315-6e8021702876': <neutron.agent.common.resource_processing_queue.ExclusiveResourceProcessor object at 0x7f63bfc3e050>} matches Contains('7b37687e-a5ea-431b-93da-1aa3a152acee')

----------------------------------------------------------------------
Ran 21377 tests in 1923.587s

FAILED (failures=6, skipped=1769)
+ echo ======> STESTR TEST SUITE FAILED FOR python3.14: displaying pip3 freeze output...
======> STESTR TEST SUITE FAILED FOR python3.14: displaying pip3 freeze output...
+ [ -x /usr/bin/pip3 ]
+ exit 1
make[1]: *** [debian/rules:67: override_dh_install] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:12: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess failed with exit status 2
--------------------------------------------------------------------------------
#1151066#10
Date:
2026-10-16 16:15:17 UTC
From:
To:
above figures (obtained with -4) may no longer be accurate.

Thanks.
#1151066#15
Date:
2026-10-16 16:58:09 UTC
From:
To:
From my phone quickly...


Indeed, many tests were leaking since using stestr dynamic scheduler, and I fixed many with help from upstream. It would be nice if you could retry.


Cheers,


Thomas
#1151066#20
Date:
2026-10-16 17:33:04 UTC
From:
To:
Retried, and failure rate is still around 95% on machines with 2 CPUs.

I've put several build logs in the same directory as before.

This is the number of times each test failed:

   7 FAIL: neutron.tests.unit.services.network_segment_range.test_plugin.TestNetworkSegmentRange.test__are_allocated_segments_in_range_unimpacted
   6 FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test_primary
   6 FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test__exit__non_primary_does_not_clean_resource_timestamps
   5 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_unsupported_logging_type
   5 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_resource_id_not_exists
   5 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_invalid_resource_constraint
   5 FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test_i_am_primary
   5 FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test__exit__cleans_resource_timestamps
   5 FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test__exit__
   4 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_target_id_not_exists
   4 FAIL: neutron.tests.unit.agent.common.test_resource_processing_queue.TestExclusiveResourceProcessor.test__enter__
   3 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_resource_id_not_exists
   3 FAIL: neutron.tests.unit.extensions.test_segment.TestNovaSegmentNotifier.test_delete_network_and_owned_segments
   2 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_with_snat_disable
   2 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_with_not_set_gw_port
   2 FAIL: neutron.tests.unit.agent.linux.test_daemon.TestUnwatchLog.test_unwatch_log
   1 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_resource_id_not_specific

Thanks.
#1151066#23
Date:
2026-10-16 23:46:08 UTC
From:
To:
Hello,

Bug #1151066 in neutron reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/openstack-team/services/neutron/-/commit/f54bf4c9ec0726ee2cad848d30cea70d314df98a
------------------------------------------------------------------------
* Fix the class level state leaks in the resource processing queue
    tests (Closes: #1151066):
    - fix-test-resource-processing-queue-state.patch
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1151066
#1151066#30
Date:
2026-10-17 00:04:57 UTC
From:
To:
We believe that the bug you reported is fixed in the latest version of
neutron, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1151066@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Thomas Goirand <zigo@debian.org> (supplier of updated neutron package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Sat, 17 Oct 2026 01:32:28 +0200
Source: neutron
Architecture: source
Version: 2:29.0.0-6
Distribution: unstable
Urgency: medium
Maintainer: Debian OpenStack <team+openstack@tracker.debian.org>
Changed-By: Thomas Goirand <zigo@debian.org>
Closes: 1151066
Changes:
 neutron (2:29.0.0-6) unstable; urgency=medium
 .
   * Fix the class level state leaks in the resource processing queue
     tests (Closes: #1151066):
     - fix-test-resource-processing-queue-state.patch
Checksums-Sha1:
 a629954194d265621388f32d379e1c0439e3f45d 4965 neutron_29.0.0-6.dsc
 b621cda80d49b982ef79daea386a778e4fa1280c 49288 neutron_29.0.0-6.debian.tar.xz
 768e66fc16ff2aed6a3c11f7725b48ce2b7cd2d5 21837 neutron_29.0.0-6_amd64.buildinfo
Checksums-Sha256:
 4b5c46253d7daa6740a9a216c9a98a4b7209d162c8db5360856abd2ba11cb39e 4965 neutron_29.0.0-6.dsc
 0f4c0fce3c46da3337d6b24b3512dede9b57b18f96309a4885dce65b8cd28d01 49288 neutron_29.0.0-6.debian.tar.xz
 6284208f7c73bf00e617add31e972cd6afdd723290813b5959492c2ae83ab4b1 21837 neutron_29.0.0-6_amd64.buildinfo
Files:
 2866081130c48d9522c93da137bf5feb 4965 net optional neutron_29.0.0-6.dsc
 ffbfc79ecf0bac6a2c5a0a323ace06a3 49288 net optional neutron_29.0.0-6.debian.tar.xz
 d6fb2ca3f0a12c927b3a90cf94264f36 21837 net optional neutron_29.0.0-6_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEoLGp81CJVhMOekJc1BatFaxrQ/4FAmrSt0EACgkQ1BatFaxr
Q/6L0Q/9GLoplJs/NYdLHXjKol6lkFvdNJRdlFwUakGjhJXBFkNJgb3P0FbO+kWP
8pbJmaEfW6rweALyU3dBXyNk8p6H7KjPNmDqU5mGdcVWHbUsfxsDGJUvNDAdjIhK
M5iIM+sV9E4NiNWGQnWISA5/fakMNzrzaesi4uwnhS/IhNMGpzShSaopu19r4c2J
cc0xHEmEDqIgIy+lcWUJJb+ur+zDBA0MH1hoEQbTO3Sa/n/k2zOk8PfBuO6AyUUN
duTahZhiCoLPk5TFnZy9Y8cInw9Dtbj6GIaWlq2cNRLICHJ4tUVIXrX2xYThsfXz
nWcHc2gmAk8QcDW5MHwGkiC+f0PW2lSAKk82ZoX7cyzUJgE8aAyP3ZNLk57iU5z7
YltyKEmTVTnE356X9QVyZH74GWcZESQSUiJH/SZ1Sy0WdI95jVo+rg+Md038L9vE
p06iMsRnitTJqJNJ9RokZteyT5IcPvcXNL7pBFfmv+6DtF1hUtOJJrykd6ZwFst5
nx8ZBchVxnUt/zQldC3AhaQNZVPLj1EuLpU4bkCbv5Pdm42b9Q7FUKp4FwBa5UAj
wmUQlKsPPJnrz7Av7ThGwlxaDsB2OGREYu6GdBgsSvt+z4KGCOhKiXGyKTkbxNq0
GajihHQljInHxKTTZpvOsz2NmjYZC5I4UAVmqo6WL9MsAcc5ojk=
=M4lE
-----END PGP SIGNATURE-----
#1151066#35
Date:
2026-10-17 15:34:25 UTC
From:
To:
reopen 1151066
thanks

Hi. I tried to build 29.0.0-6 many times (25) and it failed
many times (21). The failures I get are very similar to the ones I got
in 29.0.0-5, so I assume this is still the same bug.

I've put all those build failures in the same directory as before:

https://people.debian.org/~sanvila/build-logs/202610/

and this is an histogram of failures:

   8 FAIL: neutron.tests.unit.extensions.test_segment.TestNovaSegmentNotifier.test_delete_network_and_owned_segments
   8 FAIL: neutron.tests.unit.agent.linux.test_daemon.TestUnwatchLog.test_unwatch_log
   7 FAIL: neutron.tests.unit.services.network_segment_range.test_plugin.TestNetworkSegmentRange.test__are_allocated_segments_in_range_unimpacted
   7 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_resource_id_not_exists
   4 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_unsupported_logging_type
   3 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_with_not_set_gw_port
   3 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_target_id_not_exists
   2 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_with_snat_disable
   2 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_with_disable_events
   2 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_resource_id_not_exists
   2 FAIL: neutron.tests.unit.services.logapi.common.test_sg_validate.TestSGLogRequestValidations.test_validate_request_invalid_resource_constraint
   1 FAIL: neutron.tests.unit.services.logapi.common.test_snat_validate.TestSnatLogRequestValidations.test_validate_request_resource_id_not_specific


As before, the package still builds ok on systems with 1 CPU.
(tried 25 times, failed none).

Thanks.
#1151066#42
Date:
2026-10-17 22:48:48 UTC
From:
To:
Hello,

Bug #1151066 in neutron reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/openstack-team/services/neutron/-/commit/f0ef6ee18747e587c30c27abadf518d78fbf7263
------------------------------------------------------------------------
* Fixed many leaking tests as per below (Closes: #1151066):
    - Fixed leaking tests in services/network_segment_range/test_plugin.py
      that leads to FTBFS, adding patch "Copy the class level network
      segment range fixture before mutating it":
      - fix-test-network-segment-range-fixture-mutation.patch
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1151066
#1151066#49
Date:
2026-10-17 23:04:52 UTC
From:
To:
We believe that the bug you reported is fixed in the latest version of
neutron, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1151066@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Thomas Goirand <zigo@debian.org> (supplier of updated neutron package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Sun, 18 Oct 2026 00:29:17 +0200
Source: neutron
Architecture: source
Version: 2:29.0.0-8
Distribution: unstable
Urgency: medium
Maintainer: Debian OpenStack <team+openstack@tracker.debian.org>
Changed-By: Thomas Goirand <zigo@debian.org>
Closes: 1151066
Changes:
 neutron (2:29.0.0-8) unstable; urgency=medium
 .
   * Fixed many leaking tests as per below (Closes: #1151066):
     - Fixed leaking tests in services/network_segment_range/test_plugin.py
       that leads to FTBFS, adding patch "Copy the class level network
       segment range fixture before mutating it":
       - fix-test-network-segment-range-fixture-mutation.patch
     - Fixed racy tests in extensions/test_segment.py that leads to
       FTBFS, adding patch "Fix the segment notifier tests batching
       races":
       - fix-test-segment-notifier-batch-wait.patch
     - Fixed leaking tests in agent/linux/test_daemon.py that leads to
       FTBFS, adding patch "Clear the oslo_log logger cache in the
       unwatch_log test":
       - fix-test-unwatch-log-logger-cache.patch
Checksums-Sha1:
 55a63671f0d0db472247006b82572e51b0e58b6c 4965 neutron_29.0.0-8.dsc
 3c0c2754b0f86fb2ef56fb508e17905ec1348f76 51588 neutron_29.0.0-8.debian.tar.xz
 61eeac70f6f46bee89d24e720690bd25725bdd5c 21837 neutron_29.0.0-8_amd64.buildinfo
Checksums-Sha256:
 074128a70e125c7bb217ca96b47f1c4fe91fc6887e75d98a46f2b956cd947a83 4965 neutron_29.0.0-8.dsc
 dbf63d6c1367c77e7bce1f2a3aac6f67bdf03b5e18b83a2476b9904b81b1a555 51588 neutron_29.0.0-8.debian.tar.xz
 7905b9dc4072e6180d00450b719affa3803571ded3911ed9bbb2b9ad8e204632 21837 neutron_29.0.0-8_amd64.buildinfo
Files:
 eb1d23cc3d4f754e33188eb4aac747d0 4965 net optional neutron_29.0.0-8.dsc
 0ccc319755cf32ba5c3be98775f1dbdb 51588 net optional neutron_29.0.0-8.debian.tar.xz
 cd337dc140241133488f13750a197487 21837 net optional neutron_29.0.0-8_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEoLGp81CJVhMOekJc1BatFaxrQ/4FAmrT+1IACgkQ1BatFaxr
Q/4AnBAAi/YZT2cm3FN3gx75TX6/quuDbUk7VX1DUEAKoScvsK9O/L6s64h6u2tm
qekZkaa8jxbBiCm0INGARpJmU23TTMRvqtnmgt5BC2gZITlZgVWXMvECG1HAPlcs
9MAP9fUSge4UK1ggLblFgR0ZljQirFSP6ykLm/nx0flB5hIMwDUZb49W11hkEebh
0saOWK4jrjS1D7ZgUeFe9l4cWv6RxcvCh9qv08FI2+6Sp/FO5lXcKzWj79WvAwRP
ccEIfqLWGIX5UpWbQVpS9pwQMFQS0/cP5fN++xI/YTJjJf7p7zz/FdtvvT1/hUBd
Sf0SUOtYI4XPlioV52VgW72jrBqwv/oRwYLG3ClvdmpiwOVHE4alx+JS+hTFravy
79sLVQ24Y9HCphrbf+Tm14F0dNzHlxjW3zVtJ9xCx6Qa04To5UO3u8Gvsmto2N6/
mMnqOoJ7aWeYU/QsqW9JQjKtcoZ+5pZxB3yuJAQV2ls5pJ/rAInXqqPgxpfC48T/
x8mlj8m1j0gLh/89PKb8TcslRtAjGxeiidj6zNEh/fQxa5NsWwRfY20YpHp/tnoP
yyGyJrII4bU6uoQG0fCqj5JqjaEP9GhPpwUYWfTCL6OpOwis0wc+gqEnEGlA+fIC
shSMcR/AwZ0UiAg0Z9+nWfPy5WJPu4uNuFsLFLVNdMol5P1S7Ok=
=p86a
-----END PGP SIGNATURE-----