The bug was opened to track a potential bugs on armel(or others slow
archs, but it does not always happened).

Fix from upstream[0] may take a long time so if you use the package please
keep in mind about this unless the bug was closed.

Thanks.


[0]: https://github.com/fedora-python/tox-current-env/issues/72

Control: retitle -1 test_allenvs_print_extras tests are flaky

I recently uploaded tox/4.14.1-1. tox-current-env autopkgtests fail on
s390x[1] with a similar error, which in turn is currently preventing tox
from migrating into testing[2]

1: https://ci.debian.net/packages/t/tox-current-env/testing/s390x/44050718/
2: https://qa.debian.org/excuses.php?package=tox

This bug was originally for test_allenvs_print_extras_to_file(); for
this I would imagine think the fix is something like:
  - assert prep_tox_output(result.stdout) == expected
  + assert sorted(prep_tox_output(result.stdout)) == sorted(expected)

The s390x autopkgtest above fails on test_allenvs_print_extras(), which
is very similar. However, that one seems to have a sorted() already:
 45s     def test_allenvs_print_extras(print_extras_stdout_arg):
 45s         result = tox(print_extras_stdout_arg)
 45s         expected = []
 45s         for env in envs_from_tox_ini():
 45s             expected.extend(("dev", "full", f"{env}: OK"))
 45s         expected.pop()  # The last "py310: OK" is not there
 45s         expected.append(tox_footer(spaces=0))
 45s         expected = ("\n".join(expected)).splitlines()
 45s >       assert sorted(prep_tox_output(result.stdout).splitlines()) == sorted(expected)

I don't fully grok what this is supposed to be doing (but I've looked at
it for only a couple of minutes). That .pop() does look suspicious
though, in the sense that it operates before the output is sorted.

Faidon

Hi,

Thanks your detailed analysis for it. I have uploaded -3 to unstable to
hope to unblock tox migrating. But these are some unexpected maybe.
https://salsa.debian.org/python-team/packages/tox-current-env/-/blob/debian/main/debian/patches/sorted-output-on-tests.patch

It seems it works but I was missing another code snipshot which need be
`sorted` also:
https://salsa.debian.org/python-team/packages/tox-current-env/-/blob/debian/main/tests/test_integration_tox4.py#L217
```
@pytest.mark.parametrize("option", ("--print-deps-to", "--print-deps-to-file"))
def test_allenvs_print_deps_to_file(tmp_path, option):
     depspath = tmp_path / "deps"
     result = tox(option, str(depspath))
     assert sorted(depspath.read_text().splitlines()) == sorted(
         ["tox", "six", "py"] * len(envs_from_tox_ini())
     )
     expected = ""
     for env in envs_from_tox_ini()[:-1]:
         expected += f"{env}: OK\n"
     expected += tox_footer(spaces=0) + "\n"
     assert prep_tox_output(result.stdout) == expected
```

This will lead a autopkgtest on s390x[0] again from its tracker page[1] to
get the link(UTC+8 13:06 2024/03/22).

But another retry on s390x it passed on s390x[2].

[0]: https://ci.debian.net/packages/t/tox-current-env/testing/s390x/44235641/
[1]: https://tracker.debian.org/pkg/tox-current-env
[2]: https://ci.debian.net/packages/t/tox-current-env/

So if many retries on debci page and it passes all autopkgtest, does it
still block your migration?
For safe to unblock anything, I skip this:
https://salsa.debian.org/python-team/packages/tox-current-env/-/blob/debian/main/debian/patches/flaky-test.patch

Do you think I will need to upload it to unstable again with `sorted`
for `test_allenvs_print_deps_to_file` as I analyzed above?

Hi Bo,

Thanks for the short reaction time and for spending your time on this
bug, much appreciated!

tox has migrated since. Honestly I'm not sure why, given the regression!
So there isn't as much urgency as there was when I filed this bug, I'd
say.

Yup, looks like the same bug all over the place, effectively a race.

It's clearly a bug IMHO, so it'd be good to fix at some point. No
urgency with regards to the tox migration as I said, but this will
happen again on e.g. the next tox upload.

If I were you I'd fix it in git, submit all of these fixes to upstream,
wait a month or two so in case upstream releases a new version
(hopefully with these), and then upload regardless.

Thanks again,
Faidon