#1041538 dh-python: Unhelpful inclusion of optional packages in Depends

Package:
dh-python
Source:
dh-python
Submitter:
Scott Kitterman
Date:
2025-02-03 17:36:08 UTC
Severity:
normal
Tags:
#1041538#5
Date:
2023-07-20 14:03:12 UTC
From:
To:
I had to override dh_python3 to add --no-guessing-deps in the latest
dnspython upload because it was getting things wrong.  Here's what it
generated:

python3-httpcore | python3 (<< 3.8), python3-sniffio, python3:any

The correct answer here is actually use python3:any.  Here's what I
think is going on:

From the pyproject.toml file:

[tool.poetry.dependencies]
python = "^3.8"
httpx = {version=">=0.24.1", optional=true, python=">=3.8"}
httpcore = {version=">=0.17.3", optional=true, python=">=3.8"}
h2 = {version=">=4.1.0", optional=true, python=">=3.8"}
idna = {version=">=2.1,<4.0", optional=true}
cryptography = {version=">=2.6,<42.0", optional=true}
trio = {version=">=0.14,<0.23", optional=true}
sniffio = {version="^1.1", optional=true}
wmi = {version="^1.5.1", optional=true}
aioquic = {version=">=0.9.20", optional=true}

...

[tool.poetry.extras]
doh = ['httpx', 'h2']
idna = ['idna']
dnssec = ['cryptography']
trio = ['trio']
wmi = ['wmi']
doq = ['aioquic']

There are two issues:

1.  httpcore and sniffio shouldn't be listed at all.  They are optional.
I suspect that either poetry or dh-python is looking at the extras
section and since those packages aren't listed for one of the extras, it
assumes the package is required, despite the optional flag.  They
probably should be listed somewhere (upstream bug), but I think if it
says optional, it shouldn't be added to Depends.

2.  Generating an optional depends on python3 << 3.8 isn't helpful.  It
looks to me like {version=">=0.17.3", optional=true, python=">=3.8"} is
being mis-interpreted.  I believe the intent here is to say that the
dependency is optional when python3 > 3.8, not you need it if python3 >
3.8.  There's a larger question of why the interpreter version is there
at all, given that's now the minimum python3 version supported, but
that's an upstream issue, we ought to get it right.

Scott K
#1041538#10
Date:
2023-07-20 14:58:03 UTC
From:
To:
[Scott Kitterman, 2023-07-20]
dh_python3's manpage or /usr/share/doc/dh-python/README.PyDist for more
details)

dh_python3 doesn't look at source files. It generates dependencies by
parsing installed requires.txt file so IMHO it's an upstream / poetry bug

can you show me how poetry / pyproject / whatever parsed this file
interpreted it? I.e. what's in installed requires.txt

(I suspect it's a hard dependency there)

I agree
#1041538#15
Date:
2023-07-20 15:12:11 UTC
From:
To:
I have so far managed to avoid learning anything about how poetry works
internally (learning about flit fully exhausted my curiosity in this area).

Emmanuel,

Can you help out with this?

Thanks,

Scott K
#1041538#20
Date:
2023-07-20 15:28:01 UTC
From:
To:
Hi Piotr (2023.07.20_14:58:03_+0000)

Requires-Dist: httpcore (>=0.17.3) ; python_version >= "3.8"
Requires-Dist: sniffio (>=1.1,<2.0)

I'd call this an upstream poetry issue.

It shouldn't declare optional dependencies as required. It should rather
create an extra to encapsulate any optional dependencies that aren't
part of a defined extra.

Stefano
#1041538#25
Date:
2023-07-21 08:02:57 UTC
From:
To:
Control: reassign -1 poetry
Control: affects -1 dh-python

Reassigning the bug.

Stefano
#1041538#36
Date:
2023-07-28 19:42:50 UTC
From:
To:
Hi,

Sorry for this delay. Yes as you mentioned Scott, you need to add the
optional dependency in the extras section, not only marked it as
optional=true.

Poetry documentation[0] is not very explicit with that.

There's an issue open [1]. I can work in a patch to at least raise an
exception o message when a dependency is marked as optional and is not
added to extras. But I agree that mark a deps as optional and then is
not optional is confusing.

[0] https://python-poetry.org/docs/pyproject/#extras
[1] https://github.com/python-poetry/poetry/issues/2357


Cheers,
Emmanuel
#1041538#45
Date:
2023-07-28 20:00:49 UTC
From:
To:
Thanks,

For this particular package I pointed this out to upstream and it's fixed in
their latest bug fix update.  They were quite surprised that marking a
dependency optional didn't actually do anything, so I think it would be good
to get poetry to handle this better.

Scott K