Dear maintainer,

The package asdf-astropy recently switched to use pyproject.toml, and
now the installation is myteriously incomplete: The package consists of
a number of subpackages:

$ du asdf_astropy
8	asdf_astropy/io/tests
16	asdf_astropy/io
8	asdf_astropy/resources/schemas/table
8	asdf_astropy/resources/schemas/time
[...]

When building interactively (on the shell) in a sid environment, the
build is often complete -- it seems to depend on the installed packages
(when git is installed, it is complete; without git often not).

When building with pbuilder, on Salsa [1], or on buildd [2], the package
is however missing all subpackages:
/builds/debian-astro-team/asdf-astropy/debian/output/source_dir/.pybuild/cpython3_3.11

/usr/lib/python3/dist-packages/setuptools/config/pyprojecttoml.py:108:
_BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml`
is still *beta*.
   warnings.warn(msg, _BetaConfiguration)
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/asdf_astropy
copying asdf_astropy/_version.py -> build/lib/asdf_astropy
[... no asdf_astropy/io files here ...]
* Building wheel...
Successfully built asdf_astropy-0.3.0-py3-none-any.whl
I: pybuild plugin_pyproject:118: Unpacking wheel built for python3.11
with "installer" module
------------------------------8<---------------------------------

On a discussion on the debian-python@l.d.o list, Scott Kitterman
suggested [3] that this may be a bug in python3-installer; that's why
the bug is filed here. Please re-assign if not.

This problem currently prevents a successfull build for asdf-astropy.

Best regards

Ole

[1] https://salsa.debian.org/debian-astro-team/asdf-astropy/-/jobs/3605729
[2]
https://buildd.debian.org/status/fetch.php?pkg=asdf-astropy&arch=all&ver=0.3.0-1&stamp=1670016252&raw=0
[3] https://lists.debian.org/debian-python/2022/12/msg00036.html

Hi!

Looking at the asdf-astropy package, this is a bug in the upstream build
system.

The sequence of steps for building with pybuild's
pybuild-plugin-pyproject plugin is that the upstream build system is
asked to make a wheel, and then python3-installer unpacks the wheel;
looking at the wheels that the upstream build system is producing, they
are missing the data files and python3-installer is faithfully unpacking
these wheels. (Hence reassigning back to src:asdf-astropy.)

The bug is that the upstream build system sometimes produces wheels that
do not contain all the files. Getting data files into packages is
notoriously difficult. There are many sharp edges.

Where setuptools_scm is used, as in asdf-astropy:
  - when run inside a git checkout with git installed, all the data files
    that are checked into git get included in the wheel.
  - when run outside of a git checkout or when git is not installed, the
    other setuptools config (pyproject.toml and MANIFEST.in) is used to
    find the package files.

The latter accounts for your observation that it depends on what
packages are installed, particularly on the presence of git.

Looking at the config for asdf-astropy, there is not enough config for
setuptools (outside setuptools_scm) to pick up the data files and so
they are missing when built outside of a git checkout.

There are lots of ways to fix this and one needs to figure out
precedence rules across the multiple difference places it can be
configured. In the attached patch, I push the data files into the wheel
by explicitly including it as a 'graft' in MANIFEST.in [1].

[1]
https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in

The balance of the attached patch just lets pybuild install the module
as normal.

I see in the packaging that the tests are currently suppressed in
d/rules. The tests would currently fail if the wheel does not contain
all the files because the tests are run against unpacked wheel in
.pybuild/ (not the source - the purpose is to test the packaging, not
test that upstream's CI works). I've not looked at enabling these tests
as it looks like a bit of knowledge about various pytest plugins and
asdf plugins is needed for that.

regards
Stuart