When the reproducible=+fixfilepath feature is enabled (either through
DEB_BUILD_OPTIONS, or using a dpkg that enables this by default),
openorienteering-mapper fails to build from source:

http://qa-logs.debian.net/2020/09/26.fixfilepath/openorienteering-mapper_0.9.3-1_unstable_fixfilepath.log

While the "fixfilepath" feature is not currently enabled by
dpkg-buildflags by default, it may become the default at some point in
the future, and can by triggered manually by setting
DEB_BUILD_OPTIONS=reproducible=+fixfilepath in the build environment.

More information about this issue is available at:

https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

I have not identified the exact cause of this issue for
openorienteering-mapper, but a common triggering issue is test suites
expectinfg __FILE__ to resolve to an absolute path.

The attached patch works around this issue by disabling the fixfilepath
feature in debian/rules using DEB_BUILD_MAINT_OPTIONS=-fixfilepath.

Thanks for maintaining openorienteering-mapper!


live well,
  vagrant

Am 16.10.20 um 01:19 schrieb Vagrant Cascadian:
So far, all cases in openorienteering-mapper were tests which were
expected to be run in the build environment and indeed access the
pristine test data in the source directory.

The current issue comes from using Qt's QFINDTESTDATA(), which relies on
a cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory
from which the compiler is invoked" in order to "the absolute path of
the source directory" [!],
https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA . QT_TESTCASE_BUILDDIR
is defined by Qt's cmake file. I do see some inconsistency there, but
this is a different story.

In previous cases I "solved" the failing reproducible builds by: using
another macro, carrying the source directory. But I'm not sure if this
is what is intented. While I do have ideas how to workaround this in
other ways, I would appreciate a clear recommendation how test data in
the source dir should be handled.

Thanks,
Kai

Am 16.10.20 um 01:19 schrieb Vagrant Cascadian:
So far, all cases in openorienteering-mapper were tests which were
expected to be run in the build environment and indeed access the
pristine test data in the source directory.

The current issue comes from using Qt's QFINDTESTDATA(), which relies on
a cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory
from which the compiler is invoked" in order to "the absolute path of
the source directory" [!],
https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA . QT_TESTCASE_BUILDDIR
is defined by Qt's cmake file. I do see some inconsistency there, but
this is a different story.

In previous cases I "solved" the failing reproducible builds by: using
another macro, carrying the source directory. But I'm not sure if this
is what is intented. While I do have ideas how to workaround this in
other ways, I would appreciate a clear recommendation how test data in
the source dir should be handled.

Thanks,
Kai

... sorry for the delay...

Yes, this is part of the known issues with test data mentioned; the
majority of known build failures triggered by fixfilepath are these
use-cases with QFINDTESTDATA.

If the package in question only uses such features in test suites that
are not shipped in the binary packages, it's perfectly reasonable to
disable the fixfilepath feature; it will likely have no effect on the
resulting packages either way.

If the package embeds file paths in files shipped in the binary
packages, then it might be worth some of the workarounds you suggested
with the test suites, and further debugging what exactly is embedding
the build paths; enabling fixfilepath only catches some of the issues of
embedded build paths, so it may be a moot point for any particular
package.

Disabling fixfilepath in your package will allow
tests.reproducible-builds.org to be able to test builds of the
openorienteering-mapper package in unstable and experimental again,
where the build path is varied, and possibly help identify weather
further exploration is needed.

At this point, I'd suggest disabling fixfilepath for this particular
package. But I submitted the patch to do that, so maybe I am biased? :)

Hope that is a clear enough recommendation?


Oh, and, sorry for not getting back to you till now! Not sure how I lost
track of this...


live well,
  vagrant

Am 10.11.20 um 01:00 schrieb Vagrant Cascadian:

Trying to express my concerns in terms of reproducible-builds.org
terminology definitions:

As the author of the software, I define the "expected reproducible
artifacts" to be the files created by "make install".
With disabling fixfilepath, is there another test which not only
verifies "bit-by-bit identical copies of all specified artifacts", but
would also offer the diagnosis that the build path ended up in the
installed artifacts?

Last not least, "make install" is the canonical way of creating the set
of artifacts meant for distribution.

Thanks!
Kai

I'm not sure what exactly you are asking, but here is my attempt to
answer... :)


In this case, DEB_BUILD_OPTIONS or
DEB_BUILD_MAINT_OPTIONS=reproducible=+fixfilepath merely tells
dpkg-buildflags to set various flags (e.g. CFLAGS) to include
-ffile-prefix-map=BUILDPATH=. It does not verify reproducibility of
resulting artifacts. It can lead to artifacts which are easier to build
reproducibly in some cases.


Determining weather the artifacts are reproducible is done with
mechanisms such as comparing checksums of the artifacts, and diagnosis
of reproducibility issues when the checksums do not match is done by
comparing the resulting artifacts of multiple builds using tools such as
diffoscope... or tools such as reprotest which perform multiple builds
varying a variety of things, and then compares the results with diff or
diffoscope.

Lintian appears to be able to detect build path in some cases:

https://lintian.debian.org/tags/file-references-package-build-path.html


For openorienteering-mapper, it looks like it is affected by at least
one issue unrelated to build path:

https://tests.reproducible-builds.org/debian/rb-pkg/bullseye/amd64/diffoscope-results/openorienteering-mapper.html

Specifically:

https://tests.reproducible-builds.org/debian/issues/timestamps_in_qhc_issue.html

But test.reproducible-builds.org builds using bullseye do not vary the
build path, so you cannot know if setting fixfilepath would affect the
results or not.


The tests.reproducible-builds.org builds using unstable do vary the
build path, but also sets DEB_BUILD_OPTIONS=reproducible=+all which
enables the fixfilepath feature, triggering a build failure in
openorienteering-mapper due to use of QFINDTESTDATA in the test suites,
and hence this bug report.


So if you want to see if fixfilepath affects the build, disable the test
suite and run one build with it enabled and another with it disabled,
with a different build path each time, changing as little as possible
otherwise, and see if you get differences related to the build path in
the resulting artifacts.


Another option for openorienteering-mapper is if you could filter out
the -ffile-prefix-map and -fmacro-prefix-map flags from CFLAGS (or other
relevent *FLAGS) in the test suite, since the test suite relies on
behavior that these flags change incompatibly. That's a bit more
complicated than the proposed one-line change to debian/rules, but would
be upstreamable.


Hope that helps.


live well,
  vagrant

One way this is often done, particularly in the GNOME ecosystem, is to
check for an environment variable that is set by the build system while
running tests. There are often two environment variables - one for the
source directory and one for the build directory - so that tests can
either ask for data files that are distributed with the source code, or
data files that were compiled along with the test itself and placed in
the build directory, if those directories are different.

If the environment variable is not set, there's a fallback that would
be reasonable to use if the test has been installed system-wide,
typically into /usr/libexec/installed-tests (which is something
that various GNOME and GNOME-adjacent packages support doing, for
"as-installed" testing like Debian's autopkgtest: see
<https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests>).
In GLib the fallback is to use dirname(argv[0]), but hard-coding an
installation path would also be reasonable here.

This avoids having to hard-code the path to either the source or build
directory into any binaries, even if test executables and data are going
to be installed into /usr/libexec/installed-tests for "as-installed"
testing.

Some packages need to set environment variables for build-time testing
*anyway*, so that third-party components will find their required data
in the source or build tree: for example, you might need to set PATH,
LD_LIBRARY_PATH, XDG_DATA_DIRS and similar variables. The variables used
by this particular package's tests can be set in the same way.

The implementation that is normally used in GNOME is GLib's
g_test_build_filename(), which works something like this pseudocode:

    if file_type == G_TEST_DIST:
        dir = getenv("G_TEST_SRCDIR")
    elif file_type == G_TEST_BUILT:
        # this branch is the closest equivalent of QFINDTESTDATA
        dir = getenv("G_TEST_BUILDDIR")
    else:
        fatal error

    if dir is null:
        dir = directory containing the running executable

    return join_paths(dir, first_path, ...)

An implementation of the build-system side of this in a simple
Makefile-based build system would look something like:

    export G_TEST_BUILDDIR = $(CURDIR)
    export G_TEST_SRCDIR = $(srcdir)

    check:
            ./test-foo
            ./test-bar

Obviously the code would look a bit different for Autotools, CMake or
Meson, but all are capable of doing this (Autotools uses
AM_TESTS_ENVIRONMENT, CMake uses
set_tests_properties(... PROPERTIES ENVIRONMENT ...), Meson uses
test(..., env : ...)).

I assume qmake would also be able to do this, but I don't know how.

    smcv

Am 10.11.20 um 11:35 schrieb Simon McVittie:
I did consider these options, but I am not convinced. One perspective I
am looking at is onboarding of new contributors (upstream).
Reproducibility helps with that, no doubt. But with your favourite IDE
as a development tool, or at the command line, why to make it hard to
just run a particular test executable? Why add complexity to each and
every package's source code or build system?

... or how to fix QFINDTESTDATA? This macro is the canonical way with
Qt. It should be made compatible with reproducible builds. The solution
might include environment variables, but implemented by Qt Test, not
implemented in each and every package.

And then one more step, and standardize the variables' names (like
SOURCE_DATE_EPOCH).