While the reproducible=+fixfilepath and reproducible=+fixdebugpath are very helpful in stripping out the build path from binaries, many binaries still end up including the build path because they also embed the compiler flags (e.g. -ffile-prefix-map=/path/to/build-XYZ=.): https://tests.reproducible-builds.org/debian/issues/unstable/records_build_flags_issue.html At the moment, we have identified over 150 packages in Debian affected by this issue, though there are probably more that have not yet been identified. Some packages may work around this issue by sanitizing or stripping -ffile-prefix-map and -fdebug-prefix-map, but handling this on a per-package basis is a bit of a whack-a-mole approach. The attached patch attempts to switch fixfilepath and fixdebugpath to use .spec files with the DEB_BUILD_PATH environment variable, which is currently used internally for the fixfilepath and fixdebugpath features. Note that it is only a partial patch; it doesn't export DEB_BUILD_PATH in dpkg-buildpackage and/or dpkg-buildflags; help as to the best place to implement that would be appreciated! Instead of changing fixfilepath/fixdebugpath, it might also be better to add new options instead, as this is a significant change of the implementation. Interested to hear your thoughts everyone! live well, vagrant
Hi! Ah I like this direction, yes! The ideal place to export such variable would be from dpkg-buildpackage, but sadly as long as that's not the canonical entry point for building packages, I'm seeing only doing that if the implementation can avoid assuming the variable is set and shows no regressions in that case. If we'd be worried that the new behavior can break stuff, then that might make sense, but that would either imply that we need to manually switch current packages to explicitly use the new ones, or we need to flip the new ones as the default and disable the old ones, in case some package needs the old behavior. But that all seems a bit unnecessary to me? I'm not sure the equal sign is correct here? Also there's a significant and potentially problematic behavior change here. The perl code currently uses Cwd::getcwd(), which will return an absolute pathname, which will be stable regardless of where the option is being used. Instead when passing «.» that suddenly becomes relative to the call site of each compiler invocation (not even of the dpkg-buildflags call. :/ Ditto. Thanks, Guillem
... Great! I noticed SOURCE_DATE_EPOCH was defined in: /usr/share/dpkg/pkg-info.mk:SOURCE_DATE_EPOCH ?= $(call dpkg_late_eval,SOURCE_DATE_EPOCH,dpkg-parsechangelog -STimestamp) Would it make sense to do the same for DEB_BUILD_PATH? Or would CURDIR not be the right value at that time? Are there cases where SOURCE_DATE_EPOCH isn't defined when building packages (e.g. fakeroot debian/rules build). From a reproducible builds perspective, it seems SOURCE_DATE_EPOCH is working well enough, and defining DEB_BUILD_PATH in the same place should be good enough too. Maybe so. *If* we do it well, there should be no real chance of breakage! :) I'm not sure what's wrong with it ... Currently fixdebugpath would set something like: -fdebug-prefix-map=/absolute/path/to/build/dir=. This gets the value of DEB_BUILD_PATH from an environment variable instead. Or is there something I'm not grasping with how .spec files work? I don't *think* defining this from a .spec file really changes that behavior, as long as you set the DEB_BUILD_PATH environment variable only once per build... it should effectively be a fixed value. When I tested this, I added in debian/rules: export DEB_BUILD_PATH=$(CURDIR) And everything *seemed* to work correctly. No matter what directory you're in: -ffile-prefix-map=/path/to/build/dir=. So when calling from directory /path/to/build/dir/A/B/C any embedded paths should get remapped to ./A/B/C, and from directory /path/to/build/dir/A/B/C/.. should get remapped to ./A/B/C/.. I think. This is less complicated that the BUILD_PATH_PREFIX_MAP specifications that Ximin Luo had earlier tried to get accepted into gcc; I do recall that tried to handle more complicated cases... live well, vagrant
The difference being that a missing SOURCE_DATE_EPOCH causes at most
"just" an unreproducible build. While a missing DEB_BUILD_PATH
currently (AFAICS) would break the build. :/ See below.
Ah, misread the function documentation, and I see now that it just
gets appended to the value from the envvar, and is not a fallback
value. Sorry! Although now that I reread this carefully I see that
it's perhaps worse than I thought, as there's this part:
'getenv'
The 'getenv' spec function takes two arguments: an environment
variable name and a string. If the environment variable is
not defined, a fatal error is issued.
That would mean that if the environment variable is not defined, we
fail the build. :(
But that's a problem, because unfortunately debian/rules is still a
supported package building entry point, when using that directly there
is no guarantee that DEB_BUILD_PATH would be set. Which would make the
build fail.
Sure, but that's the thing we cannot guarantee to be happening. :)
So, one option would be to make dpkg-buildflags try to look for the
debian/ directory to try to determine the source tree root dir.
Unfortunately I don't think that'd be reliable, as I could imagine
there being calls from subdirs containing debian/ hierarchies and
similar, even though that might be contrived, but dpkg-buildflags has
never required being called from within a Debian source tree.
(If only we could finally just make dpkg-buildpackage the only
supported entry point…)
Thanks,
Guillem
... Got it. So in order to use this we would need to ensure any used environment variable is *always* set... I see. Ok, a slightly more complicated workflow pseudocode: if DEB_BUILD_PATH && fixfilepath then -spec=/usr/share/dpkg/fixfilepath.spec ... else if fixfilepath -ffile-prefix-map= build_path ... fi In other words, only pass the spec file that requires DEB_BUILD_PATH to be set when DEB_BUILD_PATH *is* set, falling back to the previous behavior setting ffile-prefix-map directly? Another pseudocode idea: if fixfilepath then setenv DEB_BUILD_PATH $build_path export DEB_BUILD_PATH -spec=/usr/share/dpkg/fixfilepath.spec fi live well, vagrant
While in theory something like this might work, I'm afraid this is
still a potentially pretty dangerous thing to do. The problem I see is
that we'd be checking for the presence of an envvar deep in a call
stack that is going to be completely independent from the ones that
are going to be evaluating the spec file with its own envvar fetching,
where something could have set the envvar for the first branch but not
the second (or unset it). But see down below…
I don't think that's workable, as per the above.
There's also the perennial problem with GNU make not exposing variables
exported within a Makefile into $(shell) directives, but I guess in
those scenarios that might work in our favor in most cases.
,--- test.mk ---
export DEB_BUILD_PATH = something
all:
echo $(shell dpkg-buildflags | grep something)
`---
Something I've noticed now is that buildflags.mk is missing passing
DEB_BUILD_PATH explicitly to the dpkg-buildflags call, so I'm merging
the following into git main:
diff --git i/scripts/mk/buildflags.mk w/scripts/mk/buildflags.mk
index 442b7d671..f7ebe8f2c 100644
--- i/scripts/mk/buildflags.mk
+++ w/scripts/mk/buildflags.mk
@@ -31,6 +31,7 @@ endef
$(eval $(call dpkg_buildflags_export_envvar,DEB_BUILD_OPTIONS))
$(eval $(call dpkg_buildflags_export_envvar,DEB_BUILD_MAINT_OPTIONS))
+$(eval $(call dpkg_buildflags_export_envvar,DEB_BUILD_PATH))
$(foreach flag,$(DPKG_BUILDFLAGS_LIST),\
$(foreach operation,SET STRIP APPEND PREPEND,\
$(eval $(call dpkg_buildflags_export_envvar,DEB_$(flag)_MAINT_$(operation)))))
And after checking with codesearch.d.o, I see only Linux is setting
DEB_BUILD_PATH explicitly from debian/*. But that does not guarantee
build machinery is not resetting envvars f.ex. which would break the
build.
Perhaps a soothing check would be to do an archive rebuild using the
first workflow you proposed above (w/ and w/o setting DEB_BUILD_PATH
externally)? Would that be too cumbersome?
If that's successful, and even though that still has the potential to
break stuff (including non-Debian source packages), at that point it
might be OKish to set it from dpkg-buildpackage because even if it is
not set I'd expect things to be more or less fine.
Thanks,
Guillem