#1002496 rakudo: pre-compiled files are unreproducible

Package:
rakudo
Source:
rakudo
Description:
Perl 6 implementation on top of Moar virtual machine
Submitter:
"Chris Lamb"
Date:
2022-02-20 17:36:03 UTC
Severity:
wishlist
#1002496#5
Date:
2021-12-23 09:19:50 UTC
From:
To:
Hi,

Dear Maintainer,

The latest version of perl6-readline seems to ship a number of
interesting-looking files under /usr/lib/perl6/vendor, such as:

  /usr/lib/perl6/vendor/dist/A8475E6287F45455F9F68569C07ADC25AA5BEFDF

Is this some kind of .pyc equivalent for Perl 6? Either way, I'd love
to know more as these files appear to be unreproducible.


Regards,
#1002496#10
Date:
2021-12-24 14:45:16 UTC
From:
To:
Hi

Yes, theses are Raku modules pre-compiled at build time.

A more detailed explanation is provided on Debian wiki:

https://wiki.debian.org/Perl6PreCompProposal

I don't really know why the pre-compilation is not reproducible.

As a matter of fact, neither rakudo, nqp or moarvm are reproducible. I've
never found the time or energy to find why.

All the best

Dod
#1002496#21
Date:
2022-01-05 12:02:06 UTC
From:
To:
Hey Dominique,
I accidentally-included files at first. Anyway, I've just spent a few
minutes looking into this issue and it seems like the pre-compiled
modules reference the absolute build path:

  $ strings EBC1B2DDA59A9D91D483BD81DAA416714D2D1276 | grep lamby
  /home/lamby/temp/cdt.20220105114153.LbQvXzNPwU.repro.perl6-readline/build-a/perl6-readline-0.1.5

(There might be other 'nondeterminisms' as well, however.)

I assume you're referring to the build of the rakudo source package
itself. If so, that feels like an important yet distinct issue: let's
keep this particular bug to the reproducibility of the *output* of
rakudo… even if it turns out to be the same underlying problem.

Anyway, just thought I'd brain-dump the above for now.


Best wishes,
#1002496#26
Date:
2022-01-20 20:30:06 UTC
From:
To:
[adding 1002496@bugs.debian.org to CC]

Hi all,

Yes, indeed. As I mentioned in that bug, I initially thought they were
accidentally-distributed temporary/build files; something that's
actually quite common in Debian and comes up quite a lot when doing
Reproducible Builds stuff.

If I had realised they were the result of deliberate pre-compilation
efforts, I would probably not have filed that bug. Or, rather: I
wouldn't have done without a patch to fix the issue! In other words,
sorry for the essentially unactionable bug, although it *is* serving as
a useful place to dump information as we inch towards a solution.

(I have included #1002496 on the CC of this thread, perhaps to avoid any
potential duplication in the future.)

Oh, don't read into that description, Vagrant! That's likely my
description based on my jejune understanding of the problem at the
time (see above). Please feel free to update it — I have nothing
against precompilation as a general rule.

Thanks for confirming in reprepro. This is also confirmed by me at the
end of #1002496. I haven't done any other investigating yet.

For completeness, another related bug in this area is:

https://bugs.debian.org/1003159

... although that is related to the contents of foo.dh-raku.list
files. It is, I think, pretty uncontroversial.


Regards,
#1002496#31
Date:
2022-01-20 21:27:33 UTC
From:
To:
Yes... of course, shortly after I sent the mail starting this thread I
found more information on this issue!

I've added links to the bug and wiki page describing perl6
precompilation files in our reproducible builds notes and will think
about how to better describe and/or even rename the issue. :)

I presume "reprotest"? Which I've also used to confirm this issue with a
few packages.


live well,
  vagrant
#1002496#36
Date:
2022-01-21 21:55:59 UTC
From:
To:
Right, I meant reprotest, sorry.


Cheers,
gregor
#1002496#45
Date:
2022-01-27 18:40:14 UTC
From:
To:
Chris Lamb wrote:

Just some further scrappy braindumping here; this is, unfortunately, not a
complete solution yet.

Looking into this further, a lot of this revolves around this method in
src/core.c/CompUnit/Repository/Installation.pm6:

    method id() {
        return $!id if $!id;
        my $name = self.path-spec;
        $name ~= ',' ~ self.next-repo.id if self.next-repo;
        my $dist-dir = $.prefix.add('dist');
        $!id = nqp::sha1(nqp::sha1($name) ~ ($dist-dir.e ?? $dist-dir.dir !! ''))
    }

... and this one in lib/CompUnit/Repository/Staging.rakumod:

    method path-spec(CompUnit::Repository::Staging:D:) {
        self.^name ~ '#name(' ~ $!name ~ ')#' ~ $.prefix.absolute;
    }


Taken together, these basically hash together two things:

  a) All of the repository names/paths used in the precompilation process.

  b) The target directory for the precompilation.

This value is then encoded in the filename of the repo-id file and
similar. (Just to check: is this file even needed?)

Just to take these two things in turn:


## A. Repository paths

It is fine that the hash includes things like /usr/lib/perl6/site.
However, pre-compilation looks in $HOME/.raku for modules and encodes
that value into the hash of the precompiled files. The hash therefore
varies on the build user's home directory location/name.

This occurs via RepositoryRegistry.pm6:

    nqp::bindkey($custom-lib,'home',
      $next-repo := self!register-repository(
        $home-spec,
        CompUnit::Repository::Installation.new(
          :prefix($home),
          :$next-repo
        )
      )
    ) if $home-spec and nqp::not_i(nqp::existskey($unique,$home-spec));

It probably isn't a good idea that Debian package builds inherits anything from
the build user's home directory anyway, so the following should be okay:
#1002496#50
Date:
2022-02-09 18:29:43 UTC
From:
To:
I get a lot of warnings like:

WARNING: unhandled Failure detected in DESTROY. If you meant to ignore it, you can mark it as handled by calling .Bool, .so, .not, or .defined methods. The Failure was:
Failed to create directory '/nonexistent/.raku/short' with mode '0o777': Failed to mkdir: No such file or directory
  in sub MAIN at /usr/bin/prove6 line 3
  in block <unit> at /usr/bin/prove6 line 1

And the build fails with:

===SORRY!=== Error while compiling /usr/lib/perl6/vendor/sources/B4401FC2C8E71132AE0D3CE2C47A7D2FBB0D50F1
Could not find Getopt::Long in:
    inst#/nonexistent/.raku
    inst#/usr/lib/perl6/site
    inst#/usr/lib/perl6/vendor
    inst#/usr/lib/perl6/core
    ap#
    nqp#
    perl5#
at /usr/lib/perl6/vendor/sources/B4401FC2C8E71132AE0D3CE2C47A7D2FBB0D50F1:2
dh_raku_test: error: /usr/bin/prove6 -l -v returned exit code 1

All the best
#1002496#55
Date:
2022-02-09 18:36:16 UTC
From:
To:
Hi Dominique,

Ah, shame. Although I wasn't experiencing a build failure, I was
getting the same or similar warnings when building "perl6-readline".

My gut sense is that this will require a change to Raku itself to not
fail if the home directory does not exist. That would feel like a
desired property anyway.


Regards,
#1002496#60
Date:
2022-02-09 22:45:10 UTC
From:
To:
(just dumping a quick point else I'm sure I'll forget by the time I get to
this mail when on the computer..)

A build process really must not create random files in HOME, and possibly
not even try to access them...

If having a nonexistent home dir (which... Isn't the case of boh default
pbuilder and sbuild?) leads to an ftbfs, that's pretty much a RC bug.
Although I don't remember the reference by heart, I'm quite sure I've seen
such RC bugs filed in the past.
#1002496#65
Date:
2022-02-20 17:33:52 UTC
From:
To:
Sorry, I got confused.

The pre-compilation phase works fine with HOME set to a non-existent directory.

On the other hand, zef tests require a writable HOME directory. I've changed
dh_raku_test to set HOME to debian/tmp/home (and cleanup afterwards).

With these changes, perl6-zef builds without problems.

This does not change the reproducible build issue.

All the best

Dod