Hi,

It seems that TESTS_ENVIRONMENT isn't used anymore now that the
default has changed from the serial to the parallel test suite.

It used to generate a Makefile that looked like:
check-TESTS: $(TESTS)
        @failed=0; all=0; xfail=0; xpass=0; skip=0; \
        srcdir=$(srcdir); export srcdir; \
        list=' $(TESTS) '; \
        $(am__tty_colors); \
        if test -n "$$list"; then \
          for tst in $$list; do \
            if test -f ./$$tst; then dir=./; \
            elif test -f $$tst; then dir=; \
            else dir="$(srcdir)/"; fi; \
            if $(TESTS_ENVIRONMENT) $${dir}$$tst; then \
[...]

Now it does:
check-TESTS:
        @list='$(RECHECK_LOGS)';           test -z "$$list" || rm -f $$list
        @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
        @test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
        @set +e; $(am__set_TESTS_bases); \
        log_list=`for i in $$bases; do echo $$i.log; done`; \
        trs_list=`for i in $$bases; do echo $$i.trs; done`; \
        log_list=`echo $$log_list`; trs_list=`echo $$trs_list`; \
        $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list"; \
        exit $$?;

And one of those *.log files then looking like:
run-arextract.sh.log: run-arextract.sh
        @p='run-arextract.sh'; \
        b='run-arextract.sh'; \
        $(am__check_pre) $(LOG_DRIVER) --test-name "$$f" \
        --log-file $$b.log --trs-file $$b.trs \
        $(am__common_driver_flags) $(AM_LOG_DRIVER_FLAGS) $(LOG_DRIVER_FLAGS) -- $(LOG_COMPILE) \
        "$$tst" $(AM_TESTS_FD_REDIRECT)

The Makefile doesn't use TESTS_ENVIRONMENT anymore.

It seems you can work around this issue with the LOG_COMPILER,
but old and current documentation says it should work with
TESTS_ENVIRONMENT.


Kurt

Hello Kurt,

I have just run into related problems with the new automake version, but also
did some more digging:

It seems TESTS_ENVIRONMENT *is* used/available, it's just now part of the
definition of am__check_pre. But if you happen to use it to specify a wrapper
(like I did), this won't work anymore, as is detailed here:

http://www.gnu.org/software/automake/manual/automake.html#Parallel-Test-Harness

I don't know what the symptoms were that you observed (which led to the
diagnosis of TESTS_ENVIRONMENT having been lost), but maybe this helps.

Best,
Michael

Hello Kurt,

I have just run into related problems with the new automake version, but also
did some more digging:

It seems TESTS_ENVIRONMENT *is* used/available, it's just now part of the
definition of am__check_pre. But if you happen to use it to specify a wrapper
(like I did), this won't work anymore, as is detailed here:

http://www.gnu.org/software/automake/manual/automake.html#Parallel-Test-Harness

I don't know what the symptoms were that you observed (which led to the
diagnosis of TESTS_ENVIRONMENT having been lost), but maybe this helps.

Best,
Michael

LD_LIBRARY_PATH.

Looking at the manual in 15.2.1:
| The AM_TESTS_ENVIRONMENT and TESTS_ENVIRONMENT variables can be
| used to run initialization code and set environment variables for
| the test scripts.
[...]
| AM_TESTS_ENVIRONMENT = \
| ## Some environment initializations are kept in a separate shell
| ## file 'tests-env.sh', which can make it easier to also run tests
| ## from the command line.
|  . $(srcdir)/tests-env.sh; \


Then in 15.2.3 it says:
| the AM_TESTS_ENVIRONMENT and TESTS_ENVIRONMENT variables _cannot_ be use
| to define a custom test runner; the LOG_COMPILER and LOG_FLAGS
| (or their extension-specific counterparts) should be used instead:
|
| ## This is WRONG!
| AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
|
| ## Do this instead.
| AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
| LOG_COMPILER = $(PERL)
| AM_LOG_FLAGS = -Mstrict -w

So the manual is contradicting itself.  And if it's not supported
anymore, it's breaking an documented way of doing things.


Kurt

Hi Kurt,

Thanks a lot for elaborating.

[...]

I think this should indeed be ok, *assuming that you are sourcing that script?*
(And put a semicolon at the end of TESTS_ENVIRONMENT.)

I don't quite see the contradiction here? The first example sources a shell
script to suck in variables defined in there; the second example, in the part
not marked *WRONG*, the variables are defined right here?

I am sure I must be missing something here, or is it maybe some bug in the rules
generated by automake that don't actually yield the behaviour as documented?

Best,
Michael

TESTS_ENVIRONMENT = LC_ALL=C LANG=C $(srcdir)/test-wrapper.sh \
                    ../libdw:../backends:../libelf:../libasm

I've tried various thing, and it still gives me the same problems
indicating that test-wrapper.sh was not run.  There are some variable
that aren't set that the script that get run depend on.

Note that the documentation talks about the semicolon is case of
AM_TESTS_ENVIRONMENT, not TESTS_ENVIRONMENT


Kurt

But then you aren't sourcing that script?! I think you should be using

TESTS_ENVIRONMENT = LC_ALL=C; LANG=C; . $(srcdir)/test-wrapper.sh \
                    ../libdw:../backends:../libelf:../libasm

Ah, yes, because they are simply concatenated.

Best,
Michael

I've tried that, it didn't work.


Kurt

Hmm, ok, so the only things left are then the obvious debugging ideas:

1. Is test-wrapper.sh being run at all? To that effect, remove the @ for one or
more of the *.log rules that are now being generated and probably write some
log entries to a file from within test-wrapper.sh.

2. Once confirmed to be running, check for missing export

Otherwise, all I can say is that at least in my package the script had been run
just fine, it's just that its calling conventions would no longer fit the
parallel-harness setup.

Best,
Michael

It seems to be called for each test at least.  Yet the exported
environment variables aren't set.

The problem seems to be that $1 is now ../config/test-driver
which results in the exports not being done.


Kurt

Hi,

AFAICT, there has not been any updates on this RC bug in the past 11 months.

 * Have anyone talked with upstream about the (intended) behaviour?
 * Do we have an idea of the scope / impact of this problem?
   - Is "elfutils" the only package affected?
   - Are affected packages still able to build correctly?
     - Without parallel execution?

~Niels

Note: My interest in this bug is solely that of a Release Team member,
who wants to reduce the number of RC bugs affecting Jessie.