Node:Autotest Logs, Previous:testsuite Scripts, Up:Using an Autotest Test Suite

Autotest Logs

When run, the test suite creates a log file named after itself, e.g., a test suite named testsuite creates testsuite.log. It contains a lot of information, usually more than maintainers actually need, but therefore most of the time it contains all that is needed:

command line arguments

A very bad Unix habit which is unfortunately wide spread consists of setting environment variables before the command, such as in CC=my-home-grown-cc ./testsuite. This results in the test suite not knowing this change, hence (i) it can't report it to you, and (ii) it cannot preserve the value of CC for subsequent runs¹. Autoconf faced exactly the same problem, and solved it by asking users to pass the variable definitions as command line arguments. Autotest requires this rule too, but has no means to enforce it; the log then contains a trace of the variables the user changed.

ChangeLog excerpts

The topmost lines of all the ChangeLogs found in the source hierarchy. This is especially useful when bugs are reported against development versions of the package, since the version string does not provide sufficient information to know the exact state of the sources the user compiled. Of course this relies on the use of a ChangeLog.

build machine

Running a test suite in a cross-compile environment is not an easy task, since it would mean having the test suite run on a machine build, while running programs on a machine host. It is much simpler to run both the test suite and the programs on host, but then, from the point of view of the test suite, there remains a single environment, host = build. The log contains relevant information on the state of the build machine, including some important environment variables.

tested programs

The absolute path and answers to --version of the tested programs (see Writing testsuite.at, AT_TESTED).

configuration log

The contents of config.log, as created by configure, are appended. It contains the configuration flags and a detailed report on the configuration itself.

Footnotes

1. When a failure occurs, the test suite is rerun, verbosely, and the user is asked to ``play'' with this failure to provide better information. It is important to keep the same environment between the first run, and bug-tracking runs.