
Directory layout
~~~~~~~~~~~~~~~~

This directory contains the test suite.  Each test lives in its own
subdirectory of this directory.  Each test is implemented by a script
named "$TESTNAME/dotest".


Exclude files
~~~~~~~~~~~~~~~~

Exclude files are hidden files that allow for tests to be disabled based
on certain conditions that may cause the test to not run properly. If an
exclude file exists, it should have a short message about why the test has
been excluded.

Exclude files can exist in any of the following formats:

.exclude_OS-TYPE    : Disables a test for an operating system that it does
                      not run successfully on. Acceptable values to fill in
                      for OS-TYPE are "windows", "linux", "mac", and "solaris"
.exclude_BP-VERSION : Disables a test for a version of the bundle protocol
                      that it does not run correctly or does not make sense
                      with. As of ION 4.0.0, the acceptable values to fill in
                      for BP-VERSION are "bpv6" and "bpv7".
.exclude_all        : Disables a test for all platforms.
.exclude_expert     : Disables a test because of additional utilities that are
                      required for the test. To work around this exclusion if
                      you want to run an expert test, you can set
                      ION_RUN_EXPERT="yes" in your shell environment to
                      enable all ION tests classified as expert.
.exclude_cbased     : Disables a test that relies on compiling a C program to
                      generate the dotest executable script. If you want to
                      exclude C based tests, you need to define the environment
                      variable ION_EXCLUDE_CBASED.


Optional tests
~~~~~~~~~~~~~~~~

Tests can be marked as optional by adding a ".optional" file to the test
directory. Optional tests are run by default (unless explicitly excluded via
an exclude file), but their results do not affect the overall pass/fail status
of the test campaign.

To mark a test as optional, create a file named ".optional" in the test
directory:

    touch tests/my-test/.optional

Optional tests are useful for:
- Experimental or unstable tests that are not yet ready for CI enforcement
- Performance tests that may be sensitive to system load
- Tests that depend on external resources that may be temporarily unavailable
- Long-running tests that provide valuable information but should not block
  the test suite

Key behaviors of optional tests:
- They are run during normal test execution
- They are clearly marked as "OPTIONAL TEST" in the output
- Their pass/fail status is reported in a separate "Optional Tests" section
- Failed optional tests do NOT cause the overall test run to fail
- Failed optional tests are NOT included in the retest file
- Optional tests are NOT retested in the automatic retest phase


Running the tests
~~~~~~~~~~~~~~~~~

The tests are run by running "make test-all" in the top-level directory,
or by running "runtests" in this directory.

An individual test can also be run:

    ./runtests test_name

A file defining a set of tests can be run with runtestset.  The arguments
to runtestset are files that contain globs of tests to run:

    ./runtestset quicktests


Writing new tests
~~~~~~~~~~~~~~~~~

A test directory must contain an executable file named "dotest".  If a
directory does not contain this, then the test directory is ignored.

The dotest program should execute the test, possibly reporting runtime
information on stdout and stderr, and indicate by its return value the
status of the test:

    0: Success

    1: Failure

    2: Skip this test

The test program starts without the ION stack running.  The test program
is responsible for starting the ION stack in the way it wants to test.
The test program *must* stop the ION stack before returning.


The test environment
~~~~~~~~~~~~~~~~~~~~

The tests' dotest scripts are run in their test directory.

The suite runs the installed ION, so run "make install" before running
it.  runtests refuses to start if ionadmin is not on PATH.  The
"./runtests cleanup" mode is exempt.

runtests exports two variables that point into the source tree, for
tests that reach source-tree resources (configuration files, link
graphs) through them:

    IONDIR is the root of the local ION source directory

    CONFIGSROOT is IONDIR/configs

Before the first test runs, runtests reports the resolved environment to
both stderr and the progress file, so a completed run records which ION
it exercised.
