Tests are written using the TestNG framework.
The default run executes all tests in the functional and unit groups. To just run the tests with txt and xml output the command is:
Alternatively, you can execute the tests and generate a report with:
A single test can be executed using the test property. The value is the short name (not the fully qualified package name) of the test. For example:
Alternatively, if there is more than one test with a given classname in your test suite, you could provide the path to the test.
Pattens are also supported:
It is sometimes desirable to install the Infinispan package in your local repository without performing a full test run. To do this, simply use the maven.test.skip.exec property:
Note that you should never use -Dmaven.test.skip=true since modules' test classes depend on other module test classes, and this will cause compilation errors.
If you want to execute tests relying on TestNG's @Parameters from an IDE (such as Eclipse or IntelliJ IDEA), please check this blog entry.
Each test should belong to one or more group. The group acts as a filter, and is used to select which tests are ran as part of the maven test lifecycle.
|If your test does not fit into one of these group, a new group should be added.|
|unit||Unit tests using stubs to isolate and test each major class in Infinispan. This is the default group run if no test group is specified|
|functional||Tests which test the general functionality of Infinispan|
|jgroups||Tests which need to send data on a JGroups Channel|
|transaction||Tests which use a transaction manager|
|profiling||Tests used for manual profiling, not meant for automated test runs|
|manual||Other tests that are run manually|
|Every test (except those not intended to be run by Hudson) should at least be in the functional or unit groups, since these are the default test groups executed by Maven, and are run when preparing a release.|
We use the term permutation to describe a test suite execution against a particular configuration. This allows us to test a variety of environments and configurations without rewriting the same basic test over and over again. For example, if we pass JVM parameter -Dinfinispan.test.jgroups.protocol=udp test suite is executed using UDP config.
Each permutation uses its own report directory, and its own html output file name. This allows you to execute multiple permutations without wiping the results from the previous run. Note that due to the way Maven operates, only one permutation can be executed per mvn command. So automating multiple runs requires shell scripting, or some other execution framework to make multiple calls to Maven.
Sometimes you want to run a test using settings other than the defaults (such as UDP for jgroups group tests or the DummyTransactionManager for transaction group tests). This can be achieved by referring to the Maven POM file to figure out which system properties are passed in to the test when doing something different. For example to run a jgroups group test in your IDE using TCP instead of the default UDP, set -Dinfinispan.test.jgroups.protocol=tcp. Or, to use JBoss JTA (Arjuna TM) instead of the DummyTransactionManager in a transaction group test, set -Dinfinispan.test.jta.tm=jbosstm Please refer to the POM file for more properties and permutations.
Infinispan runs its unit test suite in parallel; Infinispan tests are often IO rather than processor bound, so executing them in parallel offers significant speed up.
There are a number of constraints and best practices that need to be followed in order to ensure correctness and keep the execution time to a minimum. If you follow these guidelines you will find your tests are more reliable:
Each test class is run in a single thread
There is no need to synchronize unit test's fixture, as test methods will be run in sequence. However, multiple test classes are executed in parallel.
Each test class must have an unique test name
As a convention, the name of the test should be the FQN of the test class with the org.infinispan prefix removed. For example, given a test class org.infinispan.mypackage.MyTest the name of the test should be mypackage.MyTest. This convention guarantees a unique name.
Use TestCacheManagerFactory.createXyzCacheManager and _never_create managers using new DefaultCacheManager()
This ensures that there are no conflicts on resources e.g. a cluster created by one test won't interfere with a cluster created by another test.
Where possible, extend SingleCacheManagerTestorMultipleCacheManagersTest
Tests inheriting from these template method classes will only create a cache/cluster once for all the test methods, rather than before each method. This helps keep the execution time down.
Never rely onThread.sleep()
When running in heavily threaded environments this will most often not work. For example, if using ASYNC_REPL, don't use a sleep(someValue) and expect the data will be replicated to another cache instance after this delay has elpased. Instead, use a ReplicationListener (check the javadoc for more information). Generally speaking, if you expect something will happen and you don't have a guarantee when, a good approach is to try that expectation in a loop, several times, with an generous (5-10secs) timeout. For example:
Thread.sleep(10) may not work in certain OS/JRE combos (e.g. Windows XP/Sun JRE 1.5)
Use values grater than 10 for these statements, e.g. 50. Otherwise, a System.currentTimeMillis() might return same value when called before and after such a sleep statement.
For each cache that is create with TestCacheManagerFactory.createXyzCacheManager() the test harness will allocate a unique JMX domain name which can be obtained through CacheManager.getJmxDomain(). This ensures that no JMX collisions will takes place between any tests executed in parallel. If you want to enforce a JMX domain name, this can be done by using one of the TestCacheManagerFactory.createCacheManagerEnforceJmxDomain methods. These methods must be used with care, and you are responsible for ensuring no domain name collisions happen when the parallel suite is executed.
Use obscure words
Insert uncommon or obscure words into the cache that has been generated with a random word generator. In a multi-threaded environment like Infinispan's testsuite, grepping for these words can greatly help the debugging process. You may find this random word generator useful.
Use the test method name as the key
Grab the test method and use it as part of the cached key. You can dynamically grab the test method using code like this:
Even though we've tried to reduce them to a min, intermittent failures might still appear from time to time. If you see such failures in existing code please report them in the issue tracker.