Many Lua modules and applications use a simple pattern for internal tests. They have a script (usually called
test.lua) that exercises the module API using API calls and
assert() calls to verify
the results. It is also common practice in those modules to use
print() to output information about the
test progress and to use Lua comments to inform what is being tested to whoever reads the source. Although widespread
this approach may be too crude for those wanting to test more than one module at a time or to have a more detailed view
of the results.
Shake assumes that the tests have been implemented for the above scenario but offers a transparent test engine
for those wanting the execution of the tests in batch mode or those wanting to get an overview of the various results.
The Shake engine can inform not only the number of tests, failures and errors found in a group of test scripts, but it
can infer information about the test context using the output and comments associated with the
The main characteristic of Shake is being transparent, which means that the module authors and test writers do not have
to be aware that tests are going to be run with Shake. As long as the tests call
assert() Shake can obtain
quite a lot of information from the source and run time execution. This is done through the pre-processing of the tests source
code using Leg (and therefore
LPeg) to replace every call to
with one that can extract information about the expressions, values and operators involved in the assertion.
Another common alternative to testing is the one adopted by test frameworks like lunit and luaunit, which offer a quite sophisticated API for those test scripts. For those wanting to use a xUnit style framework these two Lua modules are very useful but in this case the test authors have to be aware that the tests are going to be executed by a xUnit framework.
One of the strong points of the xUnit framework style is being able to report failures in a much more detailed view
than a simple assertion. In Lua, a call to
assert() is basically equivalent to an
displays nice error messages (provided in the
assert() call). Unfortunately, an assertion just checks if an
expression is true or false and report that as an error in the second case.
This makes the test scripts much simpler, but also means that you have to provide decent error messages for every
assertion and that once one assertion fails, your test stops running. Another disvantage of a simple
call is that it is does not provide much information about what caused the error. For example, in an assertion like:
assert(items == 10, "too many items")
If the assertion does not fail then you can be certain that you have 10 items, but if it fails then all you can say is that
items is not the number 10. If you want to know what is the current value of
in that assertion you would have to add it to the error message.
The Shake engine offers an alternative way to obtain that information by preprocessing the test script before it is
executed and constructing a Lua table representation of the test execution. Once the tests are run, we can iterate over
this results table and display a lot more information about the test execution than is usually available. In the above example
Shake would not only be able to inform that the assertion failed, but also the value of
items and even that
the asserted expression involved a non terminal called "items" which was supposed to be equal to the number 10.
Since a results table is usually quite long and have a lot of information that may be useful for some users but not others, Shake offers the separation of a test runner and a test reporter. A test runner is responsible for determining which test scripts are going to be executed and accumulating the results in the runner results table. A test reporter uses the results table and displays information about the results in some way. Shake offers two built in runners, one to be used in the command line and another to be used as a web application. Shake also offers two built in reporters, one that outputs a summary of the results as a text message and another that displays a much more complete report using HTML tables and CSS to format the results.
For those wanting more control on the test execution and reporting, the Shake module (the engine itself) can be used as an standard module and offers an API for test execution and for accessing the results. For more details see the examples and the API reference.
The Shake source distribution contains two components. The first one is a Lua module and command line runner, the second is a CGILua application runner.
The easiest way to install the Shake module and command line runner is using LuaRocks:
luarocks install shake
If you prefer to install Shake manually and you are using Unix then you can install the command line runner
and the Shake module using the command below (where the optional
PREFIX parameter assumes
/usr/local as default):
make PREFIX=prefix_path install
After you have the Shake engine installed you can optionally install the CGILua application using a similar command (note that the CGILua application cannot be installed by using LuaRocks):
make KEPLER_PREFIX=prefix_path install_app
Finally, if you are using Kepler in Windows you would have to copy the Shake source files to your KEPLER_BIN, KEPLER_LUA (and CGILUA_APPS if you want to use the Shake CGILua application) directories as the example below shows:
Kepler Base /apps /cgilua /shake -- from /src/apps/shake /bin shake -- from /src/bin shake.bat -- from /src/bin /lua shake.lua -- from /src/shake /shake stir.lua -- from /src/shake
Once installed Shake can be run as a command line command or as a CGILua web application.
Command line runner
The Shake command line runner is called
shake and uses the following syntax:
shake [options] [filename]
This command runs a test file called filename (assuming
test.lua by default) in the
current directory or any of its subdirectories if the
-r option is given.
The command options are:
- -h, --help
- Prints this help message
- -r, --recursive
- Recursively scans subdirectories for
test.luafiles and runs them
- -v, --version
- Prints the Shake version
Check the examples page for an output of a Shake run.
CGILua application runner
If you have installed the Shake CGILua application in your
CGILUA_APPS directory, Shake assumes
that the tests to be run will be in the
SHAKE_TESTS directory. If you don't define
in your CGILua configuration file, Shake assumes the default of
SHAKE_TESTS directory should contain one directory for each module that you want to test.
When you access the URL
Shake will look into those directories for a
file and execute it if present. After all the test scripts are executed Shake will report a summary of the results for all the tests
This report is an HTML table that also offers a link to a more detailed view to the results of a single module test run. Once you click on that link Shake will show the full results of the test execution, including every assertion involved, even the succesfull ones
For those interested in more control over the runner and the report, Shake offers the following functions (check the examples for more information):
Returns a Shake runner that offers the functions:
Executes a test suite and stores the results in the
Outputs a summary of the results from the
- Returns a boolean value indicating if the expression refers to a terminal symbol.