Automated Testing

As OpenSim matures, we are extremely interested in adding more automated verification into the OpenSim source tree. Test exist not only to prevent bugs from creeping in, but also to provide fair warning that the behavior of the system has changed, and tests may need updating.

In OpenSim today we use NUnit tests. Our conventions are:
 * 1) Tests should not exist inside runtime assemblies, as this makes nunit a production requirement
 * 2) Tests should be in .Tests.dll assemblies.  For instance, the tests for OpenSim.Data.SQLite.dll should be in the OpenSim.Data.SQLite.Tests.dll assembly.  This allows for easy removal of test assemblies in products.
 * 3) Tests should be as close to the code as possible, but not intermingled.  So the tests for OpenSim/Data/SQLite should be in OpenSim/Data/SQLite/Tests/.  Through the use of the Exclude keyword in prebuild.xml you can ensure that directory is part of OpenSim.Data.SQLite.Tests.dll and not OpenSim.Data.SQLite.dll.
 * 4) Tests should be able to run safely in a production environment.  That means that care must be taken not to damage data on the machine that it is being run.

= Writing New Tests = Writing a new unit test is pretty easy, and very helpful in increasing the stability of opensim by nailing down bugs. I'm going to present an example here of SQLite Asset testing to show how simple such a test case is to write. The actual in tree SQLite Asset tests are a little different because the code was factored out so that it was easily applied to any database driver, so don't be concerned with the fact that what you see here isn't in the tree.

NUnit Conventions
An NUnit test suite:
 * is a class with a default constructor (takes no arguments)
 * has public methods that are tests
 * uses annotations to determine what are tests
 * runs it's tests in alphabetical order by method name

An NUnit test method:
 * must be public
 * must return void
 * must take no arguments
 * is successful if no exception or assert is thrown while running it

The run order is important if you want to have early tests that setup some complicated state (like creating objects), and have later tests remove or update that state. For that reason I find it very helpful to name all test methods Txxx_somename where xxx is a number between 000 and 999. That guaruntees no surprises in run order.

Tests should never rely on the result of another test. If you need the environment set up in a certain way, you can use the SetUp and TearDown method attributes. Only one SetUp method can exist; if more than one is defined, the assembly will compile successfully, but its tests will not run. The SetUp method is run once before every test case, and the TearDown method is run once after each test case, regardless of whether the test passed or failed.

An Example Test - SQLite Assets
You can see 4 of the important annotations here:
 * TestFixture - this class is a test suite
 * TestFixtureSetup - this code is always run before any of the tests are executed
 * TestFixtureTearDown - this code is always run after the tests are done executing, even if they fail.
 * Test - this method is a test

Setup / Teardown
In the case of testing something like the database layer, we have to actually attempt to store / retrieve things from a database. Following from rule #4 of good tests, we want to make sure not to touch the production databases to run our tests, so during startup we generate a temporary file name which is guaranteed not to be an existing file on the system, and use that as our database file name. By running db.Initialize the OpenSim migration code will correctly populate that database with the latest schema.

Once we are done with the tests we want to make sure we aren't leaving garbage temp files on the user's system. So we remove that file we created.

During setup we also create a set of state variables, such as 3 uuids, 3 strings, and a data block. You could have always just stuck these inline, but variables are there for a reason, so use them.

Asserts
You will see scattered through the code Assert.That(...). These will throw an exception if the condition is not valid. This format of assertions is called the |Contraint Model in NUnit, and provides a large number of tests with the flavor of:
 * Assert.That(foo, Is.Null)
 * Assert.That(foo, Is.Not.Null)
 * Assert.That(foo, Is.True)
 * Assert.That(foo, Is.EqualTo(bar))
 * Assert.That(foo, Text.Matches( "*bar*" ))

Of note, Is.EqualTo uses the Equals function of foo, so this can only be used on objects that are IComparable. Most of the OpenSim base objects are not, so you'll have to compare fields manually in tests.

For the complete set of conditions you can use see |the Contraint Model NUnit documentation. While there is another syntax for tests, the Constraint Model is preferred as it is far more human readable.

Simple Negative Tests
Test T001 is an example of a simple negative test. We assume a new database will not have any of those assets in them. While the value of this test may look low, it does provide a baseline in ensuring that the database connection is there, that these return false correctly, and that no other exception is thrown. Negative tests are a good way to force bounds conditions and ensure that not only does it return what you expect it also doesn't return what you don't expect. Thought of another way, it ensures your code is somewhat defensive in nature, not coughing on bad or unexpected data.

Simple Positive Tests
T010 is an example of a simple positive test. In it we create and store 3 assets (ensuring no exceptions), then load those 3 assets back from the database and ensure the fields are correct. Because AssetBase is not IComparible we just check the ID and Name fields with equals tests. If any of the Asserts fail, the whole test fails.

Stateful Tests
T011 is an example of a stateful test, because it requires the state created by T010 (i.e. the creation of those 3 objects). In order to test any kind of complicated scenario you will find that you need to use stateful tests to build up various amounts of state (testing along the way), then manipulating and possibly tearing it down. Without doing this you can't do truly deep testing of function in any complex environment. This example isn't very stateful (I tried to pick an easy example), but it should give you some ideas.

Speculative Tests
Speculative tests are tests that might or might not apply in a given situation. MySQL testing in the OpenSim tree is done by speculative testing, the tests will only run if there is a properly configured database, otherwise they will not be run. If you execute Assert.Ignore in a Test the test will end and be ignored. If you run Assert.Ignore in the TestFixtureSetup all tests in the test fixture will be skipped and ignored.

Speculative testing lets you create tests that require certain preconditions to be met which you can't guarantee on all platforms/configuration, and are an important part of deep testing.

Good / Bad Test practices
Creating good tests is an art, not a science. Tests are useful by how many bugs they find or how many bugs they avoid. Things you should think about in creating good tests is:
 * throwing edge cases, like 0, "", or Null at parameters. This ensures that people functions are hardened against incomplete data.  Many of our crashes come from the lack of this hardening showing up at just the wrong time.
 * use stateful testing to build up complex scenarios. This is more useful than just cursory get / set calls.

Adding Tests to the Tree
As we said previously all tests for assembly OpenSim.Foo (in directory OpenSim/Foo) should:
 * be in assembly OpenSim.Foo.Tests.dll
 * not be in the OpenSim.Foo.dll assembly
 * be in OpenSim/Foo/Tests directory

Also, if you have created a new test assembly you must add references to it in both .nant/local.include and .nant/bamboo.build to ensure that the assembly is added to the automated bamboo runs as well as the nant test target.

= Executing Tests =

Bamboo
On every commit to opensim all the tests are run on a |Bamboo build server on opensimulator.org. The process takes about 5 minutes to build, test, and report the results back out on #opensim-dev via the osmantis bot.

Nant
You can manually run all the tests for OpenSim on your system by running nant test as a nant target. This will run all the tests that are in the tree, though some speculative tests might be ignored if your platform does not have the right features or configuration to run these tests.

NUnit Console
If you only want to run tests for one assembly you can do that using the NUnit Console. On Linux just run nunit-console2 OpenSim.Foo.Tests.dll and it will run only the tests for OpenSim.Foo.Tests.dll. If you are only making changes to 1 dll and just want a quick sanity check, this is the fastest way to do that.

= Learning More = You should definitely read the documentation at the |NUnit homepage if you want to know more about testing. It's a very good reference for all the APIs in NUnit that you can use for creating tests.

= Testing Todo =

Coverage
At some point in the future we need to integrate test coverage into the bamboo reports so we can easily see how poorly covered our code is, and look for hotspots where we should focus. We're still at such an early stage that this is probably overkill to get setup for now.