Automated Testing

As OpenSim matures, we are extremely interested in adding more automated verification into the OpenSim source tree. Test exist not only to prevent bugs from creeping in, but also to provide fair warning that the behavior of the system has changed, and tests may need updating.

In OpenSim today we use NUnit tests. Our conventions are:

1. Tests should not exist inside runtime assemblies, as this makes nunit a production requirement
2. Tests should be in .Tests.dll assemblies. For instance, the tests for OpenSim.Data.SQLite.dll should be in the OpenSim.Data.SQLite.Tests.dll assembly. This allows for easy removal of test assemblies in products.
3. Tests should be as close to the code as possible, but not intermingled. So the tests for OpenSim/Data/SQLite should be in OpenSim/Data/SQLite/Tests/. Through the use of the Exclude keyword in prebuild.xml you can ensure that directory is part of OpenSim.Data.SQLite.Tests.dll and not OpenSim.Data.SQLite.dll. See exclude clarification in writing unit tests section.
4. Tests testing a class should be grouped into a test class file called xxxTest.cs, where xxx is the name of the class that is being tested.
5. Tests should be able to run safely in a production environment. That means that care must be taken not to damage data on the machine that it is being run.
6. Tests should be deterministic in other words repeatable. Avoid randomness in tests. See good and bad testing practices below.

Core Functionality Missing Unit Tests

This is a list of functionality which is not covered by unit tests and is identified as highly desireable test target:

1. Database Modules (These are mysql tables)
   1. region ban
   2. land
   3. landaccesslist

Good / Bad Test practices

Creating good tests is an art, not a science. Tests are useful by how many bugs they find or how many bugs they avoid. Things you should think about in creating good tests is:

• Throwing edge cases, like 0, "", or Null at parameters. This ensures that people functions are hardened against incomplete data. Many of our crashes come from the lack of this hardening showing up at just the wrong time.
• Use stateful testing to build up complex scenarios. This is more useful than just cursory get / set calls.
• Random tests are not a good idea. We need test results to be deterministic. In other words tests need to be repeatable. If you want to test for a range it is good idea to make separate tests for min and max values. Random values in fields can fail randomly. When something goes wrong for example in database schema the developer will not necessarily notice if the stored values are random. On the other hand its hard to troubleshoot randomly failing tests as you dont know which specific value caused the failure.
• Tests should be independent and should not rely on another test being run, passing or failing. An excerpt from xUnit Patterns:

If tests are interdependent and (even worse) order dependent, we will be depriving ourselves of the useful feedback test failures provide. Interacting Tests [...] tend to fail in a group. The failure of a test that moved the [subject under test] into the state required by the dependent test will lead to the failure of the dependent test too. With both tests failing, how can we tell if it is because of a problem in code that both rely on in some way or is it a problem in code that only the first relies on. With both tests failing we can't tell. We are only talking about two tests here. Imagine how much worse this is with tens or hundreds of tests.

• Only one function of the subject under test should be tested in one test. When testing a database access object, for example, write separate tests for creating DB entries, updating them and removing them.
• Do not use the subject under test to set up the state for the test or to verify the result. Use a different method. When testing a database access object, for example, use raw SQL to insert the initial data into the DB, then run the method being tested. To verify if the operation was successful, use raw SQL again to verify the DB changed as expected.

Writing Tests

Writing a new unit test is pretty easy, and very helpful in increasing the stability of opensim by nailing down bugs. I'm going to present an example here of SQLite Asset testing to show how simple such a test case is to write. The actual in tree SQLite Asset tests are a little different because the code was factored out so that it was easily applied to any database driver, so don't be concerned with the fact that what you see here isn't in the tree.

Exclude clarification: Make sure your master project (not the test project) has an entry for files like the following so that the test code is not included in the master project dll:

      <Files>
        <Match pattern="*.cs" recurse="true">
          <Exclude name="Tests" pattern="Tests" />
        </Match>
      </Files>

NUnit Conventions

An NUnit test suite:

• is a class with a default constructor (takes no arguments)
• has public methods that are tests
• uses annotations to determine what are tests
• runs it's tests in alphabetical order by method name

An NUnit test method:

• must be public
• must return void
• must take no arguments
• is successful if no exception or assert is thrown while running it

The run order is important if you want to have early tests that setup some complicated state (like creating objects), and have later tests remove or update that state. For that reason I find it very helpful to name all test methods Txxx_somename where xxx is a number between 000 and 999. That guaruntees no surprises in run order.

Setup / Teardown

In the case of testing something like the database layer, we have to actually attempt to store / retrieve things from a database. Following from rule #4 of good tests, we want to make sure not to touch the production databases to run our tests, so during startup we generate a temporary file name which is guaranteed not to be an existing file on the system, and use that as our database file name. By running db.Initialize() the OpenSim migration code will correctly populate that database with the latest schema.

Once we are done with the tests we want to make sure we aren't leaving garbage temp files on the user's system. So we remove that file we created.

During setup we also create a set of state variables, such as 3 uuids, 3 strings, and a data block. You could have always just stuck these inline, but variables are there for a reason, so use them.

Asserts

You will see scattered through the code Assert.That(...). These will throw an exception if the condition is not valid. This format of assertions is called the Contraint Model in NUnit, and provides a large number of tests with the flavor of:

• Assert.That(foo, Is.Null)
• Assert.That(foo, Is.Not.Null)
• Assert.That(foo, Is.True)
• Assert.That(foo, Is.EqualTo(bar))
• Assert.That(foo, Text.Matches( "*bar*" ))

Of note, Is.EqualTo uses the Equals function of foo, so this can only be used on objects that are IComparable. Most of the OpenSim base objects are not, so you'll have to compare fields manually in tests.

For the complete set of conditions you can use see the Contraint Model NUnit documentation. While there is another syntax for tests, the Constraint Model is preferred as it is far more human readable.

Simple Negative Tests

See Example Test SQLite Assets for this code snipped in context.

[Test]
public void T001_LoadEmpty()
{
    Assert.That(db.ExistsAsset(uuid1), Is.False);
    Assert.That(db.ExistsAsset(uuid2), Is.False);
    Assert.That(db.ExistsAsset(uuid3), Is.False);
}

Test T001 is an example of a simple negative test. We assume a new database will not have any of those assets in them. While the value of this test may look low, it does provide a baseline in ensuring that the database connection is there, that these return false correctly, and that no other exception is thrown. Negative tests are a good way to force bounds conditions and ensure that not only does it return what you expect it also doesn't return what you don't expect. Thought of another way, it ensures your code is somewhat defensive in nature, not coughing on bad or unexpected data.

Simple Positive Tests

See Example Test SQLite Assets for this code snipped in context.

[Test]
public void T010_StoreSimpleAsset()
{
    AssetBase a1 = new AssetBase(uuid1, name1);
    AssetBase a2 = new AssetBase(uuid2, name2);
    AssetBase a3 = new AssetBase(uuid3, name3);
    a1.Data = asset1;
    a2.Data = asset1;
    a3.Data = asset1;
 
    db.CreateAsset(a1);
    db.CreateAsset(a2);
    db.CreateAsset(a3);
 
    AssetBase a1a = db.FetchAsset(uuid1);
    Assert.That(a1a.ID, Is.EqualTo(uuid1));
    Assert.That(a1a.Name, Is.EqualTo(name1));
 
    AssetBase a2a = db.FetchAsset(uuid2);
    Assert.That(a2a.ID, Is.EqualTo(uuid2));
    Assert.That(a2a.Name, Is.EqualTo(name2));
 
    AssetBase a3a = db.FetchAsset(uuid3);
    Assert.That(a3a.ID, Is.EqualTo(uuid3));
    Assert.That(a3a.Name, Is.EqualTo(name3));
}

T010 is an example of a simple positive test. In it we create and store 3 assets (ensuring no exceptions), then load those 3 assets back from the database and ensure the fields are correct. Because AssetBase is not IComparible we just check the ID and Name fields with equals tests. If any of the Asserts fail, the whole test fails.

Stateful Tests

See Example Test SQLite Assets for this code snipped in context.

[Test]
public void T011_ExistsSimpleAsset()
{
    Assert.That(db.ExistsAsset(uuid1), Is.True);
    Assert.That(db.ExistsAsset(uuid2), Is.True);
    Assert.That(db.ExistsAsset(uuid3), Is.True);
}

T011 is an example of a stateful test, because it requires the state created by T010 (i.e. the creation of those 3 objects). In order to test any kind of complicated scenario you will find that you need to use stateful tests to build up various amounts of state (testing along the way), then manipulating and possibly tearing it down. Without doing this you can't do truly deep testing of function in any complex environment. This example isn't very stateful (I tried to pick an easy example), but it should give you some ideas.

Speculative Tests

Speculative tests are tests that might or might not apply in a given situation. MySQL testing in the OpenSim tree is done by speculative testing, the tests will only run if there is a properly configured database, otherwise they will not be run. If you execute Assert.Ignore() in a Test the test will end and be ignored. If you run Assert.Ignore() in the TestFixtureSetup all tests in the test fixture will be skipped and ignored.

Speculative testing lets you create tests that require certain preconditions to be met which you can't guarantee on all platforms/configuration, and are an important part of deep testing.

Adding Tests to the Tree

As we said previously all tests for assembly OpenSim.Foo (in directory OpenSim/Foo) should:

• be in assembly OpenSim.Foo.Tests.dll
• not be in the OpenSim.Foo.dll assembly
• be in OpenSim/Foo/Tests directory

Also, if you have created a new test assembly you must add references to it in both .nant/local.include and .nant/bamboo.build to ensure that the assembly is added to the automated bamboo runs as well as the nant test target.

Executing Tests

Bamboo

On every commit to opensim all the tests are run on a Bamboo build server on opensimulator.org. The process takes about 5 minutes to build, test, and report the results back out on #opensim-dev via the osmantis bot.

Nant

You can manually run all the tests for OpenSim on your system by running nant test as a nant target. This will run all the tests that are in the tree, though some speculative tests might be ignored if your platform does not have the right features or configuration to run these tests.

NUnit Console

If you only want to run tests for one assembly you can do that using the NUnit Console. On Linux just run nunit-console2 OpenSim.Foo.Tests.dll and it will run only the tests for OpenSim.Foo.Tests.dll. If you are only making changes to 1 dll and just want a quick sanity check, this is the fastest way to do that.

Debugging Tests

There is a special page dedicated to this. See Debugging Unit Tests.

Learning More

You should definitely read the documentation at the NUnit homepage if you want to know more about testing. It's a very good reference for all the APIs in NUnit that you can use for creating tests.

Code Coverage

A prototype has been included using monocov, which has a profile built-in in mono. Instructions for using monocov can be found on the Mono homepage, in Code Coverage section. Unfortunately nunit2 and nant do not support code coverage with monocov (only some other proprietary code coverages), so there is no <monocov> tag. The solution was to implement it using many nunit-console and on <exec> tags.

ATTENTION: Code coverage only works with mono 1.2.x , any other version will most likely not work. Code coverage development is being put on hold for now until it supports newer mono versions.

Running

Download monocov and nunit-console if your nunit-console does not work. To test if it works (my Ubuntu's Hardy version did not), just run one of the tests with nunit-console.

Install monocov (./configure, ./make install, there could be some minor conflicts, I had to add to the compiling line -I/usr/include/mono-1.0/) and make sure the working nunit-console.exe is in /usr/lib/nunit/nunit-console.exe. Now just run nant test-cov and it will generate .cov files and HTML directories in the cov directory. The .cov files can be seen by running monocov on them, and have the same information as the HTML directories.

Testing Todo

Coverage

A prototype has been done and documented. Now we must keep a look if monocov will evolve to support newer mono versions.