Automated Testing

Revision as of 22:14, 3 March 2012

 

Home

 

Download

 

News

 

Support

 

Admin

 

Dev

 

Screen Shots

 

Grid List

 

Bugs

Languages:

English    Deutsch

As OpenSimulator matures, we are extremely interested in adding more automated verification into the OpenSimulator source tree. Testing exists not only to prevent bugs from creeping in, but also to provide fair warning that the behavior of the system has changed, and tests may need updating.

In OpenSimulator today we use NUnit tests. Our conventions are:

1. Tests should not exist inside runtime assemblies, as this makes nunit a production requirement
2. Tests should be in .Tests.dll assemblies. For instance, the tests for OpenSim.Data.SQLite.dll should be in the OpenSim.Data.SQLite.Tests.dll assembly. This allows for easy removal of test assemblies in products.
3. Tests should be as close to the code as possible, but not intermingled. So the tests for OpenSim/Data/SQLite should be in OpenSim/Data/SQLite/Tests/. Through the use of the Exclude keyword in prebuild.xml you can ensure that directory is part of OpenSim.Data.SQLite.Tests.dll and not OpenSim.Data.SQLite.dll. See exclude clarification in writing unit tests section.
4. Tests testing a class should be grouped into a test class file called xxxTest.cs, where xxx is the name of the class that is being tested.
5. Tests should be able to run safely in a production environment. That means that care must be taken not to damage data on the machine that it is being run.
6. Tests should be deterministic in other words repeatable. Avoid randomness in tests. See good and bad testing practices below.

Core Functionality Missing Unit Tests

This is a list of functionality which is not covered by unit tests and is identified as highly desireable test target:

1. Database Modules (These are mysql tables)
   1. region ban
   2. land
   3. landaccesslist

Good / Bad Test practices

Creating good tests is an art, not a science. Tests are useful by how many bugs they find or how many bugs they avoid. Things you should think about in creating good tests is:

• Throwing edge cases, like 0, "", or Null at parameters. This ensures that people functions are hardened against incomplete data. Many of our crashes come from the lack of this hardening showing up at just the wrong time.
• Use stateful testing to build up complex scenarios. This is more useful than just cursory get / set calls.
• Random tests are not a good idea. We need test results to be deterministic. In other words tests need to be repeatable. If you want to test for a range it is good idea to make separate tests for min and max values. Random values in fields can fail randomly. When something goes wrong for example in database schema the developer will not necessarily notice if the stored values are random. On the other hand its hard to troubleshoot randomly failing tests as you dont know which specific value caused the failure.
• Tests should be independent and should not rely on another test being run, passing or failing. An excerpt from xUnit Patterns:

If tests are interdependent and (even worse) order dependent, we will be depriving ourselves of the useful feedback test failures provide. Interacting Tests [...] tend to fail in a group. The failure of a test that moved the [subject under test] into the state required by the dependent test will lead to the failure of the dependent test too. With both tests failing, how can we tell if it is because of a problem in code that both rely on in some way or is it a problem in code that only the first relies on. With both tests failing we can't tell. We are only talking about two tests here. Imagine how much worse this is with tens or hundreds of tests.

• Only one function of the subject under test should be tested in one test. When testing a database access object, for example, write separate tests for creating DB entries, updating them and removing them.
• Do not use the subject under test to set up the state for the test or to verify the result. Use a different method. When testing a database access object, for example, use raw SQL to insert the initial data into the DB, then run the method being tested. To verify if the operation was successful, use raw SQL again to verify the DB changed as expected.
• Use descriptive asserts whenever you can. All you have to do is add an extra , to the Assert() method and write a string that will show when that test fails. For example:

 Assert.That(i,Is.EqualTo(5),"i is not equal to 5! in Example.Test1()");

• When a test fails due to a uncaught exception, such as NullReference, nUnit does not report where it happened, leaving debuggers clueless. A good practive is to write something on the start of every test in your test file. This way if an exception is raised, someone could read the last lines written and see at least in what test it failed. Luckily, this routine is already implemented in OpenSim.Tests.Common.TestHelper InMethod().

Writing Tests

See NUnit Quick Start for an introduction to unit testing with NUnit.

Writing a new unit test is pretty easy, and very helpful in increasing the stability of opensim by nailing down bugs. I'm going to present an example here of SQLite Asset testing to show how simple such a test case is to write. The actual in tree SQLite Asset tests are a little different because the code was factored out so that it was easily applied to any database driver, so don't be concerned with the fact that what you see here isn't in the tree.

Exclude clarification: Make sure your master project (not the test project) has an entry for files like the following so that the test code is not included in the master project dll:

      <Files>
        <Match pattern="*.cs" recurse="true">
          <Exclude name="Tests" pattern="Tests" />
        </Match>
      </Files>

NUnit Conventions

An NUnit test suite:

• is a class with a default constructor (takes no arguments)
• has public methods that are tests
• uses annotations to determine what are tests
• runs it's tests in alphabetical order by method name

An NUnit test method:

• must be public
• must return void
• must take no arguments
• is successful if no exception or assert is thrown while running it

Once the tests are moved to NUnit 2.5+:

• a test class can be generic and can have one or more constructors with parameters. in this case, one or more [TestFixture(...)] attributes must be used to provide the types for the generic arguments and values for the constructors. This means it is possible to create a single test class which would, for example, provide tests for different database engines.
• each test method may have parameters supplied by attributes such as [TestCase(...)] or [Values]. The test will run automatically for various combinations of these attributes.
• NUnit no longer guarantees that the tests will be performed in any particular order.

The run order is important if you want to have early tests that setup some complicated state (like creating objects), and have later tests remove or update that state. For that reason I find it very helpful to name all test methods Txxx_somename where xxx is a number between 000 and 999. That guarantees no surprises in run order.

Fixture Setup / Teardown

See Example Test SQLite Assets for this code snipped in context.

[TestFixtureSetUp]
public void Init()
{
    uuid1 = UUID.Random();
    uuid2 = UUID.Random();
    uuid3 = UUID.Random();
    name1 = "asset one";
    name2 = "asset two";
    name3 = "asset three";
 
    asset1 = new byte[100];
    asset1.Initialize();
    file = Path.GetTempFileName() + ".db";
    connect = "URI=file:" + file + ",version=3";
    db = new SQLiteAssetData();
    db.Initialise(connect);
}
 
[TestFixtureTearDown]
public void Cleanup()
{
    db.Dispose();
    System.IO.File.Delete(file);
}

In the case of testing something like the database layer, we have to actually attempt to store / retrieve things from a database. Following from rule #4 of good tests, we want to make sure not to touch the production databases to run our tests, so during startup we generate a temporary file name which is guaranteed not to be an existing file on the system, and use that as our database file name. By running db.Initialize() the OpenSimulator migration code will correctly populate that database with the latest schema.

Once we are done with the tests we want to make sure we aren't leaving garbage temp files on the user's system. So we remove that file we created.

During setup we also create a set of state variables, such as 3 uuids, 3 strings, and a data block. You could have always just stuck these inline, but variables are there for a reason, so use them.

Test Setup / Teardown

What's missing in Example Test SQLite Assets are individual test Setup and Teardown methods. These methods allow each test to be completely self sufficient without the code duplication needed to set up the test environment at the start of each test.

Let's assume the SQLiteAssetData class provided a FetchAsset() method and a UpdateAsset() method. Since every test should be independent of any other test, and FetchAsset() and UpdateAsset() should be tested in separate tests, that means each test would need to create its own entries in the asset table in order to succeed. You may have something like this (see Example Test SQLite Assets for an explanation of sqldb.executeSQL()):

[Test]
public void T101_TestFetchAsset()
{
    AssetBase a1 = new AssetBase(...);
    sqldb.executeSQL("INSERT INTO assets VALUES({0}, ...)", a1.uuid, ...);
 
    AssetBase a1_actual = db.FetchAsset(a1.uuid);
 
    Assert.Equal(a1_actual.uuid, a1.uuid);
    Assert.Equal(a1_actual.Name, a1.Name);
    // etc
}
 
[Test]
public void T103_TestUpdateAsset()
{
    AssetBase a1 = new AssetBase(...);
    sqldb.executeSQL("INSERT INTO assets VALUES({0}, ...)", a1.uuid, ...);
 
    a1.Name = "new name";
 
    db.UpdateAsset(a1.uuid, a1);
 
    AssetBase a1_actual = sqldb.executeSQL("SELECT * FROM assets WHERE uuid = {0}", a1.uuid);
 
    Assert.Equal(a1_actual.uuid, a1.uuid);
    Assert.Equal(a1_actual.Name, a1.Name);
    // etc
}

You will note that both tests have the same code at the top in which they create an entry in the assets table. This duplicate code can be factored out into a Setup method, which is called before every test is executed (assume a1 is a class attribute):

[SetUp]
public void SetUp()
{
    a1 = new AssetBase(...);
    sqldb.executeSQL("INSERT INTO assets VALUES({0}, ...)", a1.uuid, ...);
}
 
[TearDown]
public void TearDown()
{
    // clean up after ourselves so the next test has a clean DB to start with
    sqldb.executeSQL("DELETE FROM assets");
}
 
[Test]
public void T101_TestFetchAsset()
{
    AssetBase a1_actual = db.FetchAsset(a1.uuid);
 
    Assert.Equal(a1_actual.uuid, a1.uuid);
    Assert.Equal(a1_actual.Name, a1.Name);
    // etc
}
 
[Test]
public void T103_TestUpdateAsset()
{
    a1.Name = "new name";
 
    db.UpdateAsset(a1.uuid, a1);
 
    AssetBase a1_actual = sqldb.executeSQL("SELECT * FROM assets WHERE uuid = {0}", a1.uuid);
 
    Assert.Equal(a1_actual.uuid, a1.uuid);
    Assert.Equal(a1_actual.Name, a1.Name);
    // etc
}

Also note the TearDown() method; it is called after each test has run, regardless whether the test passed or failed. It deletes all the entries in the assets table so that there is no leftover data in the database to interfere with the next test.

Multiple Setup Methods

Not all setup and teardown must happen in methods declared [SetUp] and [TearDown]. It may be useful to provide methods which perform part of the setup and to call them from whichever test may need it:

private AssetBase InsertAssetWithRandomData(UUID assetUuid)
{
    AssetBase asset = new AssetBase(assetUuid);
    asset.Name = somethingRandom();
    asset.Data = somethingRandom();
 
    sqldb.executeSQL("INSERT INTO assets VALUES({0}, ...)", asset.uuid, ...);
 
    return asset;
}
 
[Test]
public void T201_TestNeedsTwoAssets()
{
    AssetBase a1 = InsertAssetWithRandomData(uuid1);
    AssetBase a2 = InsertAssetWithRandomData(uuid2);
    // etc
}
 
[Test]
public void T203_TestNeedsFiveAssets()
{
    AssetBase a1 = InsertAssetWithRandomData(uuid1);
    AssetBase a2 = InsertAssetWithRandomData(uuid2);
    AssetBase a3 = InsertAssetWithRandomData(uuid3);
    AssetBase a4 = InsertAssetWithRandomData(uuid4);
    AssetBase a5 = InsertAssetWithRandomData(uuid5);
    // etc
}

Notice that InsertAssetWithRandomData is private as it's only called from within the class.

Asserts

You will see scattered through the code Assert.That(...). These will throw an exception if the condition is not valid. This format of assertions is called the Constraint Model in NUnit, and provides a large number of tests with the flavor of:

• Assert.That(foo, Is.Null)
• Assert.That(foo, Is.Not.Null)
• Assert.That(foo, Is.True)
• Assert.That(foo, Is.EqualTo(bar))
• Assert.That(foo, Text.Matches( "*bar*" ))

Of note, Is.EqualTo uses the Equals function of foo, so this can only be used on objects that are IComparable. Most of the OpenSimulator base objects are not, so you'll have to compare fields manually in tests.

For the complete set of conditions you can use see the Constraint Model NUnit documentation. While there is another syntax for tests, the Constraint Model is preferred as it is far more human readable.

Simple Negative Tests

See Example Test SQLite Assets for this code snipped in context.

[Test]
public void T001_LoadEmpty()
{
    Assert.That(db.ExistsAsset(uuid1), Is.False);
    Assert.That(db.ExistsAsset(uuid2), Is.False);
    Assert.That(db.ExistsAsset(uuid3), Is.False);
}

Test T001 is an example of a simple negative test. We assume a new database will not have any of those assets in them. While the value of this test may look low, it does provide a baseline in ensuring that the database connection is there, that these return false correctly, and that no other exception is thrown. Negative tests are a good way to force bounds conditions and ensure that not only does it return what you expect it also doesn't return what you don't expect. Thought of another way, it ensures your code is somewhat defensive in nature, not coughing on bad or unexpected data.

Simple Positive Tests

See Example Test SQLite Assets for this code snipped in context.

[Test]
public void T010_StoreSimpleAsset()
{
    AssetBase a1 = new AssetBase(uuid1, name1);
    AssetBase a2 = new AssetBase(uuid2, name2);
    AssetBase a3 = new AssetBase(uuid3, name3);
    a1.Data = asset1;
    a2.Data = asset1;
    a3.Data = asset1;
 
    db.CreateAsset(a1);
    db.CreateAsset(a2);
    db.CreateAsset(a3);
 
    AssetBase a1a = db.FetchAsset(uuid1);
    Assert.That(a1a.ID, Is.EqualTo(uuid1));
    Assert.That(a1a.Name, Is.EqualTo(name1));
 
    AssetBase a2a = db.FetchAsset(uuid2);
    Assert.That(a2a.ID, Is.EqualTo(uuid2));
    Assert.That(a2a.Name, Is.EqualTo(name2));
 
    AssetBase a3a = db.FetchAsset(uuid3);
    Assert.That(a3a.ID, Is.EqualTo(uuid3));
    Assert.That(a3a.Name, Is.EqualTo(name3));
}

T010 is an example of a simple positive test. In it we create and store 3 assets (ensuring no exceptions), then load those 3 assets back from the database and ensure the fields are correct. Because AssetBase is not IComparible we just check the ID and Name fields with equals tests. If any of the Asserts fail, the whole test fails.

Stateful Tests

See Example Test SQLite Assets for this code snipped in context.

[Test]
public void T011_ExistsSimpleAsset()
{
    Assert.That(db.ExistsAsset(uuid1), Is.True);
    Assert.That(db.ExistsAsset(uuid2), Is.True);
    Assert.That(db.ExistsAsset(uuid3), Is.True);
}

T011 is an example of a stateful test, because it requires the state created by T010 (i.e. the creation of those 3 objects). In order to test any kind of complicated scenario you will find that you need to use stateful tests to build up various amounts of state (testing along the way), then manipulating and possibly tearing it down. Without doing this you can't do truly deep testing of function in any complex environment. This example isn't very stateful (I tried to pick an easy example), but it should give you some ideas.

Speculative Tests

Speculative tests are tests that might or might not apply in a given situation. MySQL testing in the OpenSimulator tree is done by speculative testing, the tests will only run if there is a properly configured database, otherwise they will not be run. If you execute Assert.Ignore() in a Test the test will end and be ignored. If you run Assert.Ignore() in the TestFixtureSetup all tests in the test fixture will be skipped and ignored.

Speculative testing lets you create tests that require certain preconditions to be met which you can't guarantee on all platforms/configuration, and are an important part of deep testing.

Adding Tests to the Tree

As we said previously all tests for assembly OpenSim.Foo (in directory OpenSim/Foo) should:

• be in assembly OpenSim.Foo.Tests.dll
• not be in the OpenSim.Foo.dll assembly
• be in OpenSim/Foo/Tests directory

Also, if you have created a new test assembly you must add references to it in .nant/local.include, both to the

• test target, and the
• test-xml target

to ensure that the assembly is added to the automated panda runs as well as the nant test target.

Executing Tests

Database Layer Tests

The connection strings for the database dependent unit tests can be configured in the file OpenSim/Data/Tests/Resources/TestDataConnections.ini. This file is an embedded resource, so the project must be recompiled before changes to it will take effect. If the connection strings are not configured then the relevant tests will just be ignored.

Panda

On every commit to opensim all the tests are run on the panda build server on opensimulator.org. The process takes about 5 minutes to build, test, and report the results back out on #opensim-dev via the osmantis bot.

Nant

You can manually run all the tests for OpenSimulator on your system by running nant test as a nant target. This will run all the tests that are in the tree, though some speculative tests might be ignored if your platform does not have the right features or configuration to run these tests.

NUnit Console

If you only want to run tests for one assembly you can do that using the NUnit Console. On Linux just run nunit-console2 OpenSim.Foo.Tests.dll and it will run only the tests for OpenSim.Foo.Tests.dll. If you are only making changes to 1 dll and just want a quick sanity check, this is the fastest way to do that.

Debugging Tests

There is a special page dedicated to this. See Debugging Unit Tests.

Learning More

You should definitely read the documentation at the NUnit homepage if you want to know more about testing. It's a very good reference for all the APIs in NUnit that you can use for creating tests.

Code Coverage

A prototype has been included using monocov, which has a profile built-in in mono. Instructions for using monocov can be found on the Mono homepage, in Code Coverage section. Unfortunately nunit2 and nant do not support code coverage with monocov (only some other proprietary code coverages), so there is no <monocov> tag. The solution was to implement it using many nunit-console and on <exec> tags.

ATTENTION: Code coverage only works with mono 1.2.x , any other version will most likely not work. Code coverage development is being put on hold for now until it supports newer mono versions.

Running

Download monocov and nunit-console if your nunit-console does not work. To test if it works (my Ubuntu's Hardy version did not), just run one of the tests with nunit-console.

Install monocov (./configure, ./make install, there could be some minor conflicts, I had to add to the compiling line -I/usr/include/mono-1.0/) and make sure the working nunit-console.exe is in /usr/lib/nunit/nunit-console.exe. Now just run nant test-cov and it will generate .cov files and HTML directories in the cov directory. The .cov files can be seen by running monocov on them, and have the same information as the HTML directories.

Testing Todo

Coverage

A prototype has been done and documented. Now we must keep a look if monocov will evolve to support newer mono versions.

Links to More Information on Unit Testing

• NUnit Quick Start
• xUnit Patterns book homepage, with lots of information on good practices, patterns, code smells, etc.
• There is a lot of information on unit testing at the Cunningham & Cunningham, Inc wiki at c2.com