Automated Testing - Revision history

Manni at 11:28, 4 December 2023

2023-12-04T11:28:55Z

Justincc: Coverage is long dead.

2012-07-12T20:58:19Z

Coverage is long dead.

Justincc: /* Adding Tests to the Tree */

2012-07-12T20:57:54Z

‎Adding Tests to the Tree

Justincc: /* Adding Tests to the Tree */

2012-07-12T20:57:47Z

‎Adding Tests to the Tree

Justincc: /* Executing Tests */

2012-05-02T19:45:02Z

‎Executing Tests

Justincc at 19:19, 2 May 2012

2012-05-02T19:19:16Z

Justincc: Remove/amend obsolete parts of this document

2012-05-02T18:57:34Z

Remove/amend obsolete parts of this document

MakoBot: Robot: Replacing 'OpenSim' to 'OpenSimulator', which is the precise name

2012-03-04T06:14:24Z

Robot: Replacing 'OpenSim' to 'OpenSimulator', which is the precise name

MakoBot: Robot: Cosmetic changes

2012-03-04T02:00:02Z

Robot: Cosmetic changes

Show changes

Randomhuman: /* Executing Tests */

2011-09-17T13:13:44Z

‎Executing Tests

← Older revision		Revision as of 11:28, 4 December 2023
Line 1:		Line 1:
−	{{Quicklinks}}	+	{{Quicklinks\|Automated_Testing}}

	== Introduction ==		== Introduction ==

← Older revision		Revision as of 20:58, 12 July 2012
Line 337:		Line 337:
	You should definitely read the documentation at the [http://www.nunit.org/index.php?p=documentation NUnit homepage] if you want to know more about testing. It's a very good reference for all the APIs in NUnit that you can use for creating tests.		You should definitely read the documentation at the [http://www.nunit.org/index.php?p=documentation NUnit homepage] if you want to know more about testing. It's a very good reference for all the APIs in NUnit that you can use for creating tests.

−	~~=== Coverage ===~~
−
−	~~A prototype has been done and documented. Now we must keep a look if monocov will evolve to support newer mono versions.~~
	== Links to More Information on Unit Testing ==		== Links to More Information on Unit Testing ==

← Older revision		Revision as of 20:57, 12 July 2012
Line 325:		Line 325:

	For example		For example
−

	<exec program="${nunitcmd}" failonerror="true" resultproperty="testresult.opensim.my.new.tests">		<exec program="${nunitcmd}" failonerror="true" resultproperty="testresult.opensim.my.new.tests">

@@ Line 320: / Line 320: @@
 Also, if you have created a new test assembly you must add references
-to it in ''.nant/local.include'', both to the
+to it in ''.nant/local.include'
-* ''test'' target, and the
+to ensure that the assembly is added to the automated continuous integration test server as
-* ''test-xml'' target
-to ensure that the assembly is added to the automated panda runs as
 well as the nant ''test'' target.
 === Debugging Tests ===

@@ Line 8: / Line 8: @@
 == Executing Tests ==
 === Nant ===
-You can manually run all the tests for OpenSimulator on your system by running '''nant test''' as a nant target. This will run all the tests that are in the tree, though some speculative tests might be ignored if your platform does not have the right features or configuration to run these tests.
+You can manually run all the tests for OpenSimulator on your system by running '''nant test''' as a nant target. This will run all the tests that are in the tree, though the database layer tests will be ignored if you haven't configured a database for them (see below).  The database layer tests only comprise a small portion of the test suite and concern only the database layer - other tests use in-memory implementations of the database layer where necessary.
 === NUnit Console ===
 If you only want to run tests for one assembly you can do that using the NUnit Console. On Linux just run '''nunit-console2 OpenSim.Foo.Tests.dll''' and it will run only the tests for OpenSim.Foo.Tests.dll. If you are only making changes to 1 dll and just want a quick sanity check, this is the fastest way to do that.
 == Developing tests ==

@@ Line 1: / Line 1: @@
 {{Quicklinks}}
 As OpenSimulator matures, we are extremely interested in adding more automated verification into the OpenSimulator source tree. Testing exists not only to prevent bugs from creeping in, but also to provide fair warning that the behavior of the system has changed, and tests may need updating.
@@ Line 304: / Line 325: @@
 to ensure that the assembly is added to the automated panda runs as
 well as the nant ''test'' target.
 === Debugging Tests ===

@@ Line 10: / Line 10: @@
 # Tests testing a class should be grouped into a test class file called xxxTest.cs, where xxx is the name of the class that is being tested.
 # Tests should be able to run safely in a production environment. That means that care must be taken not to damage data on the machine that it is being run.
-# Tests should be deterministic in other words repeatable. Avoid randomness in tests. See good and bad testing practices below.
+# Tests should be deterministic in other words repeatable. Avoid randomness in tests. Avoid multiple threads whenever possible - extra test modes may be created within OpenSimulator itself to facilitate single-threaded testing.  See good and bad testing practices below.
 == Core Functionality Missing Unit Tests ==
@@ Line 24: / Line 24: @@
 Creating good tests is an art, not a science. Tests are useful by how many bugs they find or how many bugs they avoid. Things you should think about in creating good tests is:
 * Throwing edge cases, like 0, "", or Null at parameters. This ensures that people functions are hardened against incomplete data. Many of our crashes come from the lack of this hardening showing up at just the wrong time.
 * Random tests are not a good idea. We need test results to be deterministic. In other words tests need to be repeatable. If you want to test for a range it is good idea to make separate tests for min and max values. Random values in fields can fail randomly. When something goes wrong for example in database schema the developer will not necessarily notice if the stored values are random. On the other hand its hard to troubleshoot randomly failing tests as you dont know which specific value caused the failure.
 * Tests should be independent and should not rely on another test being run, passing or failing. An excerpt from [http://xunitpatterns.com/Principles%20of%20Test%20Automation.html#Independent%20Test xUnit Patterns]:
@@ Line 115: / Line 114: @@
 <source lang="csharp">[Test]
-public void T101_TestFetchAsset()
+public void TestFetchAsset()
+{
      AssetBase a1 = new AssetBase(...);
@@ Line 128: / Line 127: @@
 [Test]
-public void T103_TestUpdateAsset()
+public void TestUpdateAsset()
+{
      AssetBase a1 = new AssetBase(...);
@@ Line 163: / Line 162: @@
 [Test]
-public void T101_TestFetchAsset()
+public void TestFetchAsset()
+{
      AssetBase a1_actual = db.FetchAsset(a1.uuid);
@@ Line 173: / Line 172: @@
 [Test]
-public void T103_TestUpdateAsset()
+public void TestUpdateAsset()
+{
      a1.Name = "new name";
@@ Line 206: / Line 205: @@
 [Test]
-public void T201_TestNeedsTwoAssets()
+public void TestNeedsTwoAssets()
+{
      AssetBase a1 = InsertAssetWithRandomData(uuid1);
@@ Line 214: / Line 213: @@
 [Test]
-public void T203_TestNeedsFiveAssets()
+public void TestNeedsFiveAssets()
+{
      AssetBase a1 = InsertAssetWithRandomData(uuid1);
@@ Line 244: / Line 243: @@
 <source lang="csharp">[Test]
-public void T001_LoadEmpty()
+public void TestLoadEmpty()
+{
      Assert.That(db.ExistsAsset(uuid1), Is.False);
@@ Line 260: / Line 259: @@
 <source lang="csharp">
 [Test]
-public void T010_StoreSimpleAsset()
+public void TestStoreSimpleAsset()
+{
      AssetBase a1 = new AssetBase(uuid1, name1);
@@ Line 287: / Line 286: @@
 </source>
 T010 is an example of a simple positive test. In it we create and store 3 assets (ensuring no exceptions), then load those 3 assets back from the database and ensure the fields are correct. Because AssetBase is not IComparible we just check the ID and Name fields with equals tests. If any of the Asserts fail, the whole test fails.
 === Speculative Tests ===
@@ Line 327: / Line 310: @@
 The connection strings for the database dependent unit tests can be configured in the file OpenSim/Data/Tests/Resources/TestDataConnections.ini. This file is an embedded resource, so the project must be recompiled before changes to it will take effect. If the connection strings are not configured then the relevant tests will just be ignored.
-=== Panda ===
+=== Jenkins ===
-On every commit to opensim all the tests are run on the [http://panda.opensimulator.org/ panda build server on opensimulator.org]. The process takes about 5 minutes to build, test, and report the results back out on #opensim-dev via the osmantis bot.
+On every commit to opensim all the tests are run on the [http://jenkins.opensimulator.org/ Jenkins build server on opensimulator.org]. The process takes about 5 minutes to build, test, and report the results back out on #opensim-dev via the osmantis bot.
 === Nant ===
@@ Line 341: / Line 324: @@
 == Learning More ==
 You should definitely read the documentation at the [http://www.nunit.org/index.php?p=documentation NUnit homepage] if you want to know more about testing. It's a very good reference for all the APIs in NUnit that you can use for creating tests.
 === Coverage ===

@@ Line 2: / Line 2: @@
 {{Quicklinks}}
-As OpenSim matures, we are extremely interested in adding more automated verification into the OpenSim source tree. Testing exists not only to prevent bugs from creeping in, but also to provide fair warning that the behavior of the system has changed, and tests may need updating.
+As OpenSimulator matures, we are extremely interested in adding more automated verification into the OpenSimulator source tree. Testing exists not only to prevent bugs from creeping in, but also to provide fair warning that the behavior of the system has changed, and tests may need updating.
-In OpenSim today we use NUnit tests. Our conventions are:
+In OpenSimulator today we use NUnit tests. Our conventions are:
 # Tests should '''not''' exist inside runtime assemblies, as this makes nunit a production requirement
 # Tests should be in .Tests.dll assemblies. For instance, the tests for OpenSim.Data.SQLite.dll should be in the OpenSim.Data.SQLite.Tests.dll assembly. This allows for easy removal of test assemblies in products.
@@ Line 102: / Line 102: @@
 </source>
-In the case of testing something like the database layer, we have to actually attempt to store / retrieve things from a database. Following from rule #4 of good tests, we want to make sure not to touch the production databases to run our tests, so during startup we generate a temporary file name which is guaranteed not to be an existing file on the system, and use that as our database file name. By running db.Initialize() the OpenSim migration code will correctly populate that database with the latest schema.
+In the case of testing something like the database layer, we have to actually attempt to store / retrieve things from a database. Following from rule #4 of good tests, we want to make sure not to touch the production databases to run our tests, so during startup we generate a temporary file name which is guaranteed not to be an existing file on the system, and use that as our database file name. By running db.Initialize() the OpenSimulator migration code will correctly populate that database with the latest schema.
 Once we are done with the tests we want to make sure we aren't leaving garbage temp files on the user's system. So we remove that file we created.
@@ Line 235: / Line 235: @@
 * Assert.That(foo, Text.Matches( "*bar*" ))
-Of note, Is.EqualTo uses the Equals function of foo, so this can only be used on objects that are IComparable. Most of the OpenSim base objects are not, so you'll have to compare fields manually in tests.
+Of note, Is.EqualTo uses the Equals function of foo, so this can only be used on objects that are IComparable. Most of the OpenSimulator base objects are not, so you'll have to compare fields manually in tests.
 For the complete set of conditions you can use see [http://www.nunit.org/index.php?p=constraintModel&r=2.4 the Constraint Model NUnit documentation]. While there is another syntax for tests, the Constraint Model is preferred as it is far more human readable.
@@ Line 305: / Line 305: @@
 === Speculative Tests ===
-Speculative tests are tests that might or might not apply in a given situation. MySQL testing in the OpenSim tree is done by speculative testing, the tests will only run if there is a properly configured database, otherwise they will not be run. If you execute '''Assert.Ignore()''' in a '''Test''' the test will end and be ignored. If you run '''Assert.Ignore()''' in the '''TestFixtureSetup''' all tests in the test fixture will be skipped and ignored.
+Speculative tests are tests that might or might not apply in a given situation. MySQL testing in the OpenSimulator tree is done by speculative testing, the tests will only run if there is a properly configured database, otherwise they will not be run. If you execute '''Assert.Ignore()''' in a '''Test''' the test will end and be ignored. If you run '''Assert.Ignore()''' in the '''TestFixtureSetup''' all tests in the test fixture will be skipped and ignored.
 Speculative testing lets you create tests that require certain preconditions to be met which you can't guarantee on all platforms/configuration, and are an important part of deep testing.
@@ Line 331: / Line 331: @@
 === Nant ===
-You can manually run all the tests for OpenSim on your system by running '''nant test''' as a nant target. This will run all the tests that are in the tree, though some speculative tests might be ignored if your platform does not have the right features or configuration to run these tests.
+You can manually run all the tests for OpenSimulator on your system by running '''nant test''' as a nant target. This will run all the tests that are in the tree, though some speculative tests might be ignored if your platform does not have the right features or configuration to run these tests.
 === NUnit Console ===

← Older revision		Revision as of 13:13, 17 September 2011
Line 323:		Line 323:

	== Executing Tests ==		== Executing Tests ==
		+
		+	=== Database Layer Tests ===
		+	The connection strings for the database dependent unit tests can be configured in the file OpenSim/Data/Tests/Resources/TestDataConnections.ini. This file is an embedded resource, so the project must be recompiled before changes to it will take effect. If the connection strings are not configured then the relevant tests will just be ignored.

	=== Panda ===		=== Panda ===