[Opensim-dev] Unit tests should not be coupled (WAS: Re: [Opensim-commits] r8737 - trunk/OpenSim/Framework/Communications/Tests)

[Sean Dague]

Mike Mazur wrote:
> Hi,
> 
> On Wed, 11 Mar 2009 12:30:18 -0300
> Arthur Valadares <arthursv at linux.vnet.ibm.com> wrote:
> 
>> Mike, you seem to have some good knowledge in this area, would be nice
>> if you could contribute to Automated Testing in wiki with this
>> information, as none of this is talked about there. 
> 
> Thank you.
> 
> I have spent some time today to put some information on the wiki page.
> Have a look at
> 
> http://opensimulator.org/wiki/Automated_Testing
> 
> and also
> 
> http://opensimulator.org/wiki/Example_Test_SQLite_Assets
> 
> where I placed the whole SQLiteAssetTest example.
> 
> I'm not sure that an in-depth tutorial on unit testing itself really
> belongs on the OpenSim wiki, though. Other resources should be perused
> to learn more about unit testing, such as the NUnit tutorial:
> 
> http://www.nunit.org/index.php?p=quickStart&r=2.4

I know I'm late on this thread (sorry, was out for a couple weeks), but
wanted to put a little perspective here.

I created the model for testing the database code.  We can play
semantics on whether database tests are unit tests or not, but at the
end of the day, if the database don't work, all breaks, and the end user
cries.  Hence, testing there is good. :)

In my experience, first order unit tests, i.e. stateless tests for
objects, aren't usually very interesting, and rarely find bugs.
Complexity comes in the interaction between a couple of components where
state is involved.

The database tests have found at least a dozen tree breaks that might
have taken a long time to sort out otherwise, so I think there is a lot
of value there.

While it is possible to make each of these tests stateless, you cause
some new problems if you do that:
 * it's more work, which means people are less likely to contribute the
tests
 * it takes more time to execute, as you are constantly building up and
destroying the same state.  If the tests take longer to run, less people
will run them.

If we had a battery of tests that had a bunch of expected fails, then it
would be more important to know more than just the first break point.
But our tests are all supposed to always run, so the fact that we stop
at the first break is ok.  That exposes the earliest point where things
when wrong.  You fix that and move onto the next issue.

Chaining of tests for complex interactions is a shortcut, but I really
think it's a useful one, as it provides much higher levels of complexity
testing for less code and less run time.

I really think that this is a case of theory vs. practice.  Theory says
we shouldn't do it this way, but in practice it works better (at least
from my experience doing this with 2 prior large scale projects).

	-Sean

-- 
Sean Dague / Neas Bade
sdague at gmail.com
http://dague.net


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20090323/f6bd2e2f/attachment-0001.pgp>