[Opensim-dev] Unit tests should not be coupled (WAS: Re: [Opensim-commits] r8737 - trunk/OpenSim/Framework/Communications/Tests)

[Stefan Andersson]

Yo,

 

> I created the model for testing the database code. We can play
> semantics on whether database tests are unit tests or not, but at the
> end of the day, if the database don't work, all breaks, and the end user
> cries. Hence, testing there is good. :)


No arguing from me there.

 
> In my experience, first order unit tests, i.e. stateless tests for
> objects, aren't usually very interesting, and rarely find bugs.

 

That of course depends on how it was coded. One of the benefits with TDD is that it pushes the code towards low-state low-coupling. Which makes the code even simpler to test, and needed changes in test cases more localized.


> Complexity comes in the interaction between a couple of components where
> state is involved.


Which, both of them, pushes towards entangled difficult-to-maintain test-cases where you don't really know what the test is actually testing. The complexity should be adressed on each side of the contract. If something fails, it's should be one component or the other, not both in tandem.


> The database tests have found at least a dozen tree breaks that might
> have taken a long time to sort out otherwise, so I think there is a lot
> of value there.


Of course there is. The full test suite should be run before each commit. But the tests should be divided into unit tests and integration tests, so that you can run the unit tests far more often than the integration tests.


> While it is possible to make each of these tests stateless, you cause
> some new problems if you do that:
> * it's more work, which means people are less likely to contribute the
> tests

 

Stateful tests are bad for a number of very good reasons, most already mentioned here.


> * it takes more time to execute, as you are constantly building up and
> destroying the same state. If the tests take longer to run, less people
> will run them.


Which won't be a problem if you're only supposed to run them before commit, and/or the CI catches them.


> If we had a battery of tests that had a bunch of expected fails, then it
> would be more important to know more than just the first break point.
> But our tests are all supposed to always run, so the fact that we stop
> at the first break is ok. That exposes the earliest point where things
> when wrong. You fix that and move onto the next issue.


It's a very common thing to wanting to re-run only the test that failed.


> I really think that this is a case of theory vs. practice. Theory says
> we shouldn't do it this way, but in practice it works better (at least
> from my experience doing this with 2 prior large scale projects).


And in my experience from two prior large scale projects, practise says the opposite.

 

One of the 'killers' in CI involving tests is when the CI builder starts failing intermittently on integration tests just because something on the CI machine is configured in a certain way, or takes a certain time to execute. Having chained integrations tests in this setting will lead to real, practical, grief and can stall the testing for a long time, until the group identifies the failing link. This is most oftenly the point in time where the team re-writes the stateful tests into state-less.


/Stefan

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20090323/a6f52f98/attachment-0001.html>