[Opensim-dev] Lockless Lists?

[Mariusz Nowostawski]

Dr Scofield wrote:
> Christopher Yeoh wrote:
>   
>> On Mon, 24 Nov 2008 20:43:09 +0100
>> "Homer Horwitz" <homerhorwitz at googlemail.com> wrote:
>>     
>>> So, I'm not sure if we really should do that move. If at all, I'm for
>>> a very slow move to lock-free versions from a rather stable software
>>> base (which we currently don't have in trunk), so errors that are
>>> introduced during that move are more easily identifiable, with much
>>> testing in-between. Even then, I'm absolutely sure we will get a lot
>>> of Heisenbugs in the process, which will take us weeks to find.
>>>
>>>       
>> Perhaps what is really needed here is some performance benchmarks 
>> which highlight existing problems? So for individual
>> changes to lockless versions we better see what improvements we'd
>> get on both small and large SMP machines and whether its worth the
>> increase in complexity.
>>
>> I did some debugging of a deadlock a couple of weeks ago and found it
>> already pretty complicated. Any suggestions on how other people
>> approach these problems with OpenSim? I sprinkled lots of console
>> messages around as mdb doesn't seem to work for me, but in retrospect
>> it would have been really handy to have been able to just turn on
>> a lock debugging flag and have debug output when locks are taken
>> and released.
>>     
>
> in theory, you can trace specific instructions with mono...in theory, because i
> haven't succeeded yet in getting this really done without being drowned in
> console messages (sean might have more experience with this).
>
> i agree console messages are not really the cat's whiskers and the additional
> problem with console messages is that it will change timing and might get you
> nowhere..


Hi,

I've been following this discussion with interest.

Would it be possible for someone with a good understanding of opensim 
architecture provide a schematic description of where the threads are 
being created, for what purpose and how much inter-thread dependencies 
are there, please? In particular, what are the shared datastructures and 
the like. Digging it all from the code is possible but if some has it in 
her/his head it would speed up the analysis.

We've done some testing of different software packages on multicore 
systems, and generally for the best performance one needs to plan the 
threading model, thread pooling and thread syncing explicitly, without 
leaving much to the runtime and operating systems. What I mean here is 
that multi-threaded applications with "normal" threads and mutexes can 
achieve better performance to a single threaded ones in many cases - 
however, applications that plan and work with custom threading usually 
outperform the latter. We are investigating some areas of improving the 
code, and here is a list of some "hints" that may trigger some further 
discussions.

- mutexes should be removed and lockless datastructures used, but, only 
where it make sense. In some cases having a lock is unavoidable. 
Suggested removal of normal blocking locks with spin-locks will not help 
with performance at all - to the contrary - on 2 core machine one core 
will be busy spin-locking waiting for the lock to become available 
instead of doing something useful in the meantime

- many thread usually decrease performance. Having more than 2-4 threads 
on 2-core machine will decrease overall performance. Given that most 
opensim deployments are on 2- or 4-core chips, having 10s or more 
threads is not helping in terms performance. Rather to the contrary. 

- creation and destruction of threads is quite costly. Having thread  
pools instead would help.

- scheduling events and packet processing subject to priorities would 
definitely help, and I'm quite surprised that this has not been 
investigated more thoroughly. Servers need to deal with increasing 
number of events and network traffic. Having a well-designed priority 
mechanisms - not on the thread level, but rather on the event type and 
packet type level would help in managing responsiveness and general 
perceived performance of the server.


Note - I'm not as familiar with certain opensim internals as many of you 
are and I would appreciate some guidance.

best regards
Mariusz