<div class="gmail_extra">:-) just to be clear I am talking about git hash 007711 ( opensim 0.7.4 recommended in the osgrid twitter at the first of march 2012 ). I've seen a newer recommended version just came out ... and I'll give that one a try.<br>
<br>Thank you for supporting my thought about a deadlock! I'll have a look at the 700 MB of data I just have and try to go deeper into the analysis. I expect the YourKit profiler to be helpful in this regard.<br><br>Regards,<br>
Akira<br><br><div class="gmail_quote">Am 24. April 2012 02:45 schrieb Justin Clark-Casey <span dir="ltr"><<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Those kind of traces (where lots of threads are waiting for WaitHandler.WaitOne() or similar) are usually indicative of deadlock somewhere in the system.<br>
<br>
When this happens, I find that the best thing to do is take a vm thread dump and inspect the code to find out where the deadlock is happening. I don't know how to do this on Windows but I'm sure it's possible. Windows may well even provide some nice tools to identify the deadlock place rather than eyeballing the stacks.<br>
<br>
But to be honest, if that's opensim 0.7.1.1 then the code has changed a lot since then. There's a very good chance any problems you find will not apply to OpenSim 0.7.3 or even 0.7.2. I suggest you upgrade before doing any other analysis. Even OpenSim 0.7.3.1 will quickly become out of date - if you really want to winkle out bugs then mainline is often the place to be (though you could also try opensim-0.7.3-extended git branch which is 0.7.3.1 + selected things from git master - should be stable but not absolutely guaranteed.<br>
<br>
Just to be clear, I'm happy to eyeball problems on recent releases or master but I lack the time to do thorough analysis. However, I'm happy to add more stats/diagnostic commands to OpenSimulator if useful stuff can be identified and it's not super-difficult to do.<br>
<br>
Best,<br>
<br>
Justin<div class="im"><br>
<br>
<br>
On 22/04/12 23:43, Akira Sonoda wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Hi Justin,<br>
<br>
I am still investigating the Slow GetTexture problem and the resulting instability under high Load.<br>
<br>
What i did so far:<br>
<br></div>
1. I'm still using opensim-0007711 ( i didn't have the time to upgrade, the first upgrade was not so good due an error<div class="im"><br>
on my side and since then i did not look too much into it)<br></div>
2. I've created a Windows Instance in the Amazon cloud in order to be able to connect some profiling tools.<br>
3. I've run the last two Friday Parties from there the first Party was quite okay ( MaxPoolThreads=90 in the<div class="im"><br>
SmartPoolThreads settings, but i saw more on peaks, strange )<br></div>
4. The second party from the 20. April went crazy after 3 hours here's a picture:<div class="im"><br>
<br>
<a href="http://farm9.staticflickr.com/8017/6957771466_4412ee83c4_b_d.jpg" target="_blank">http://farm9.staticflickr.com/<u></u>8017/6957771466_4412ee83c4_b_<u></u>d.jpg</a><br>
<br>
Most of the threads had a stack trace like this:<br>
<br>
<a href="http://farm9.staticflickr.com/8155/6957875232_0203631ed0_b_d.jpg" target="_blank">http://farm9.staticflickr.com/<u></u>8155/6957875232_0203631ed0_b_<u></u>d.jpg</a><br>
<br>
Wondering why this increase started after approx 3 hours. We had at max about 18 avies on the region/sim with various<br>
different viewers. Because I did not attach this amazon Cloud instance to my splunk server i have no statistics about<br>
the viewers during the party ... i probably should do that in future.<br>
<br>
I will upgrade to a more recent version next week ...<br>
<br>
Thanks a lot!<br>
<br>
Akira<br>
<br>
<br></div>
Am 19. März 2012 01:57 schrieb Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a> <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>>:<div class="im">
<br>
<br>
It's quite possible the 3rd party HTTP server doesn't use the threadpool though I've never looked in detail.<br>
<br>
You could supply any other http address in the GetTexture cap (e.g. the asset service directly with a suitable<br>
handler). However, I'm not sure that asset serving is such a bottleneck at the moment compared with scripting and<br>
physics issues.<br>
<br>
<br>
On 17/03/12 21:10, Dahlia Trimble wrote:<br>
<br>
I've done a bit of tracing through the code and I can't seem to find where the http server in OpenSimulator uses<br>
threadpool threads. I did find them used in the LLUDP server and in asyncronous requests from the asset service,<br>
but I<br>
have yet to find any other uses. is it possible that the http server is still using system threads, bypassing the<br>
threadpool? I'm rather curious as I use the built-in http server in a few personal applications and I'm<br>
concerned about<br>
performance.<br>
<br>
On another note, I believe part of the impetus behind LL designing the texture fetch capability was to allow a<br>
separate<br>
service from the simulator to supply assets to viewers, thereby reducing load on the individual simulator processes.<br>
Perhaps this is something OpenSimulator can take advantage of? Probably some kind of asset proxy cache could do<br>
a much<br>
better job of serving textures and other assets to viewers than the existing monolithic process? I believe it<br>
could even<br>
be moved to a separate server with a different IP address.<br>
<br>
<br>
On Fri, Mar 16, 2012 at 8:36 PM, Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a> <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>><br>
</div><div class="im">
<mailto:<a href="mailto:jjustincc@googlemail." target="_blank">jjustincc@googlemail.</a>_<u></u>_com <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>>> wrote:<br>
<br>
Hi Akira. I have now updated the "show threads" method to show threadpool statistics for the main threadpool.<br>
Please note that each XEngine script engine will also have it's own threadpool (which can be seen using the<br>
"xengine status" command). Might need to improve this further.<br>
<br>
"show threads" will also show all in-use threads. However, at least on mono 2.6.7 this isn't reported by the VM so<br>
won't be shown.<br>
<br>
I'm not sure whether this will help you or not in tracking down performance issues. In some situations it could<br>
help (e.g. if threads are encountering deadlock the number of 'in use' threads will leap up, though you've<br>
probably<br>
already noticed deadlock by the long-running threads reporting monitoring failures and the sim locking up).<br>
<br>
So I'd be happy to hear suggestions for additional data and I'll implement them if I can, since I think this is<br>
going to be a growing area of concern. Unfortunately pinning down performance issues with a system as<br>
complex as<br>
OpenSimulator (with massive numbers of threads and user generated scripts) is likely to remain a significant<br>
challenge for the forseeable future.<br>
<br>
<br>
On 11/03/12 19:15, Akira Sonoda wrote:<br>
<br>
Am 10. März 2012 03:25 schrieb Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a><br>
<mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>> <mailto:<a href="mailto:jjustincc@googlemail." target="_blank">jjustincc@googlemail.</a>_<u></u>_com <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>><br>
</div>
<mailto:<a href="mailto:jjustincc@googlemail" target="_blank">jjustincc@googlemail</a>. <mailto:<a href="mailto:jjustincc@googlemail" target="_blank">jjustincc@googlemail</a>.><u></u>____com <mailto:<a href="mailto:jjustincc@googlemail." target="_blank">jjustincc@googlemail.</a>_<u></u>_com<div class="im">
<br>
<mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>>>>:<br>
<br>
<br>
<br>
I'm sorry to say that you'll have to take the ThreadPool numbers with a very very very large pinch<br>
of salt. I<br>
believe they only refer to the built-in mono thread pool and not the SmartThreadPool which is the one<br>
actually used<br>
(and beyond that the core simulator and xengine use separate pools). I will try and improve this<br>
situation<br>
soon.<br>
<br>
<br>
Thank you Justin!<br>
<br>
Would be nice to have some meaningful statistics for all those ThreadPools! Maybe there is a possibility to<br>
write those<br>
statistics to the log from time to time ( e.g. every 30 seconds). Together with some documented "best<br>
practices"<br>
from<br>
those who operate Sims, with lots of avatars on it - I'm thinking mainly the OSgrid Plazas are good<br>
references -<br>
this<br>
could be highly valuable information for those who operate Sims for similar purposes ( meeting points,<br>
parties,<br>
concerts<br>
etc. )<br>
<br>
<br>
<br>
<br>
<br>
<br></div>
______________________________<u></u>_____________________<br>
<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a> <mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>> <mailto:<a href="mailto:Opensim-dev@lists." target="_blank">Opensim-dev@lists.</a>__<a href="http://berlios.de" target="_blank">be<u></u>rlios.de</a><br>
<mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>>><br>
<a href="https://lists.berlios.de/____mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/____<u></u>mailman/listinfo/opensim-dev</a><br>
<<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a>><div class="im"><br>
<<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a> <<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a>>><br>
<br>
<br>
<br>
<br>
--<br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br></div>
______________________________<u></u>_____________________<br>
<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a> <mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>> <mailto:<a href="mailto:Opensim-dev@lists." target="_blank">Opensim-dev@lists.</a>__<a href="http://berlios.de" target="_blank">be<u></u>rlios.de</a><br>
<mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>>><br>
<a href="https://lists.berlios.de/____mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/____<u></u>mailman/listinfo/opensim-dev</a><br>
<<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a>><br>
<<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a> <<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a>>><div class="im">
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a> <mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>><br>
<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a> <<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a>><br>
<br>
<br>
<br>
--<br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
______________________________<u></u>___________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a> <mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>><br>
<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a> <<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a>><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a><br>
</div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
-- <br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a><br>
</div></div></blockquote></div><br></div>