I was able to re-create the "slow get" error. In my case I had a region with many unique textures. As my viewer was downloading them I was apparently running short of memory and the computer started swapping and thrashing the disk and the viewer would momentarily freeze. During the freezes the OpenSimulator console would show several of the "slow get" messages. Apparently the viewer was not able to receive and process the requested textures as fast as the simulator wanted to send them and this is when the messages appeared. I reduced the draw distance and graphics quality level in the viewer and the messages stopped. Given this experience I suspect some of the users at your parties may be having similar viewer issues and the problem may be mitigated by suggesting they lower their draw distance and graphics quality settings.<div>
<br></div><div>I believe most viewers send information back to the simulator about memory use and graphics performance but I'm not aware if OpenSimulator is collecting and making these data available for analysis.</div>
<div><br><br><div class="gmail_quote">On Sun, Apr 22, 2012 at 3:43 PM, Akira Sonoda <span dir="ltr"><<a href="mailto:akira.sonoda.1@gmail.com">akira.sonoda.1@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_extra">Hi Justin, <br><br>I am still investigating the Slow GetTexture problem and the resulting instability under high Load.<br><br>What i did so far: <br><br><ol><li>I'm still using opensim-0007711 ( i didn't have the time to upgrade, the first upgrade was not so good due an error on my side and since then i did not look too much into it)</li>
<li>I've created a Windows Instance in the Amazon cloud in order to be able to connect some profiling tools.</li><li>I've run the last two Friday Parties from there the first Party was quite okay ( MaxPoolThreads=90 in the SmartPoolThreads settings, but i saw more on peaks, strange )</li>
<li>The second party from the 20. April went crazy after 3 hours here's a picture:</li></ol><p><a href="http://farm9.staticflickr.com/8017/6957771466_4412ee83c4_b_d.jpg" target="_blank">http://farm9.staticflickr.com/8017/6957771466_4412ee83c4_b_d.jpg</a></p>
<p>Most of the threads had a stack trace like this:</p><p><a href="http://farm9.staticflickr.com/8155/6957875232_0203631ed0_b_d.jpg" target="_blank">http://farm9.staticflickr.com/8155/6957875232_0203631ed0_b_d.jpg</a></p>
<p>Wondering why this increase started after approx 3 hours. We had at max about 18 avies on the region/sim with various different viewers. Because I did not attach this amazon Cloud instance to my splunk server i have no statistics about the viewers during the party ... i probably should do that in future. <br>
</p><p>I will upgrade to a more recent version next week ... <br></p><p>Thanks a lot!</p><p>Akira<br></p><p><br></p><div class="gmail_quote">Am 19. März 2012 01:57 schrieb Justin Clark-Casey <span dir="ltr"><<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a>></span>:<div>
<div class="h5"><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It's quite possible the 3rd party HTTP server doesn't use the threadpool though I've never looked in detail.<br>
<br>
You could supply any other http address in the GetTexture cap (e.g. the asset service directly with a suitable handler). However, I'm not sure that asset serving is such a bottleneck at the moment compared with scripting and physics issues.<div>
<br>
<br>
On 17/03/12 21:10, Dahlia Trimble wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
I've done a bit of tracing through the code and I can't seem to find where the http server in OpenSimulator uses<br>
threadpool threads. I did find them used in the LLUDP server and in asyncronous requests from the asset service, but I<br>
have yet to find any other uses. is it possible that the http server is still using system threads, bypassing the<br>
threadpool? I'm rather curious as I use the built-in http server in a few personal applications and I'm concerned about<br>
performance.<br>
<br>
On another note, I believe part of the impetus behind LL designing the texture fetch capability was to allow a separate<br>
service from the simulator to supply assets to viewers, thereby reducing load on the individual simulator processes.<br>
Perhaps this is something OpenSimulator can take advantage of? Probably some kind of asset proxy cache could do a much<br>
better job of serving textures and other assets to viewers than the existing monolithic process? I believe it could even<br>
be moved to a separate server with a different IP address.<br>
<br>
<br></div><div>
On Fri, Mar 16, 2012 at 8:36 PM, Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a> <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>> wrote:<br>
<br>
Hi Akira. I have now updated the "show threads" method to show threadpool statistics for the main threadpool.<br>
Please note that each XEngine script engine will also have it's own threadpool (which can be seen using the<br>
"xengine status" command). Might need to improve this further.<br>
<br>
"show threads" will also show all in-use threads. However, at least on mono 2.6.7 this isn't reported by the VM so<br>
won't be shown.<br>
<br>
I'm not sure whether this will help you or not in tracking down performance issues. In some situations it could<br>
help (e.g. if threads are encountering deadlock the number of 'in use' threads will leap up, though you've probably<br>
already noticed deadlock by the long-running threads reporting monitoring failures and the sim locking up).<br>
<br>
So I'd be happy to hear suggestions for additional data and I'll implement them if I can, since I think this is<br>
going to be a growing area of concern. Unfortunately pinning down performance issues with a system as complex as<br>
OpenSimulator (with massive numbers of threads and user generated scripts) is likely to remain a significant<br>
challenge for the forseeable future.<br>
<br>
<br>
On 11/03/12 19:15, Akira Sonoda wrote:<br>
<br>
Am 10. März 2012 03:25 schrieb Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a> <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>><br>
</div>
<mailto:<a href="mailto:jjustincc@googlemail." target="_blank">jjustincc@googlemail.</a>_<u></u>_com <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>>>:<div>
<br>
<br>
<br>
I'm sorry to say that you'll have to take the ThreadPool numbers with a very very very large pinch of salt. I<br>
believe they only refer to the built-in mono thread pool and not the SmartThreadPool which is the one<br>
actually used<br>
(and beyond that the core simulator and xengine use separate pools). I will try and improve this situation<br>
soon.<br>
<br>
<br>
Thank you Justin!<br>
<br>
Would be nice to have some meaningful statistics for all those ThreadPools! Maybe there is a possibility to<br>
write those<br>
statistics to the log from time to time ( e.g. every 30 seconds). Together with some documented "best practices"<br>
from<br>
those who operate Sims, with lots of avatars on it - I'm thinking mainly the OSgrid Plazas are good references -<br>
this<br>
could be highly valuable information for those who operate Sims for similar purposes ( meeting points, parties,<br>
concerts<br>
etc. )<br>
<br>
<br>
<br>
<br>
<br>
<br></div>
______________________________<u></u>___________________<div><br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a> <mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>><br>
</div>
<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a> <<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a>><div>
<br>
<br>
<br>
<br>
--<br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br></div>
______________________________<u></u>___________________<div><br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a> <mailto:<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.<u></u>berlios.de</a>><br></div>
<a href="https://lists.berlios.de/__mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/__<u></u>mailman/listinfo/opensim-dev</a> <<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a>><div>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a><br>
</div></blockquote><div><div>
<br>
<br>
-- <br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/<u></u>mailman/listinfo/opensim-dev</a><br>
</div></div></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><br></blockquote></div><br></div>