++<br><br><div class="gmail_quote">On Fri, Nov 27, 2009 at 9:26 PM, orion hax <span dir="ltr"><<a href="mailto:orion.hax@gmail.com">orion.hax@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
+1 for me. From the perspective of someone getting into hosting regions haveing this kind of information is critical to do proper load balancing and haveing a good idea the health of all the servers.<div><div></div><div class="h5">
<br><br><div class="gmail_quote">
On Fri, Nov 27, 2009 at 8:21 PM, Kyle <span dir="ltr"><<a href="mailto:create@reactiongrid.com" target="_blank">create@reactiongrid.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Implementation is over my head but as a former hardware technician & old<br>
enough to have troubleshot mainframes by tape reels and blinking red lights<br>
I lived by front panel indicators to help me quickly isolate issues and<br>
train operators on what trouble indicators to look for to catch issues<br>
before they caused downtime.<br>
<br>
So to me this is a must have to help diagnose and improve stability and<br>
uptime-Brilliant!....+1<br>
<br>
Kyle G<br>
<div><div></div><div><br>
-----Original Message-----<br>
From: <a href="mailto:opensim-dev-bounces@lists.berlios.de" target="_blank">opensim-dev-bounces@lists.berlios.de</a><br>
[mailto:<a href="mailto:opensim-dev-bounces@lists.berlios.de" target="_blank">opensim-dev-bounces@lists.berlios.de</a>] On Behalf Of Teravus Ovares<br>
Sent: Friday, November 27, 2009 9:10 PM<br>
To: <a href="mailto:opensim-dev@lists.berlios.de" target="_blank">opensim-dev@lists.berlios.de</a><br>
Subject: [Opensim-dev] Designing with Instrumentation in mind.<br>
<br>
Hey there,<br>
<br>
A while back, we had somewhat reasonable statistics being generated<br>
and presented to the client. They were not always accurate, but<br>
based on what I saw, I could, pretty much pin certain parts of the<br>
simulator as the limiting factor during load tests. I'd say, the<br>
number 1 reason that they were semi-accurate and not accurate.. in<br>
the past.. is because nobody ever thought about instrumentation<br>
during the functionality design. It was always 'tacked on later'.<br>
One example of this.. is the current AssetCache implementation.<br>
There's no way, currently, to know, at a glance.. how many<br>
external requests it has open. Additionally, it will be extremely<br>
difficult to put one in because of the way the objects are designed<br>
and accessed. To put one in, an event needs to be added to the<br>
IAssetService interface and each AssetCache implementation will need<br>
an interlocked int to count how many gets and puts it currently has<br>
open to the external data source as well as it's own event calling<br>
schedule. Then, the IAssetService property in Scene, (AssetService)<br>
will need an event handler.. which updates the values in<br>
SimStatsReporter in Scene (StatsReporter). This idea of external<br>
access resource instrumentation should really have been built in to<br>
the design of the AssetService.<br>
<br>
This last recent load test, there were no real statistics that I could<br>
use to determine what the limiting factor was.<br>
Time Dilation was pegged at 1.0.. even when the simulator was<br>
obviously struggling. Total Frame time (MS) was -50ms even when the<br>
simulation MS was 850ms and the Physics ms was 250ms, so the<br>
inconsistencies made it impossible to know what part of the simulator<br>
was struggling. Agent Updates were erratic.. sometimes high..<br>
sometimes low when the simulator was fine and when it was struggling.<br>
Pending Uploads and Downloads were always 0, so there was no way to<br>
know how well the simulator was downloading and uploading assets to<br>
and from the grid. Packet stats were non-existant, so there was no<br>
way to know how well the UDP handlers were faring under the load.<br>
When it crashed, it crashed with a mono based stack trace which<br>
pointed to out of memory errors, so the only way that you could,<br>
scientifically, find out what the issue is.. is to run a load test<br>
under a memory profiler. We know, that running a public load test<br>
under a memory profiler is quite impractical.<br>
<br>
To make something better, I need to know two things, where it is, and<br>
where I want it to be. How can we make OpenSimulator better if we<br>
don't have statistics that point to where we are currently?<br>
<br>
On that note, I propose that, when designing objects for functionality<br>
in OpenSimulator, that we also consider if the objects should be<br>
instrumented and, what would be the best way to go about instrumenting<br>
the objects. We should incorporate instrumentation into the design of<br>
the objects. Some of that instrumentation is appropriate for a<br>
client to see, some of it might not be. Consider that, many of them<br>
should be client facing and be included in the SimStats that get sent<br>
to the client.. so that we can have a reasonable idea of what's<br>
going on with a simulator at a glance. Also, in the design of the<br>
instrumentation, we make sure that the instrumentation is accurate and<br>
lightweight.<br>
<br>
The load test went reasonably... but, we didn't get half of the<br>
information on the simulator that we needed to be able to improve it.<br>
<br>
<br>
Please comment :) I look forward to hearing your responses.<br>
<br>
Regards<br>
<br>
Teravus<br>
_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><br>
<br>
</div></div>No virus found in this incoming message.<br>
Checked by AVG - <a href="http://www.avg.com" target="_blank">www.avg.com</a><br>
Version: 9.0.709 / Virus Database: 270.14.79/2522 - Release Date: 11/27/09<br>
14:39:00<br>
<div><div></div><div><br>
<br>
_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><br>
</div></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><br>
<br></blockquote></div><br>