<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Justin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Are you seeing duplicate serial numbered packets arriving at the viewer? What is the packet type of those packets? After the patch you mentioned, we no longer
save any entity update or property packets so they cannot be resent. Instead, the update is requeued into the entity update and eventually a new packet (with new serial number) will be built and sent. Knowing the type of packet being resent will help to determine
the cause. Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">~Dan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> opensim-dev-bounces@lists.berlios.de [mailto:opensim-dev-bounces@lists.berlios.de]
<b>On Behalf Of </b>Mic Bowman<br>
<b>Sent:</b> Wednesday, October 12, 2011 8:32 AM<br>
<b>To:</b> opensim-dev@lists.berlios.de<br>
<b>Subject:</b> Re: [Opensim-dev] Continual object update resends if acks are missed<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Couple things...<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> First... here's the link to the thread on some of the original issues... not sure if this went in to dan's retransmission fix but i'll get to that in a minute: <a href="https://lists.berlios.de/pipermail/opensim-dev/2011-March/010029.html">https://lists.berlios.de/pipermail/opensim-dev/2011-March/010029.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Second... In spite of all the questions/comments below that this is the "expected" behavior... there may be a bug & we need more data to find it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Question... do you have the adaptive throttling turned on? Look for enable_adaptive_throttles. It is turned off by default. That is the switch that is designed to fix the problem you are seeing... but the code has some other behaviors which
are correct but... "different" than what most admins expect (the throttle mimics the TCP slow start to identify and avoid congestion though not quite as conservatively as the TCP algorithm).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One explanation for what you are seeing is that the network simply cannot send as many packets as you are asking. This is frequently true for long-haul connections. When we did our analysis, we saw a "cliff". Once you pass the point where
the number of outgoing packets exceeds the physical networks ability to deliver them ACKs are not processed in time and very bad things happen. That is... enough ACKs are coming late enough so that the resend packet is already in the queue. The current version
of the code handles that situation far better than the old version (the current version is bound by the number of dynamic objects... the old version would hold *every* update to that object... even ones that were superceded by later updates). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The only real solution to this problem is to stop sending updates so fast or to be much more conservative about resends. The first can be addressed one of three ways... First, configure the simulator so that the outgoing bw per connection
is throttled much lower. Second, turn down the BW setting on the viewer. Third, turn on the adaptive throttling code in the simulator. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Being conservative about resends has its own share of problems... Basically what this means is that you wait longer to resend a packet. However... if you wait too long to retransmit, then a portion of your scene is not updating. The algorithm
for determining when to retransmit is based on a commonly used algorithm described in one of the RFCs (the actual RFC is referenced in a comment in the opensim code).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As to the UDP packet vs going back through the SceneEntityUpdate queue... well that one is a no brainer... If you use the UDP packet for retranmission you are almost always sending useless old data and just filling up the network with fluff.
Outside of the initial scene load, you see streams of updates for objects more often than a single update. If you retransmit the UDP packet, you are almost always sending old data. Going through the sceneentityupdate queue basically says "retransmit the current
state of the object".<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--mic<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Oct 12, 2011 at 5:56 AM, Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com">jjustincc@googlemail.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal">Yes, I may not have the correct cause in my original e-mail but I'm certainly seeing a lot of resends that never appear to stop. The effect is not consistent - sometimes I can log in to the remote sim and there will be no continuous resend.
But most of the time a large proportion of updates appear to be continuously resent.<br>
<br>
I'm surprised no-one else has reported it though I did see a lot of continuous updating on Wright Plaza @
<a href="http://osgrid.org" target="_blank">osgrid.org</a> last night, which I don't think is related to changes made by scripts. Perhaps it's only really noticeable if you start looking, have visual object updates development/advanced settings on in a viewer
or are wondering why viewer received packets keeps spiking.<br>
<br>
As far as I can see from LLClientView/LLUDPServer, both prim and property updates resends are happening but I need to study the code more closely and possibly old e-mails.<br>
<br>
I hope to do some more analysis later on today and be around on IRC.<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
On 12/10/11 09:20, Lake, Dan wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Thanks for the additional info on this, Melanie. You are correct that the resend should get a new sequence number and that old updates for an object will no longer get sent when there is a newer update to send. Properties and updates are
handled almost the same but different code. I'll look it over with Mic in the morning so I understand what is going on.<br>
<br>
~Dan<br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:opensim-dev-bounces@lists.berlios.de" target="_blank">opensim-dev-bounces@lists.berlios.de</a> [mailto:<a href="mailto:opensim-dev-bounces@lists.berlios.de" target="_blank">opensim-dev-bounces@lists.berlios.de</a>] On Behalf Of Melanie<br>
Sent: Tuesday, October 11, 2011 11:22 PM<br>
To: <a href="mailto:opensim-dev@lists.berlios.de" target="_blank">opensim-dev@lists.berlios.de</a><br>
Subject: Re: [Opensim-dev] Continual object update resends if acks are missed<br>
<br>
It appears this mechanism has already been removed for prim updates<br>
because intermediate updates are needed fro proper motion of<br>
physical prims, but was left in for property updates. However,<br>
culling of later updates as well as preservation of sequence numbers<br>
was not done.<br>
<br>
Melanie<br>
<br>
On 12/10/2011 07:43, Mic Bowman wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">We should collect more information on what is actually happening. The old<br>
behavior was to resend infinitely (and very badly). It would send old<br>
updates even when there were new updates for the same object. We changed the<br>
resend to transmit the current data rather than old data. If there are still<br>
no acks coming back after some time then we need to figure out what the<o:p></o:p></p>
</div>
<p class="MsoNormal">viewer is expecting& when it no longer acks packets.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
There is a write up on the procedure in an old email message on this list.<br>
<br>
--mic<br>
<br>
<br>
On Tue, Oct 11, 2011 at 3:55 PM, Justin Clark-Casey<<br>
<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi dslake (since this is chiefly addressed to you :)<br>
<br>
In commit b5ab33b5 back on Wednesday 20th April 2011, you made a change so<br>
that the resend of object updates or property updates is threaded back<br>
through Resend methods on LLClientView rather than the normal resend within<br>
UDPServer.<br>
<br>
The normal resend only performs the resend once, while going through the<br>
LLClientView.Resent*() methods will continually attempt the resend until<br>
acked as they put the requests back on the m_entityUpdates/m_entityProps<br>
queue.<br>
<br>
From my experience, often the client will not reack such packets. This<br>
means that they are continuously resent until the client logs out. I can<br>
see this happening by uncommenting the log lines in LLClientView.**ResendPrimUpdates()<br>
and ResentPropertyUpdates(). This is chiefly seen on remote regions (I<br>
can't repro on a localhost).<br>
<br>
What do you think? Can we resend such packets just once?<br>
<br>
Thanks,<br>
<br>
--<br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
______________________________**_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/**mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/**mailman/listinfo/opensim-dev</a><<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a>><o:p></o:p></p>
<p class="MsoNormal"><br>
<br>
<br>
_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><br>
_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><br>
<br>
-- <br>
Justin Clark-Casey (justincc)<br>
<a href="http://justincc.org/blog" target="_blank">http://justincc.org/blog</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@lists.berlios.de" target="_blank">Opensim-dev@lists.berlios.de</a><br>
<a href="https://lists.berlios.de/mailman/listinfo/opensim-dev" target="_blank">https://lists.berlios.de/mailman/listinfo/opensim-dev</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>