[Opensim-dev] Continual object update resends if acks are missed
Justin Clark-Casey
jjustincc at googlemail.com
Mon Oct 17 21:34:42 UTC 2011
Okay, I've turned the setting on with git master 352002f. Let's see how it goes.
I prefer a ramp up because of the massive packet loss we always saw at the start of a connection.
I'm somewhat surprised that the client itself doesn't do this throttling - I thought that's why it was sending throttle
requests and adjustments in the first place.
On 12/10/11 19:08, Mic Bowman wrote:
> The adaptive throttle is basically a slow start mechanism. It starts with a low throttle rate & increases the rate until
> packets start to be dropped. At that point, it backs off. TCP slow start grows very slowly and cuts the rate very
> rapidly. The current configuration in the adaptive throttle starts a little faster & doesn't back off quite so much. I
> think you can use show pqueues and show throttles to get a feel for how quickly it adapts.
>
> The biggest issue I had from people who tested it (while it is experimental its been the default on scisim since the
> code went in originally) was the comment "hey, my network settings are 2Mbps but I'm only seeing 500kbps"... never mind
> that the reason is that their network connection couldn't handle 2Mbps... so... I'm a bit leary of the perception.
>
> One possible way to address that concern is to start with the adaptive throttle fully open & let it shrink back rather
> than start slowly and grow. I'm a bit concerned that once you get into a bad state, you'll have a hard time getting out
> & the initial scene download puts a lot of pressure on the network connection. Either way... I'm good with trying out
> some changes in the configuration.
>
> --mic
>
>
> On Wed, Oct 12, 2011 at 10:57 AM, Justin Clark-Casey <jjustincc at googlemail.com <mailto:jjustincc at googlemail.com>> wrote:
>
> You're right - the mass resend is consistently overloading the connection. Logging client acks shows that the
> resent packet bursts are not acked. Raising the default RTO only delays the resend since the client doesn't send
> acks no matter how long the wait.
>
> Setting enable_adaptive_throttles = true makes the problem go away for me - the client now receives the object
> update packets at a rate that it can cope with. Is there really any reason not to turn this on by default? I hear
> what you're saying about potential issues but it seems far better to get real world data to iron them out than
> persist with our current naive implementation. At the worst we can always just disable it again.
>
> Lowering the bw in the client settings also works, though it still allows a noticeable overload at the beginning of
> the client connection. I'm very surprised that the throttling isn't being properly adjusted automatically, or is
> this what enable_adaptive_throttles does (I traced it into the AdaptiveTokenBucket but no further yet)?
>
>
> On 12/10/11 16:32, Mic Bowman wrote:
>
> Couple things...
>
> First... here's the link to the thread on some of the original issues... not sure if this went in to dan's
> retransmission fix but i'll get to that in a minute:
> https://lists.berlios.de/__pipermail/opensim-dev/2011-__March/010029.html
> <https://lists.berlios.de/pipermail/opensim-dev/2011-March/010029.html>
>
> Second... In spite of all the questions/comments below that this is the "expected" behavior... there may be a
> bug & we
> need more data to find it.
>
> Question... do you have the adaptive throttling turned on? Look for enable_adaptive_throttles. It is turned off by
> default. That is the switch that is designed to fix the problem you are seeing... but the code has some other
> behaviors
> which are correct but... "different" than what most admins expect (the throttle mimics the TCP slow start to
> identify
> and avoid congestion though not quite as conservatively as the TCP algorithm).
>
> One explanation for what you are seeing is that the network simply cannot send as many packets as you are
> asking. This
> is frequently true for long-haul connections. When we did our analysis, we saw a "cliff". Once you pass the
> point where
> the number of outgoing packets exceeds the physical networks ability to deliver them ACKs are not processed in
> time and
> very bad things happen. That is... enough ACKs are coming late enough so that the resend packet is already in
> the queue.
> The current version of the code handles that situation far better than the old version (the current version is
> bound by
> the number of dynamic objects... the old version would hold *every* update to that object... even ones that were
> superceded by later updates).
>
> The only real solution to this problem is to stop sending updates so fast or to be much more conservative about
> resends.
> The first can be addressed one of three ways... First, configure the simulator so that the outgoing bw per
> connection is
> throttled much lower. Second, turn down the BW setting on the viewer. Third, turn on the adaptive throttling
> code in the
> simulator.
>
> Being conservative about resends has its own share of problems... Basically what this means is that you wait
> longer to
> resend a packet. However... if you wait too long to retransmit, then a portion of your scene is not updating. The
> algorithm for determining when to retransmit is based on a commonly used algorithm described in one of the RFCs (the
> actual RFC is referenced in a comment in the opensim code).
>
> As to the UDP packet vs going back through the SceneEntityUpdate queue... well that one is a no brainer... If
> you use
> the UDP packet for retranmission you are almost always sending useless old data and just filling up the network with
> fluff. Outside of the initial scene load, you see streams of updates for objects more often than a single
> update. If you
> retransmit the UDP packet, you are almost always sending old data. Going through the sceneentityupdate queue
> basically
> says "retransmit the current state of the object".
>
> --mic
>
>
> On Wed, Oct 12, 2011 at 5:56 AM, Justin Clark-Casey <jjustincc at googlemail.com <mailto:jjustincc at googlemail.com>
> <mailto:jjustincc at googlemail.__com <mailto:jjustincc at googlemail.com>>> wrote:
>
> Yes, I may not have the correct cause in my original e-mail but I'm certainly seeing a lot of resends that never
> appear to stop. The effect is not consistent - sometimes I can log in to the remote sim and there will be no
> continuous resend. But most of the time a large proportion of updates appear to be continuously resent.
>
> I'm surprised no-one else has reported it though I did see a lot of continuous updating on Wright Plaza @
> osgrid.org <http://osgrid.org>
> <http://osgrid.org> last night, which I don't think is related to changes made by scripts. Perhaps it's only really
>
> noticeable if you start looking, have visual object updates development/advanced settings on in a viewer or are
> wondering why viewer received packets keeps spiking.
>
> As far as I can see from LLClientView/LLUDPServer, both prim and property updates resends are happening but
> I need
> to study the code more closely and possibly old e-mails.
>
> I hope to do some more analysis later on today and be around on IRC.
>
>
> On 12/10/11 09:20, Lake, Dan wrote:
>
> Thanks for the additional info on this, Melanie. You are correct that the resend should get a new sequence
> number and that old updates for an object will no longer get sent when there is a newer update to send.
> Properties and updates are handled almost the same but different code. I'll look it over with Mic in the
> morning
> so I understand what is going on.
>
> ~Dan
>
>
> -----Original Message-----
> From: opensim-dev-bounces at lists.__be__rlios.de <http://berlios.de>
> <mailto:opensim-dev-bounces at __lists.berlios.de <mailto:opensim-dev-bounces at lists.berlios.de>>
> [mailto:opensim-dev-bounces@ <mailto:opensim-dev-bounces@>____lists.berlios.de <http://lists.berlios.de>
> <mailto:opensim-dev-bounces at __lists.berlios.de <mailto:opensim-dev-bounces at lists.berlios.de>>] On Behalf Of Melanie
>
> Sent: Tuesday, October 11, 2011 11:22 PM
> To: opensim-dev at lists.berlios.de <mailto:opensim-dev at lists.berlios.de>
> <mailto:opensim-dev at lists.__berlios.de <mailto:opensim-dev at lists.berlios.de>>
>
> Subject: Re: [Opensim-dev] Continual object update resends if acks are missed
>
> It appears this mechanism has already been removed for prim updates
> because intermediate updates are needed fro proper motion of
> physical prims, but was left in for property updates. However,
> culling of later updates as well as preservation of sequence numbers
> was not done.
>
> Melanie
>
> On 12/10/2011 07:43, Mic Bowman wrote:
>
> We should collect more information on what is actually happening. The old
> behavior was to resend infinitely (and very badly). It would send old
> updates even when there were new updates for the same object. We changed the
> resend to transmit the current data rather than old data. If there are still
> no acks coming back after some time then we need to figure out what the
> viewer is expecting& when it no longer acks packets.
>
>
> There is a write up on the procedure in an old email message on this list.
>
> --mic
>
>
> On Tue, Oct 11, 2011 at 3:55 PM, Justin Clark-Casey<
> jjustincc at googlemail.com <mailto:jjustincc at googlemail.com> <mailto:jjustincc at googlemail.__com
> <mailto:jjustincc at googlemail.com>>> wrote:
>
>
> Hi dslake (since this is chiefly addressed to you :)
>
> In commit b5ab33b5 back on Wednesday 20th April 2011, you made a change so
> that the resend of object updates or property updates is threaded back
> through Resend methods on LLClientView rather than the normal resend within
> UDPServer.
>
> The normal resend only performs the resend once, while going through the
> LLClientView.Resent*() methods will continually attempt the resend until
> acked as they put the requests back on the m_entityUpdates/m_entityProps
> queue.
>
> From my experience, often the client will not reack such packets. This
> means that they are continuously resent until the client logs out. I can
> see this happening by uncommenting the log lines in LLClientView.**____ResendPrimUpdates()
>
> and ResentPropertyUpdates(). This is chiefly seen on remote regions (I
> can't repro on a localhost).
>
> What do you think? Can we resend such packets just once?
>
> Thanks,
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc
> __________________________________**_________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de> <mailto:Opensim-dev at lists.__berlios.de
> <mailto:Opensim-dev at lists.berlios.de>>
> https://lists.berlios.de/**____mailman/listinfo/opensim-dev
> <https://lists.berlios.de/**__mailman/listinfo/opensim-dev>
> <https://lists.berlios.de/**__mailman/listinfo/opensim-dev
> <https://lists.berlios.de/**mailman/listinfo/opensim-dev>><__h__ttps://lists.berlios.de/____mailman/listinfo/opensim-dev
> <http://lists.berlios.de/__mailman/listinfo/opensim-dev>
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>>>
>
>
>
>
> ___________________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de> <mailto:Opensim-dev at lists.__berlios.de
> <mailto:Opensim-dev at lists.berlios.de>>
> https://lists.berlios.de/____mailman/listinfo/opensim-dev
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev>
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>>
>
> ___________________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de> <mailto:Opensim-dev at lists.__berlios.de
> <mailto:Opensim-dev at lists.berlios.de>>
> https://lists.berlios.de/____mailman/listinfo/opensim-dev
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev>
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>>
> ___________________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de> <mailto:Opensim-dev at lists.__berlios.de
> <mailto:Opensim-dev at lists.berlios.de>>
> https://lists.berlios.de/____mailman/listinfo/opensim-dev
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev>
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>>
>
>
>
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc
> ___________________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de> <mailto:Opensim-dev at lists.__berlios.de
> <mailto:Opensim-dev at lists.berlios.de>>
> https://lists.berlios.de/____mailman/listinfo/opensim-dev
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev>
> <https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>>
>
>
>
>
>
> _________________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
> https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>
>
>
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc
> _________________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
> https://lists.berlios.de/__mailman/listinfo/opensim-dev <https://lists.berlios.de/mailman/listinfo/opensim-dev>
>
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
More information about the Opensim-dev
mailing list