<div dir="ltr">Hi Michael!<div><br></div><div>Pulling out what should already be http over tcp based (textures & inventory & profiles & ...)...</div><div><br></div><div>Just to be clear... talking very specifically about object updates only (though terrain updates are similar in nature), you DO NOT want TCP if your error rates are over 1%. you'll waste a huge amount of bw with network-level retransmission of old, out-dated updates. Its specifically in situations where you have high update rates that you want to send current object state on re-transmission rather than re-transmitting an old packet that contains an update that has already been superceded.</div><div><br></div><div>That being said... the place where UDP is the wrong protocol is the initial scene load. It looks a lot like a bunch of one-time object update packets where the state of the object does not change. </div><div><br></div><div>The approach I spent some time looking at was to create a local scene proxy... run an opensim instance on your local client that you connect to once... then that instance mirrors remote scenes (kind of like the dynamic scene demonstration at oscc except that the scenes are coming from existing scenes rather than an archive). The proxy idea was inspired by the architecture that Stefan Anderson used years ago in their opensim service (I can't remember what it was called).</div><div><br></div><div>The proxy design assumes that making major protocol changes to the viewer is really hard & expensive, and that local accesses are fast and highly reliable. The proxy would enable custom protocols for connection to the remote scene which would be a lot easier to optimize (e.g. bulk transfer of an oar-like file for initial scene load). The proxy idea got far enough to show that it could work for a "consumption" experience... but it would take a lot of work to hook up enough events to make it completely transparent. </div><div><br></div><div>--mic</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 14, 2014 at 9:11 AM, Michael Heilmann <span dir="ltr"><<a href="mailto:mheilman@ist.ucf.edu" target="_blank">mheilman@ist.ucf.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
I agree completely with the fact that many packets are
time-sensitive and expire quickly, which is why I was looking for a
TCP/UDP pair, not dropping UDP in favour of TCP. I am concerned,
however, that UDP is the only mechanism for asynchronously sending
information to the client that may be important, and not expire
quickly.<br>
<br>
We regularly deal with users who have less then reliable networks,
such as cellular wifi access points at remote locations, and cannot
assume that our packet loss will be below 1%.<br>
<pre cols="72"><span class="">--
Michael Heilmann
Research Associate
Institute for Simulation and Training
University of Central Florida
</span><blockquote type="cite"><pre>Date: Fri, 14 Nov 2014 08:30:07 -0800
From: Mic Bowman <a href="mailto:cmickeyb@gmail.com" target="_blank"><cmickeyb@gmail.com></a>
To: <a href="mailto:opensim-dev@opensimulator.org" target="_blank">opensim-dev@opensimulator.org</a>
Subject: Re: [Opensim-dev] Modifying the networking stack
Message-ID:
<a href="mailto:CAJaF1_HXXSTFj2KhxDXH4-f_8aSYnWon+QON-NWUn1DommLh4A@mail.gmail.com" target="_blank"><CAJaF1_HXXSTFj2KhxDXH4-f_8aSYnWon+QON-NWUn1DommLh4A@mail.gmail.com></a>
Content-Type: text/plain; charset="utf-8"
having done a lot of the token bucket & retransmission work in opensim... i
started with a strong belief that TCP & reliable streams were the right way
to do object updates (for all bulk content like asset transfers, tcp, or
http layered on tcp, is definitely the right way). however... what we found
is that object updates should not be retransmitted at the packet level
which is what you get from tcp (that is... tcp handles retransmission of
dropped packets not opensim). the reason is pretty simple... dynamic
objects change position repeatedly (even if only in short bursts).
retransmission at the packet level would almost always retransmit <b><span>*</span>OLD<span>*</span></b><div><div class="h5">
state. for any kind of connection (this was often a problem for
international connections) where packet drops were even as high as 1%, we
would see multiple updates for the same object queued to send. only the
last one has any meaning.
so... while the situations where application level retransmission are rare
(<b><span>*</span>very<span>*</span></b> rare)... object updates happens to be one of them. moving
reliability into a lower level (either layering some protocol on tcp or os
implemented reliable udp) will result in lower efficiency for faulty
connections (and if your connection isnt faulty then udp works just fine).
--mic
On Fri, Nov 14, 2014 at 7:17 AM, Amit Goel <a href="mailto:Amit.Goel@ucf.edu" target="_blank"><Amit.Goel@ucf.edu></a> wrote:
</div></div></pre><div><div class="h5"><blockquote type="cite" style="color:#000000"><pre>Agree with Michael here that having TCP is better then using UDP with your
own homegrown TCP implementation on top of it in application software.
How about Reliable UDP :
<a href="http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Reliable-UDP-%28RUDP%29-The-Next-Big-Streaming-Protocol-85316.aspx" target="_blank">http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Reliable-UDP-(RUDP)-The-Next-Big-Streaming-Protocol-85316.aspx</a>
<a href="http://www.javvin.com/protocolRUDP.html" target="_blank">http://www.javvin.com/protocolRUDP.html</a>
I have not studied it in detail that how it fits in between lossy UDP and
time-consuming TCP.
Regards
-- amit
________________________________________
From: <a href="mailto:opensim-dev-bounces@opensimulator.org" target="_blank">opensim-dev-bounces@opensimulator.org</a> [
<a href="mailto:opensim-dev-bounces@opensimulator.org" target="_blank">opensim-dev-bounces@opensimulator.org</a>] on behalf of Michael Heilmann [
<a href="mailto:mheilman@ist.ucf.edu" target="_blank">mheilman@ist.ucf.edu</a>]
Sent: Friday, November 14, 2014 9:23 AM
To: <a href="mailto:opensim-dev@opensimulator.org" target="_blank">opensim-dev@opensimulator.org</a>
Subject: Re: [Opensim-dev] Modifying the networking stack
Thanks for the responses. I'll go into a little more detail:
We have been running several profilers against OpenSimulator on the
MOSES grid, and on my development machine. The tests were to examine
the loading on the server under several different loads, specifically
mesh and physics loads. What we found appears to be that no matter what
kind of load we placed on the region, even to the point of becoming
unresponsive due to physics and mesh, that scripting and physics load
were nowhere near the amount of time spent in
OpenSim.Region.ClientStack.LindenUDP once we had more than one or two
avatars logged in. We know from previous investigations at our firewall
that network traffic for OpenSim is not that heavy, especially with low
numbers of users.
I ran several Wireshark captures against a Firestorm viewer logging into
the MOSES public grid ABWIS region, where we hold our office hours. I
saw that with our current configuration, all traffic between the server
and my client, with the exception of http CAPS and fsapi calls, were UDP
traffic. This is not immediately concerning, as we have simian serve
our mesh and textures directly. The messages are mostly binary
information, so I could not examine closely, but I did see a lot of
messages containing identical ASCII strings, such as the name of my avatar.
My primary concern is the amount of time spent handling networking, not
necessarily the networking its-self. But there is at least a portion of
messages on the UDP pipeline that are either reliable, or perhaps should
be; and re-implementing a reliable transport over udp introduces load at
the application layer, instead of letting a low-level reliable transport
such as tcp handle it. I went to university with a guy who implemented
a java networking library completely over UDP, believing that it was
faster than a normal TCP socket; but he was neglecting that the
networking hardware handles the ACK and retransmission transparently,
and without needing for the messages to be handled manually by the
application.
This may just be my opinion, but since I was going to be ecamining the
network stack anyways, and typically in a client-server scenario the
ability to maintain a persistent reliable connection where the server
can push important events to the client, that it would be a good idea.
The points about network throttling and QoS are taken, but wouldn't they
also typically affect the UDP stream? Working on MOSES I have plenty of
problems dealing with external users who operate on restricted networks,
and they cannot see traffic aside from 80 and 443 without dealing with
their own IT personnel. The fact that it is HTTP over TCP instead of
raw TCP makes no difference once it is on a non-standard HTTP port.
I agree that it would be more prudent to look at improving the websocket
code and the http server, rather than replace it with a raw TCP socket,
especially given that there are multiple plugins, such as jsonsimstats,
that use the http functionality directly.
I hope that explains my position a little better. I would love to hear
if there are other plans/ideas in the community to address time-sinks
like this one, networking simply appears to us as a good starting point
to increase performance and scalability of the system.
--
Michael Heilmann
Research Associate
Institute for Simulation and Training
University of Central Florida
</pre><blockquote type="cite" style="color:#000000"><pre>Date: Thu, 13 Nov 2014 13:50:32 -0800
From: Diva Canto<a href="mailto:diva@metaverseink.com" target="_blank"><diva@metaverseink.com></a>
<a href="mailto:To:opensim-dev@opensimulator.org" target="_blank">To:opensim-dev@opensimulator.org</a>
Subject: Re: [Opensim-dev] Modifying the networking stack
Message-ID:<a href="mailto:546527A8.3040909@metaverseink.com" target="_blank"><546527A8.3040909@metaverseink.com></a>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
What problem are you trying to solve? It's hard to comment without
knowing what you want to achieve, but here are some random observations
that you may want to take into account.
As far as I remember, the reliable packets are a very small percentage
of the UDP traffic at this point, so I'm not sure it's worth creating a
dedicated TCP channel for them. Moving the HTTP traffic to the same TCP
connection seems like a bad idea, as the HTTP traffic tends to be
dominated by big data (textures, assets, etc) which would then get in
the way of the small packets like AgentOnline, etc. I suspect it would
make the client less responsive than what it is now.
There is already support for WebSockets in OpenSim. It may not be
complete, so I would encourage you to build on that. I am aware of WebGL
clients that use WebSockets with OpenSim, and they have the same problem
as described above: the big data gets in the way of the small packets,
making the clients less responsive at points. But since WebGL is
inevitable, your effort is probably best invested in this than in a TCP
channel.
Best,
Diva
On 11/13/2014 1:18 PM, Michael Heilmann wrote:
</pre><blockquote type="cite" style="color:#000000"><pre>Greetings everyone
I and another MOSES developer are going to be looking at the
client/server network stack, as well as the processing queue's used
for incoming and outgoing packets. I am going to see if I can
implement a client stack on opensim and firestorm that uses the
traditional TCP/UDP pairing for this type of client<->server
relationship. I have two thoughts, but I am interested in hearing if
you have ideas or insight into this particular space.
Idea 1:
Add a dedicated tcp port next to the UDP port, and move reliable
transport transmissions to the tcp port. I am uncomfortable
increasing the required ports for each region, but the http server is
in the way. I can look to move all communications from http to a tcp
socket-server type of deployment, at the expense of simple POST/GET
operations
idea 2:
Look into increasing the performance of the http server of the
regions, as well as testing/implementing a full websockets
implementation, and using the websockets upgrade for consistent client
connections. This could eventually lead to javascript-based clients,
and does not remove http functionality.
Either idea would see any traffic requiring reliable transport shifted
off of the current UDP stack, and onto the tcp reliable transport.
Either idea also will require modifications to a client to match. I,
and another developer here, would be developing the client code, the
region code, and testing against a MOSES deployment. As we are MOSES
developers, we would be working against simian instead of Robust; so
there would be a gap for regular Robust-based grids.
If you could lend me your opinions about these ideas, the management
queues, associated problems in opensim, etc, I would really appreciate
it. We would be working completely in the open on github, and obeying
all licensing. We would welcome any and all cooperation, and we will
cooperate ourselves wherever we are welcome, but we are not interested
in avoiding positive changes to maintain SecondLife compatibility.
Thanks.
</pre></blockquote></blockquote><pre>_______________________________________________
Opensim-dev mailing list
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a>
<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a>
_______________________________________________
Opensim-dev mailing list
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a>
<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a>
</pre></blockquote></div></div></blockquote>
</pre>
</div>
<br>_______________________________________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org">Opensim-dev@opensimulator.org</a><br>
<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a><br>
<br></blockquote></div><br></div>