[Opensim-dev] Modifying the networking stack

Fri Nov 14 14:23:03 UTC 2014

Thanks for the responses.  I'll go into a little more detail:

We have been running several profilers against OpenSimulator on the 
MOSES grid, and on my development machine.  The tests were to examine 
the loading on the server under several different loads, specifically 
mesh and physics loads.  What we found appears to be that no matter what 
kind of load we placed on the region, even to the point of becoming 
unresponsive due to physics and mesh, that scripting and physics load 
were nowhere near the amount of time spent in 
OpenSim.Region.ClientStack.LindenUDP once we had more than one or two 
avatars logged in.  We know from previous investigations at our firewall 
that network traffic for OpenSim is not that heavy, especially with low 
numbers of users.

I ran several Wireshark captures against a Firestorm viewer logging into 
the MOSES public grid ABWIS region, where we hold our office hours.  I 
saw that with our current configuration, all traffic between the server 
and my client, with the exception of http CAPS and fsapi calls, were UDP 
traffic.  This is not immediately concerning, as we have simian serve 
our mesh and textures directly. The messages are mostly binary 
information, so I could not examine closely, but I did see a lot of 
messages containing identical ASCII strings, such as the name of my avatar.

My primary concern is the amount of time spent handling networking, not 
necessarily the networking its-self.  But there is at least a portion of 
messages on the UDP pipeline that are either reliable, or perhaps should 
be; and re-implementing a reliable transport over udp introduces load at 
the application layer, instead of letting a low-level reliable transport 
such as tcp handle it.  I went to university with a guy who implemented 
a java networking library completely over UDP, believing that it was 
faster than a normal TCP socket; but he was neglecting that the 
networking hardware handles the ACK and retransmission transparently, 
and without needing for the messages to be handled manually by the 
application.

This may just be my opinion, but since I was going to be ecamining the 
network stack anyways, and typically in a client-server scenario the 
ability to maintain a persistent reliable connection where the server 
can push important events to the client, that it would be a good idea.  
The points about network throttling and QoS are taken, but wouldn't they 
also typically affect the UDP  stream? Working on MOSES I have plenty of 
problems dealing with external users who operate on restricted networks, 
and they cannot see traffic aside from 80 and 443 without dealing with 
their own IT personnel.  The fact that it is HTTP over TCP instead of 
raw TCP makes no difference once it is on a non-standard HTTP port.

I agree that it would be more prudent to look at improving the websocket 
code and the http server, rather than replace it with a raw TCP socket, 
especially given that there are multiple plugins, such as jsonsimstats, 
that use the http functionality directly.

I hope that explains my position a little better.  I would love to hear 
if there are other plans/ideas in the community to address time-sinks 
like this one, networking simply appears to us as a good starting point 
to increase performance and scalability of the system.

-- 
Michael Heilmann
Research Associate
Institute for Simulation and Training
University of Central Florida

> Date: Thu, 13 Nov 2014 13:50:32 -0800
> From: Diva Canto<diva at metaverseink.com>
> To:opensim-dev at opensimulator.org
> Subject: Re: [Opensim-dev] Modifying the networking stack
> Message-ID:<546527A8.3040909 at metaverseink.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> What problem are you trying to solve? It's hard to comment without
> knowing what you want to achieve, but here are some random observations
> that you may want to take into account.
>
> As far as I remember, the reliable packets are a very small percentage
> of the UDP traffic at this point, so I'm not sure it's worth creating a
> dedicated TCP channel for them. Moving the HTTP traffic to the same TCP
> connection seems like a bad idea, as the HTTP traffic tends to be
> dominated by big data (textures, assets, etc) which would then get in
> the way of the small packets like AgentOnline, etc. I suspect it would
> make the client less responsive than what it is now.
>
> There is already support for WebSockets in OpenSim. It may not be
> complete, so I would encourage you to build on that. I am aware of WebGL
> clients that use WebSockets with OpenSim, and they have the same problem
> as described above: the big data gets in the way of the small packets,
> making the clients less responsive at points. But since WebGL is
> inevitable, your effort is probably best invested in this than in a TCP
> channel.
>
> Best,
> Diva
>
> On 11/13/2014 1:18 PM, Michael Heilmann wrote:
>> Greetings everyone
>>
>> I and another MOSES developer are going to be looking at the
>> client/server network stack, as well as the processing queue's used
>> for incoming and outgoing packets.  I am going to see if I can
>> implement a client stack on opensim and firestorm that uses the
>> traditional TCP/UDP pairing for this type of client<->server
>> relationship.  I have two thoughts, but I am interested in hearing if
>> you have ideas or insight into this particular space.
>>
>> Idea 1:
>>      Add a dedicated tcp port next to the UDP port, and move reliable
>> transport transmissions to the tcp port.  I am uncomfortable
>> increasing the required ports for each region, but the http server is
>> in the way.  I can look to move all communications from http to a tcp
>> socket-server type of deployment, at the expense of simple POST/GET
>> operations
>>
>> idea 2:
>>      Look into increasing the performance of the http server of the
>> regions, as well as testing/implementing a full websockets
>> implementation, and using the websockets upgrade for consistent client
>> connections.  This could eventually lead to javascript-based clients,
>> and does not remove http functionality.
>>
>> Either idea would see any traffic requiring reliable transport shifted
>> off of the current UDP stack, and onto the tcp reliable transport.
>> Either idea also will require modifications to a client to match.  I,
>> and another developer here, would be developing the client code, the
>> region code, and testing against a MOSES deployment.  As we are MOSES
>> developers, we would be working against simian instead of Robust; so
>> there would be a gap for regular Robust-based grids.
>>
>> If you could lend me your opinions about these ideas, the management
>> queues, associated problems in opensim, etc, I would really appreciate
>> it.  We would be working completely in the open on github, and obeying
>> all licensing.  We would welcome any and all cooperation, and we will
>> cooperate ourselves wherever we are welcome, but we are not interested
>> in avoiding positive changes to maintain SecondLife compatibility.
>>
>> Thanks.
>>