[Opensim-dev] Modifying the networking stack

[Dahlia Trimble]

I wrote a Unity3D based client a few years back and I wrote a custom client
stack for it. The client stack implemented the IClientAPI interface as it
was designed with the intent of adding new protocols and client types to
OpenSimulator. I did not originally use UDP, rather I initially used all
HTTP and since WebSockets did not exist at the time, I used a long-poll
mechanism for synchronizing real-time state between the simulator and the
Unity clients. This worked remarkably well on the server side however the
Unity clients would have reduced HTTP performance as the scene complexity
would increase and the result was a lot of lag in prim and texture heavy
regions. I eventually added a UDP transport for real-time updates which
solved that problem. Another problem I had was since the client stack
module was not committed to core, it would not be updated as IClientAPI
would change in core and this happened quite frequently. I had a difficult
time maintaining compatibility with several versions of IClientAPI and I
had to publish and maintain many versions of the client stack module.

Since then I've developed other viewers which use custom protocols and I've
avoided IClientAPI altogether and implemented most of the additional
protocols in region modules. The region module interfaces are much simpler
and more stable and this has made coding and maintenance a lot easier.
There have been times where the region module interfaces would not expose
the required functinality for some feature but when this happened I would
modify the interfaces or messaging systems and commit the modifications to
core OpenSimulator where they would be available to all.

Regarding websockets, I've used the websocket implementation that is
currently in core OpenSimulator but I've had trouble with connections
randomly dying over long distance links. All works well in local testing
but the problem usually pops up for overseas or cross-continent links with
several in-between hops. I've since used the Fleck library with much better
reliability. I personally did not try to debug the built-in websocket
implementation and I don't really have any insight into what may be failing.

I don't think I'd recommend replacing any of the current stacks for use
with LL-based viewers. These protocols have been refined over several years
of use in SL and to a lesser extent, in OpenSimulator. If you are seeing
some issues with certain packets then maybe there is another bug or race
condition between simulator and viewer processes which should be addressed
rather than just replacing a network stack. Network protocols are difficult
to design and test across a wide variaty of installations and what may seem
a simple solution such as "we'll just replace that part with TCP" will end
up being a rather large amount of work and will likely fail in unforeseen
ways and may not even result in a usable solution at all.

There is already an alternative means of reliable message delivery in place
called EventQueueGet. While a lot of the code that uses it in OpenSimulator
may be somewhat haphazardly written and difficult to comprehend. it was an
attempt by LL to provide a alternative means of extending the reliable
messaging protocol over TCP and many messages were migrated to it. It also
uses the existing HTTP server and port in OpenSimulator for these messages.
I'd recommend investigating using this method before attempting a new
protocol design.

On Thu, Nov 13, 2014 at 1:18 PM, Michael Heilmann <mheilman at ist.ucf.edu>
wrote:

> Greetings everyone
>
> I and another MOSES developer are going to be looking at the client/server
> network stack, as well as the processing queue's used for incoming and
> outgoing packets.  I am going to see if I can implement a client stack on
> opensim and firestorm that uses the traditional TCP/UDP pairing for this
> type of client<->server relationship.  I have two thoughts, but I am
> interested in hearing if you have ideas or insight into this particular
> space.
>
> Idea 1:
>     Add a dedicated tcp port next to the UDP port, and move reliable
> transport transmissions to the tcp port.  I am uncomfortable increasing the
> required ports for each region, but the http server is in the way.  I can
> look to move all communications from http to a tcp socket-server type of
> deployment, at the expense of simple POST/GET operations
>
> idea 2:
>     Look into increasing the performance of the http server of the
> regions, as well as testing/implementing a full websockets implementation,
> and using the websockets upgrade for consistent client connections.  This
> could eventually lead to javascript-based clients, and does not remove http
> functionality.
>
> Either idea would see any traffic requiring reliable transport shifted off
> of the current UDP stack, and onto the tcp reliable transport.  Either idea
> also will require modifications to a client to match.  I, and another
> developer here, would be developing the client code, the region code, and
> testing against a MOSES deployment.  As we are MOSES developers, we would
> be working against simian instead of Robust; so there would be a gap for
> regular Robust-based grids.
>
> If you could lend me your opinions about these ideas, the management
> queues, associated problems in opensim, etc, I would really appreciate it.
> We would be working completely in the open on github, and obeying all
> licensing.  We would welcome any and all cooperation, and we will cooperate
> ourselves wherever we are welcome, but we are not interested in avoiding
> positive changes to maintain SecondLife compatibility.
>
> Thanks.
>
> --
> Michael Heilmann
> Research Associate
> Institute for Simulation and Training
> University of Central Florida
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20141114/3fe36ff9/attachment.html>