[Opensim-dev] Modifying the networking stack

Fri Nov 14 16:24:45 UTC 2014

If I recall correctly the problem we ran into was that the server had
kernel-desktop installed and because we have dual socket hexa xeon
processors for a total of 24 virtual cores, this kernel was not optimized
for this, switching the server to kernel-default I think resolved the
problem that mono was having.

On Fri, Nov 14, 2014 at 11:09 AM, Diva Canto <diva at metaverseink.com> wrote:

> Also, for debugging network issues with OpenSim, WinGridProxy (included in
> libomv) is much more appropriate than WireShark. You can see the content of
> all messages.
>
>
> On 11/14/2014 8:05 AM, Diva Canto wrote:
>
>> On 11/14/2014 6:23 AM, Michael Heilmann wrote:
>>
>>> Thanks for the responses.  I'll go into a little more detail:
>>>
>>> We have been running several profilers against OpenSimulator on the
>>> MOSES grid, and on my development machine.  The tests were to examine the
>>> loading on the server under several different loads, specifically mesh and
>>> physics loads.  What we found appears to be that no matter what kind of
>>> load we placed on the region, even to the point of becoming unresponsive
>>> due to physics and mesh, that scripting and physics load were nowhere near
>>> the amount of time spent in OpenSim.Region.ClientStack.LindenUDP once
>>> we had more than one or two avatars logged in.  We know from previous
>>> investigations at our firewall that network traffic for OpenSim is not that
>>> heavy, especially with low numbers of users.
>>>
>>
>> If this is a problem, and you are running a recent-ish version of core
>> OpenSim, it sounds like some misconfiguration somewhere. Back in the summer
>> of 2013 we had a problem with the server running OSCC'13; the kernel was
>> configured to run in some sort of special mode that was making everything
>> run badly and unpredictably. We fixed the kernel configuration, and
>> suddenly things started running much more smoothly-- I don't remember the
>> details, but Nebadon may clarify things.
>>
>> OpenSim these days can handle 50 people on a single simulator without
>> much trouble. If you look at figure 7 of my paper (
>> http://www.ics.uci.edu/~lopes/documents/summersim14/
>> gabrielova_lopes_preprint.pdf) you will see the quantification of
>> "without much trouble." I suggest that you reproduce my experimental
>> conditions with pCamBot and check whether your numbers are very different
>> from ours. If they are very different, then there's definitely something
>> odd in your setup, as we were able to reproduce these numbers in several
>> machines. Feel free to contact me directly for details about pCamBot
>> configuration.
>>
>> Bots aren't real viewers, but they are much better for measuring things
>> systematically and detecting problems and bottlenecks than relying on real
>> users driven by real people. The performance you get with pCamBot will be
>> correlated with the performance you get with real users.
>>
>>
>>  I ran several Wireshark captures against a Firestorm viewer logging into
>>> the MOSES public grid ABWIS region, where we hold our office hours.  I saw
>>> that with our current configuration, all traffic between the server and my
>>> client, with the exception of http CAPS and fsapi calls, were UDP traffic.
>>> This is not immediately concerning, as we have simian serve our mesh and
>>> textures directly. The messages are mostly binary information, so I could
>>> not examine closely, but I did see a lot of messages containing identical
>>> ASCII strings, such as the name of my avatar.
>>>
>>
>> Hard to say what you saw, but I bet those are the AgentUpdate messages
>> that I mentioned before. The viewer sends at least 10/sec. At points, the
>> viewer sends much more than 10/sec, up to 60/sec. Again, take a look at my
>> paper for understanding what those are, and how OpenSim deals with them
>> since OSCC'13.
>>
>> As I said before, it would be nice to understand why the viewer is so
>> eager to blabber its status to the server when nothing is going on.
>>
>>
>>  My primary concern is the amount of time spent handling networking, not
>>> necessarily the networking its-self.  But there is at least a portion of
>>> messages on the UDP pipeline that are either reliable, or perhaps should
>>> be; and re-implementing a reliable transport over udp introduces load at
>>> the application layer, instead of letting a low-level reliable transport
>>> such as tcp handle it.  I went to university with a guy who implemented a
>>> java networking library completely over UDP, believing that it was faster
>>> than a normal TCP socket; but he was neglecting that the networking
>>> hardware handles the ACK and retransmission transparently, and without
>>> needing for the messages to be handled manually by the application.
>>>
>>> This may just be my opinion, but since I was going to be ecamining the
>>> network stack anyways, and typically in a client-server scenario the
>>> ability to maintain a persistent reliable connection where the server can
>>> push important events to the client, that it would be a good idea.  The
>>> points about network throttling and QoS are taken, but wouldn't they also
>>> typically affect the UDP stream? Working on MOSES I have plenty of problems
>>> dealing with external users who operate on restricted networks, and they
>>> cannot see traffic aside from 80 and 443 without dealing with their own IT
>>> personnel.  The fact that it is HTTP over TCP instead of raw TCP makes no
>>> difference once it is on a non-standard HTTP port.
>>>
>>> I agree that it would be more prudent to look at improving the websocket
>>> code and the http server, rather than replace it with a raw TCP socket,
>>> especially given that there are multiple plugins, such as jsonsimstats,
>>> that use the http functionality directly.
>>>
>>> I hope that explains my position a little better.  I would love to hear
>>> if there are other plans/ideas in the community to address time-sinks like
>>> this one, networking simply appears to us as a good starting point to
>>> increase performance and scalability of the system.
>>>
>>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>>
>>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>

-- 
Michael Emory Cerquoni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20141114/9cd43b42/attachment-0001.html>