[Opensim-dev] UDP async performance

[Myron Curtis]

Good point. I will explore that more.

 

From: opensim-dev-bounces at opensimulator.org [mailto:opensim-dev-bounces at opensimulator.org] On Behalf Of Michael Emory Cerquoni
Sent: Monday, March 16, 2015 10:13 AM
To: opensim-dev at opensimulator.org
Subject: Re: [Opensim-dev] UDP async performance

 

I am not sure the HTTP server handles UDP, I want to say it does not handle that.

 

On Mon, Mar 16, 2015 at 11:50 AM, Myron Curtis <myronjc at virtualworldsgrid.com <mailto:myronjc at virtualworldsgrid.com> > wrote:

Could this be a case of the Http server in opensim just being too slow to
respond to this traffic quickly enough?


-----Original Message-----
From: opensim-dev-bounces at opensimulator.org <mailto:opensim-dev-bounces at opensimulator.org> 
[mailto:opensim-dev-bounces at opensimulator.org <mailto:opensim-dev-bounces at opensimulator.org> ] On Behalf Of Michael Heilmann
Sent: Monday, March 16, 2015 6:55 AM
To: opensim-dev at opensimulator.org <mailto:opensim-dev at opensimulator.org> 
Subject: [Opensim-dev] UDP async performance

Opensim Devs

I have been working on an external process that I hope to link to an opensim
plugin I am authoring.  As a sanity check, I ran a simple socket test
against the components to test for obvious problems before I get to the
heavy lifting.  This test was not meant to reveal anything, just to confirm
that the pieces are in place. However, I am not seeing the expected
behavior.

My simple sanity test:  send 1000000 small udp packets at a server, and have
it count packets received.  This is trivial, and the sending client is
python, so all packets should be received.

While running this on localhost: (~160K packets per second)
> Go, c++, nodejs, c# synchronous : negligible to no packet loss, as
> expected
>
> c# async: 80% packet loss


I have reviewed msdn documentation, and went through the opensim code (as it
uses c# async) to clean up and run my tests.  That 80% packet loss is after
forcing my threadpool and iocp thread counts up, and giving c# async extra
time before and after the test to warm up threadpools and process any
queues.

I have read that in some cases C# and .Net has a packet loss bug for udp on
localhost.

I re-ran this test between two linux servers with a 10gbe LAN interconnect
to remove any .net localhost packet loss issues, with the following results:
(~115K packets per second)
> c# synchronous: ~998K messages received (average over 3 runs) Go :
> ~980K messages received (average over 3 runs)
>
> c# async: ~40K messages received (average over 3 runs)
disclosure: Go on my workstation is version 1.4.2, but on the servers is
version 1.2.1.  Mono versions are identical

Now, the 3 runs were very nearly identical, with differences in the
thousands of packets.  As before, C# async was given extra treatment to help
it along.

I am on linux 64 bit, so I had a coworker write his own version of this test
using Visual Studio 2013 on Windows (no code sharing), and he saw the same
behavior:  c# async suffering massive packet loss while c# sync is keeping
up easily.

Is there anything in the opensim client stack that I am missing, that allows
for higher async performance?  Or am I on a witch hunt, and this performance
is within Opensim bounds?  I have been increasing the threadpool and iocp
threads as Opensim does.  I have pasted important pieces of the code below,
as the entirety could be too long for this
messageboard: (The UDPPacketBuffer class is coped from opensim code)

>         private void AsyncBeginReceive()
>         {
>             UDPPacketBuffer buf = new UDPPacketBuffer();
>             this.u.BeginReceiveFrom(
>                 buf.Data,
>                 0,
>                 UDPPacketBuffer.BUFFER_SIZE,
>                 SocketFlags.None,
>                 ref buf.RemoteEndPoint,
>                 AsyncEndReceive,
>                 buf);
>         }
>
>         private void AsyncEndReceive(IAsyncResult iar)
>         {
>             //schedule another receive
>             AsyncBeginReceive();
>
>             lock(this)
>             {
>                 this.i++;
>              }
>         }
>
>         public static void Main (string[] args)
>         {
>             int iocpThreads;
>             int workerThreads;
>             ThreadPool.GetMinThreads(out workerThreads, out iocpThreads);
>             workerThreads = 32;
>             iocpThreads = 32;
>             Console.WriteLine(workerThreads);
>             ThreadPool.SetMinThreads(workerThreads,iocpThreads);
>             MainClass mc = new MainClass();
>             mc.AsyncBeginReceive();
>             //manually trigger packet report after test run completion
>             Console.ReadLine();
>             Console.WriteLine(mc.i);
>         }

I have run this test repeatedly, varying workerThread and iocpThreads from
8, to 16, 32, 64 ... 512, without any change in packets received.
If anyone had any insight into why this is suffering from so much packet
loss, I would appreciate it.  Thanks.

--
Michael Heilmann
Research Associate
Institute for Simulation and Training
University of Central Florida

_______________________________________________
Opensim-dev mailing list
Opensim-dev at opensimulator.org <mailto:Opensim-dev at opensimulator.org> 
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev

_______________________________________________
Opensim-dev mailing list
Opensim-dev at opensimulator.org <mailto:Opensim-dev at opensimulator.org> 
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev




-- 

Michael Emory Cerquoni

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20150316/68de563a/attachment-0001.html>