[Opensim-dev] UDP async performance

[Michael Heilmann]

Opensim Devs

I have been working on an external process that I hope to link to an 
opensim plugin I am authoring.  As a sanity check, I ran a simple socket 
test against the components to test for obvious problems before I get to 
the heavy lifting.  This test was not meant to reveal anything, just to 
confirm that the pieces are in place. However, I am not seeing the 
expected behavior.

My simple sanity test:  send 1000000 small udp packets at a server, and 
have it count packets received.  This is trivial, and the sending client 
is python, so all packets should be received.

While running this on localhost: (~160K packets per second)
> Go, c++, nodejs, c# synchronous : negligible to no packet loss, as 
> expected
>
> c# async: 80% packet loss


I have reviewed msdn documentation, and went through the opensim code 
(as it uses c# async) to clean up and run my tests.  That 80% packet 
loss is after forcing my threadpool and iocp thread counts up, and 
giving c# async extra time before and after the test to warm up 
threadpools and process any queues.

I have read that in some cases C# and .Net has a packet loss bug for udp 
on localhost.

I re-ran this test between two linux servers with a 10gbe LAN 
interconnect to remove any .net localhost packet loss issues, with the 
following results: (~115K packets per second)
> c# synchronous: ~998K messages received (average over 3 runs)
> Go : ~980K messages received (average over 3 runs)
>
> c# async: ~40K messages received (average over 3 runs)
disclosure: Go on my workstation is version 1.4.2, but on the servers is 
version 1.2.1.  Mono versions are identical

Now, the 3 runs were very nearly identical, with differences in the 
thousands of packets.  As before, C# async was given extra treatment to 
help it along.

I am on linux 64 bit, so I had a coworker write his own version of this 
test using Visual Studio 2013 on Windows (no code sharing), and he saw 
the same behavior:  c# async suffering massive packet loss while c# sync 
is keeping up easily.

Is there anything in the opensim client stack that I am missing, that 
allows for higher async performance?  Or am I on a witch hunt, and this 
performance is within Opensim bounds?  I have been increasing the 
threadpool and iocp threads as Opensim does.  I have pasted important 
pieces of the code below, as the entirety could be too long for this 
messageboard: (The UDPPacketBuffer class is coped from opensim code)

>         private void AsyncBeginReceive()
>         {
>             UDPPacketBuffer buf = new UDPPacketBuffer();
>             this.u.BeginReceiveFrom(
>                 buf.Data,
>                 0,
>                 UDPPacketBuffer.BUFFER_SIZE,
>                 SocketFlags.None,
>                 ref buf.RemoteEndPoint,
>                 AsyncEndReceive,
>                 buf);
>         }
>
>         private void AsyncEndReceive(IAsyncResult iar)
>         {
>             //schedule another receive
>             AsyncBeginReceive();
>
>             lock(this)
>             {
>                 this.i++;
>              }
>         }
>
>         public static void Main (string[] args)
>         {
>             int iocpThreads;
>             int workerThreads;
>             ThreadPool.GetMinThreads(out workerThreads, out iocpThreads);
>             workerThreads = 32;
>             iocpThreads = 32;
>             Console.WriteLine(workerThreads);
>             ThreadPool.SetMinThreads(workerThreads,iocpThreads);
>             MainClass mc = new MainClass();
>             mc.AsyncBeginReceive();
>             //manually trigger packet report after test run completion
>             Console.ReadLine();
>             Console.WriteLine(mc.i);
>         }

I have run this test repeatedly, varying workerThread and iocpThreads 
from 8, to 16, 32, 64 ... 512, without any change in packets received.  
If anyone had any insight into why this is suffering from so much packet 
loss, I would appreciate it.  Thanks.

-- 
Michael Heilmann
Research Associate
Institute for Simulation and Training
University of Central Florida