<div dir="ltr">Some information which may be useful for those implementing protocols over UDP: <a href="https://tools.ietf.org/html/rfc5405">https://tools.ietf.org/html/rfc5405</a><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 16, 2015 at 10:46 AM, Dahlia Trimble <span dir="ltr"><<a href="mailto:dahliatrimble@gmail.com" target="_blank">dahliatrimble@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm not sure where the loss occurs but I've seen similar behavior in other network layer implementations based on UDP. You can only send so much before the receiving end sees packet loss and this makes sense as there is no (or very little) buffering at the receiving end to store unprocessed packets. I believe what most implementations do is throttle the packets at the sending end to try to minimuze this loss. I'm not sure why you see async showing more loss than other patterns but I have also seen loss with select() polling as well. I believe viewers typically don't send packets at such a high rate that such loss becomes a problem, and also any packets sent by the viewer at any "high" rate are assumed to be unreliable anyway and some loss is acceptable. OpenSImulator also throttles UDP packets it sends to viewers to try to mitigate packet loss. If you were using TCP instead, it will do this for you behind the scene so if you try to send too much data down a TCP socket it will just fill buffers and/or eventually block until it can successfully send the data.<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 16, 2015 at 6:54 AM, Michael Heilmann <span dir="ltr"><<a href="mailto:mheilman@ist.ucf.edu" target="_blank">mheilman@ist.ucf.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Opensim Devs<br>
<br>
I have been working on an external process that I hope to link to an opensim plugin I am authoring. As a sanity check, I ran a simple socket test against the components to test for obvious problems before I get to the heavy lifting. This test was not meant to reveal anything, just to confirm that the pieces are in place. However, I am not seeing the expected behavior.<br>
<br>
My simple sanity test: send 1000000 small udp packets at a server, and have it count packets received. This is trivial, and the sending client is python, so all packets should be received.<br>
<br>
While running this on localhost: (~160K packets per second)<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Go, c++, nodejs, c# synchronous : negligible to no packet loss, as expected<br>
<br>
c# async: 80% packet loss<br>
</blockquote>
<br>
<br>
I have reviewed msdn documentation, and went through the opensim code (as it uses c# async) to clean up and run my tests. That 80% packet loss is after forcing my threadpool and iocp thread counts up, and giving c# async extra time before and after the test to warm up threadpools and process any queues.<br>
<br>
I have read that in some cases C# and .Net has a packet loss bug for udp on localhost.<br>
<br>
I re-ran this test between two linux servers with a 10gbe LAN interconnect to remove any .net localhost packet loss issues, with the following results: (~115K packets per second)<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
c# synchronous: ~998K messages received (average over 3 runs)<br>
Go : ~980K messages received (average over 3 runs)<br>
<br>
c# async: ~40K messages received (average over 3 runs)<br>
</blockquote>
disclosure: Go on my workstation is version 1.4.2, but on the servers is version 1.2.1. Mono versions are identical<br>
<br>
Now, the 3 runs were very nearly identical, with differences in the thousands of packets. As before, C# async was given extra treatment to help it along.<br>
<br>
I am on linux 64 bit, so I had a coworker write his own version of this test using Visual Studio 2013 on Windows (no code sharing), and he saw the same behavior: c# async suffering massive packet loss while c# sync is keeping up easily.<br>
<br>
Is there anything in the opensim client stack that I am missing, that allows for higher async performance? Or am I on a witch hunt, and this performance is within Opensim bounds? I have been increasing the threadpool and iocp threads as Opensim does. I have pasted important pieces of the code below, as the entirety could be too long for this messageboard: (The UDPPacketBuffer class is coped from opensim code)<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
private void AsyncBeginReceive()<br>
{<br>
UDPPacketBuffer buf = new UDPPacketBuffer();<br>
this.u.BeginReceiveFrom(<br>
buf.Data,<br>
0,<br>
UDPPacketBuffer.BUFFER_SIZE,<br>
SocketFlags.None,<br>
ref buf.RemoteEndPoint,<br>
AsyncEndReceive,<br>
buf);<br>
}<br>
<br>
private void AsyncEndReceive(IAsyncResult iar)<br>
{<br>
//schedule another receive<br>
AsyncBeginReceive();<br>
<br>
lock(this)<br>
{<br>
this.i++;<br>
}<br>
}<br>
<br>
public static void Main (string[] args)<br>
{<br>
int iocpThreads;<br>
int workerThreads;<br>
ThreadPool.GetMinThreads(out workerThreads, out iocpThreads);<br>
workerThreads = 32;<br>
iocpThreads = 32;<br>
Console.WriteLine(<u></u>workerThreads);<br>
ThreadPool.SetMinThreads(<u></u>workerThreads,iocpThreads);<br>
MainClass mc = new MainClass();<br>
mc.AsyncBeginReceive();<br>
//manually trigger packet report after test run completion<br>
Console.ReadLine();<br>
Console.WriteLine(mc.i);<br>
}<br>
</blockquote>
<br>
I have run this test repeatedly, varying workerThread and iocpThreads from 8, to 16, 32, 64 ... 512, without any change in packets received. If anyone had any insight into why this is suffering from so much packet loss, I would appreciate it. Thanks.<span><font color="#888888"><br>
<br>
-- <br>
Michael Heilmann<br>
Research Associate<br>
Institute for Simulation and Training<br>
University of Central Florida<br>
<br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a><br>
<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-<u></u>bin/mailman/listinfo/opensim-<u></u>dev</a><br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>