[Opensim-dev] Synchronous or Asynchronous.. Packet handler synchronicity

[Teravus Ovares]

Hello Again

I just wanted to update the people subscribed to opensim-dev about the
current situation with regards to packet handler synchronicity and
thread context.

The History:
Shortly after OpenSimulator started, it was decided that each user
should have it's own packet processing thread.  This worked well at
the time, but limited concurrency because of lock and thread
contention.   5 UDP connections = 5 dedicated threads that spent most
of their time waiting(this got expensive when child agents were
considered).

Recently:
A few months ago, Intel provided some test results and provided a few
developers to re-write the UDP server and client handling stack.
One of the things that they did was make each UDP server use two
threads.  One for the incoming packet processing and one for the
outgoing packet processing, regardless of how many clients were
connected to OpenSimulator.    Essentially, this made it 'one thread
per network connection resource'.   This was great because it reduced
the amount of threads that opensimulator uses quite significantly.
There were some problems though.  Some code handling the upper level
processing (Scene, SceneObjectGroup, SceneObjectPart..  etc.. )  of
the data in the packet took long enough to process..   to cause the
simulator to slowly come to a crawl over time and disconnect users.

Because the infrastructure wasn't in place to selectively delegate
which packets are handled asynchronously and synchronously, the choice
was made to process each packet on the threadpool so that a long
running packet processor wouldn't cause the UDP server to get
backlogged and cause all users connected to get disconnected.   This
solution wasn't really optimal..    because a single client, moving
around, causes ~15 AgentUpdate packets to be sent to the server per
second.      15 packets means that up to 15 threadpool requests were
happening a second per connected root agent.       There are various
reasons why this isn't optimal, I won't go into them here :).

Now:
I'm happy to announce, that the infrastructure for selectively
delegating whether a packet will be processed synchronously or
asynchronously is in place as of
4ef8dc7d96fa2d4efd992ff7d304b8894f004c4f.   You can find the way to
set a packet handler as synchronous in
OpenSim.Region.ClientStack.LindenUDP.LLClientView.RegisterLocalPacketHandler.
      Note that packet handlers with 3 parameters with the third one
set to 'false' means that the packet handler is synchronous.    Packet
handlers in that method with 2 parameters or 3 parameters and the
third one set to 'true' means that the packet is processed
asynchronously on the threadpool using the method defined by
async_call_method in OpenSim.ini

I've also taken the liberty of setting the AgentUpdate Packet, the
ViewerEffect Packet, and a few others as synchronous.   The
AgentUpdate Packet and ViewerEffect Packet are the two packets that
are sent from the client the most often, so, these alone should reduce
the threads required per client significantly.

Testers:
What users and testers should look for, however, is the Simulator
suddenly disconnecting all users.    If this occurs, then, likely, the
UDP server is not moving fast enough to effectively process packets
for the current amount of users.

Developers:
It's time to look at the length of time it takes for packet processors
to finish processing and decide if the packet handler should be
executed synchronously or asynchronously.
Any synchronously processed packet is processed in the thread context
beginning with the UDPServer.

Hopefully, this will allow us to increase the number of concurrently
connected users further.

Regards

Teravus