Revision as of 13:50, 6 October 2014

Introduction

These are draft notes on OpenSimulator's implementation of the LLUDP ClientStack, a chunk of code which is used to send and receive packets from viewers (clients) implementing the Linden Labs virtual environment protocol.

As such, this stack handles

• Setup of an inbound UDP handling stack for each region.
• Handling of inbound UDP messages from a connected viewer.
• Distribution of recieved UDP messages to appropriate handling code (e.g. selection handling code if a prim is selected).
• Sending of outbound UDP messages to a connected viewer.
• Throttling of outbound messages.
• Sending and receive of ack messages, both inline within other messages and as standalone messages.
• Resending of messages that are marked as reliable but for which receipt has not been acknowledged by the viewer.
• Throttling of outgoing UDP messages.
• Pooling of clientstack structures (e.g. classes representing messages) in order to improve efficiency and reduce memory usage.

Useful console commands

These are console commands which are useful in debugging or investigating the LLUDP protocol.

• debug lludp packet [--default | --all] <level> [<avatar-first-name> <avatar-last-name>] - Turn on packet debugging. In OpenSimulator 0.7.5 and previous this command was "debug packet". "--default" applies the new logging level to all avatar that login after that point. "--all" applies it to all existing avatars as well.
• debug lludp pool <on|off> - Turn object pooling within the lludp component on or off.
• debug lludp start <in|out|all> - Control LLUDP packet processing.
• debug lludp status - Return status of LLUDP packet processing.
• debug lludp stop <in|out|all> - Stop LLUDP packet processing.

Inbound UDP

TODO: Need to fill out more as required.

Code paths

Part 1

• Inbound UDP messages are received on a region's listening UDP port (currently in OpenSimUDPBase). These are IOCP threads from the runtime as processing is done asynchronously.
• On receive, the thread kicks off another asychrnous receive (which may be handled by a different IOCP thread).
• Each received packet is sanity checked for length and basic malformations.
• Data is inserted into a packet structure drawn from a pool.
• If packet
  • Is UseCircuitCode (used to set up circuits with viewers) then this is handled on a separate thread.
  • Is CompleteAgentMovement (used to complete the insertion of an avatar into a region) then this is handled on a separate thread.
  • Has appended acks to a normal packet or is PacketAck then these are processed.
  • Is reliable then a pending ack is queued for acknowledgement to the client.
  • Is AgentUpdate and is not significant (same as last packet, where these are received about 10 times a second) then it is discarded.
  • StartPingCheck then this is handled.
• All other packets are placed into a queue (packetInbox) for further processing.
• The thread is released (synchronous fetch not currently operational).

Part 2

• A continuously running thread fetches the next packet at the front of the packetInbox queue.
• It enters the LLClientView structure, logging details of the packet for debug purposes if required.
• The packet is processed either synchronously (on this same thread) or asynchronously by queueing it as a work item to a threadpool (SmartThreadPool by default).
  • Some packets are handled synchronously due to issues if they are processed out of order (e.g. multi-face texture changes not occuring properly). Others may be synchronous by default where they could actually be handled asynchronously.
• The long running thread loops.

Outbound UDP

Code paths

This section is very incomplete.

• LLUDPServer.SendPacketFinal() is the penultimate method before UDP data is placed on the wire.
  • If the packet is not zero-coded and has room, then any waiting acks are appended.
  • If the packet is reliable then the server records its expectation of acks from the client.

• If there are no packets waiting to be sent for a throttle and none in the outbox, then BeginFireQueueEmpty().

• LLUDPClient.BeingFireQueueEmpty fires HasUpdates event is fired to determine if updates are waiting.
  • LLClientView.HandleHasUpdates() handles this and returns true if it has queued entity updates, entity property updates or image manager updates (UDP asset download).

• If there are such updates waiting, a threadpool FireQueueEmpty() thread is triggered
• This then fires the OnQueueEmpty event.
  • Which LLClientView handles with HandleQueueEmpty() which places a certain number of updates via OutPacket()

Throttles

This section is very incomplete. Outbound UDP is throttled per client.

If

[ClientStack.LindenUDP]
enable_adaptive_throttles = true

(the current default). Then client throttles are adjusted automatically by the server. Whilst the connection is fine, rates are increased. However, if packets marked as reliable are not received in a timely manner, then the throttle is reduced until no more packets are losed.

If

[ClientStack.LindenUDP]
enable_adaptive_throttles = false

Then throttles are set by the client. Throttle rates may still be restricted on the server side by setting either or both of

[ClientStack.LindenUDP]
client_throttle_max_bps = <some-client-limit>
scene_throttle_max_bps = <some-entire-scene-limit>

The client_throttle_max_bps setting will enforce a maximum limit for each client connection. The scene_throttle_max_bps setting will set a maximum output limit for all connections to a scene. So for this latter limit, every extra connection will reduce the bandwidth for everybody as a fixed limit is shared between more clients. This includes child connections (used for clients with avatars in neighbouring regions to see activity in this region).

Other references

• Sim Throttles contains very old information on the implementation of throttles. This has likely changed considerably but could still be useful.