General

In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command.

Internal tools

There are also logging statements through the OpenSimulator code, some of which are commented out. Generally these can be uncommented to provide extra useful information, though you may have to resort to adding new log statements.

There are also some internal methods that might be useful.

• Util.PrintCallStack() - This will print the call stack at the time the method is executed before continuing execution. Unfortunately, due to Mono limitations can only work on the current thread.

Specific areas

However, there are a few things we say about debugging specific OpenSimulator issues. This section will be expanded as time goes on.

Initial connection

This trips people up a lot because of the complex interplay between different server processes in grid mode and between simulator and viewer, with viewers on both the local LAN and over the Internet. Problems here are generally due to configuration rather than OpenSimulator code issues. See Troubleshooting for more details.

Problems after connection established

Here are some commands that can help you get more information is you have problems when a connection has already been established (e.g. avatars stopping in place), etc. You can get more help on any of these commands by typing "help <command-name>" on the simulator console.

• debug packet <0-256> [<first-name>] [<last-name>] - this command logs various incoming and outgoing packets depending on the packet level (0-256). This can help determine whether there is any packet flow between simulator and client at all.

• show queues [<full>] - this command shows packet in, out and resent statistics for particular clients, as well as the packets being queued. Monitoring the in and outbound packet numbers will show if packets are getting through between simulator and viewer. If resent is climbing rapidly then its likely you have connection issues since the simulator is having to resend lots of packets that aren't being acknowledged by the viewer. If the Q numbers increase without going down then clients may have extremely small bandwidth limits or the outgoing network connection from the simulator may be overwhelmed.

• show users - show users active on the simulator

• show connections - show information about individual client connections, including their IP address and whether they are active (in OpenSimulator 0.7.4 dev code only).

Threadpool Activity

A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can enable logging that shows activity in this threadpool.

HTTP Activity

Use this console command to enable logging of HTTP requests and responses in OpenSim or Robust:

debug http in|out|all [<level>]

Options:

If in or all and
  level <= 0 then no extra logging is done.
  level >= 1 then short warnings are logged when receiving bad input data.
  level >= 2 then long warnings are logged when receiving bad input data.
  level >= 3 then short notices about all incoming non-poll HTTP requests are logged.
  level >= 4 then the time taken to fulfill the request is logged.
  level >= 5 then a sample from the beginning of the data is logged.
  level >= 6 then the entire data is logged.
  no level is specified then the current level is returned.

If out or all and
  level >= 3 then short notices about all outgoing requests going through WebUtil are logged.
  level >= 4 then the time taken to fulfill the request is logged.
  level >= 5 then a sample from the beginning of the data is logged.
  level >= 6 then the entire data is logged.

In particular, to get a good idea of the communications flow turn on debug level 5 on both the simulator and Robust to see short snippets from each request and response. This will also show HTTP requests between the viewer and the Simulator.