Debugging
From OpenSimulator
m (show connections remove useless old rev info.) |
|||
| (6 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| + | {{Quicklinks}} | ||
| + | |||
=General= | =General= | ||
In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command. | In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command. | ||
| + | |||
| + | =Thread Dumps= | ||
| + | |||
| + | If the simulator or some aspect of it appears to be frozen, one possibility is that threads are in deadlock for some reason. In this case, a dump of all threads can provide useful information. | ||
| + | |||
| + | In Mono, you can get this dump by sending the SIGQUIT unix signal to the Mono process. You can do this either with CTRL + \ on the console or by a command such as | ||
| + | |||
| + | kill -SIGQUIT <pid> | ||
| + | |||
| + | This dump may be very large. If you are reported this trace for debug purposes, it is essential that it is captured and reported accurately. | ||
| + | |||
| + | If at all possible, please start Mono with the --debug switch so that line numbers are added to traces (there is no major performance penalty on using this switch). This is sometimes the only way to pin down which parts of the code threads may be deadlocked on. | ||
| + | |||
| + | Sending SIGQUIT to Mono may make it unstable. If you do this, you are advised to restart the process in question after all useful debugging information has been captured. | ||
| + | |||
| + | '''Not sure how to achieve the same result on Windows, if this is possible at all. If someone knows would be great to get this section filled in.''' | ||
=Internal tools= | =Internal tools= | ||
| Line 26: | Line 44: | ||
* show users - show users active on the simulator | * show users - show users active on the simulator | ||
| − | * show connections - show information about individual client connections, including their IP address and whether they are active | + | * show connections - show information about individual client connections, including their IP address and whether they are active. |
| + | |||
| + | == Threadpool Activity == | ||
| + | |||
| + | A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can [[General-Purpose Threadpool|enable logging that shows activity in this threadpool]]. | ||
| + | |||
| + | == HTTP Activity == | ||
| + | |||
| + | Use this console command to enable logging of HTTP requests and responses in OpenSim or Robust: | ||
| + | |||
| + | debug http in|out|all [<level>] | ||
| + | |||
| + | Options: | ||
| + | <pre> | ||
| + | If in or all and | ||
| + | level <= 0 then no extra logging is done. | ||
| + | level >= 1 then short warnings are logged when receiving bad input data. | ||
| + | level >= 2 then long warnings are logged when receiving bad input data. | ||
| + | level >= 3 then short notices about all incoming non-poll HTTP requests are logged. | ||
| + | level >= 4 then the time taken to fulfill the request is logged. | ||
| + | level >= 5 then a sample from the beginning of the data is logged. | ||
| + | level >= 6 then the entire data is logged. | ||
| + | no level is specified then the current level is returned. | ||
| + | |||
| + | If out or all and | ||
| + | level >= 3 then short notices about all outgoing requests going through WebUtil are logged. | ||
| + | level >= 4 then the time taken to fulfill the request is logged. | ||
| + | level >= 5 then a sample from the beginning of the data is logged. | ||
| + | level >= 6 then the entire data is logged. | ||
| + | </pre> | ||
| + | |||
| + | In particular, to get a good idea of the communications flow turn on debug level 5 on both the simulator and Robust to see short snippets from each request and response. This will also show HTTP requests between the viewer and the Simulator. | ||
Latest revision as of 16:51, 10 August 2019
 |
Languages: |
|
Contents |
[edit] General
In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command.
[edit] Thread Dumps
If the simulator or some aspect of it appears to be frozen, one possibility is that threads are in deadlock for some reason. In this case, a dump of all threads can provide useful information.
In Mono, you can get this dump by sending the SIGQUIT unix signal to the Mono process. You can do this either with CTRL + \ on the console or by a command such as
kill -SIGQUIT <pid>
This dump may be very large. If you are reported this trace for debug purposes, it is essential that it is captured and reported accurately.
If at all possible, please start Mono with the --debug switch so that line numbers are added to traces (there is no major performance penalty on using this switch). This is sometimes the only way to pin down which parts of the code threads may be deadlocked on.
Sending SIGQUIT to Mono may make it unstable. If you do this, you are advised to restart the process in question after all useful debugging information has been captured.
Not sure how to achieve the same result on Windows, if this is possible at all. If someone knows would be great to get this section filled in.
[edit] Internal tools
There are also logging statements through the OpenSimulator code, some of which are commented out. Generally these can be uncommented to provide extra useful information, though you may have to resort to adding new log statements.
There are also some internal methods that might be useful.
- Util.PrintCallStack() - This will print the call stack at the time the method is executed before continuing execution. Unfortunately, due to Mono limitations can only work on the current thread.
[edit] Specific areas
However, there are a few things we say about debugging specific OpenSimulator issues. This section will be expanded as time goes on.
[edit] Initial connection
This trips people up a lot because of the complex interplay between different server processes in grid mode and between simulator and viewer, with viewers on both the local LAN and over the Internet. Problems here are generally due to configuration rather than OpenSimulator code issues. See Troubleshooting for more details.
[edit] Problems after connection established
Here are some commands that can help you get more information is you have problems when a connection has already been established (e.g. avatars stopping in place), etc. You can get more help on any of these commands by typing "help <command-name>" on the simulator console.
- debug packet <0-256> [<first-name>] [<last-name>] - this command logs various incoming and outgoing packets depending on the packet level (0-256). This can help determine whether there is any packet flow between simulator and client at all.
- show queues [<full>] - this command shows packet in, out and resent statistics for particular clients, as well as the packets being queued. Monitoring the in and outbound packet numbers will show if packets are getting through between simulator and viewer. If resent is climbing rapidly then its likely you have connection issues since the simulator is having to resend lots of packets that aren't being acknowledged by the viewer. If the Q numbers increase without going down then clients may have extremely small bandwidth limits or the outgoing network connection from the simulator may be overwhelmed.
- show users - show users active on the simulator
- show connections - show information about individual client connections, including their IP address and whether they are active.
[edit] Threadpool Activity
A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can enable logging that shows activity in this threadpool.
[edit] HTTP Activity
Use this console command to enable logging of HTTP requests and responses in OpenSim or Robust:
debug http in|out|all [<level>]
Options:
If in or all and level <= 0 then no extra logging is done. level >= 1 then short warnings are logged when receiving bad input data. level >= 2 then long warnings are logged when receiving bad input data. level >= 3 then short notices about all incoming non-poll HTTP requests are logged. level >= 4 then the time taken to fulfill the request is logged. level >= 5 then a sample from the beginning of the data is logged. level >= 6 then the entire data is logged. no level is specified then the current level is returned. If out or all and level >= 3 then short notices about all outgoing requests going through WebUtil are logged. level >= 4 then the time taken to fulfill the request is logged. level >= 5 then a sample from the beginning of the data is logged. level >= 6 then the entire data is logged.
In particular, to get a good idea of the communications flow turn on debug level 5 on both the simulator and Robust to see short snippets from each request and response. This will also show HTTP requests between the viewer and the Simulator.
