Debugging - Revision history

Ferd Frederix: show connections remove useless old rev info.

2019-08-11T00:51:01Z

show connections remove useless old rev info.

Manni at 01:15, 26 April 2016

2016-04-26T01:15:42Z

Justincc at 02:01, 29 November 2014

2014-11-29T02:01:03Z

Orenh: /* HTTP Activity in Simulator and Robust */

2014-03-25T08:35:50Z

‎HTTP Activity in Simulator and Robust

Orenh: /* HTTP Messages Between Simulator and Robust */

2014-03-25T08:35:26Z

‎HTTP Messages Between Simulator and Robust

Orenh at 08:24, 25 March 2014

2014-03-25T08:24:45Z

Orenh: /* Specific areas */

2014-03-25T07:08:51Z

‎Specific areas

Justincc at 23:46, 12 July 2012

2012-07-12T23:46:17Z

Justincc: /* General */

2012-07-12T23:00:02Z

‎General

Justincc: Created page with "=General= In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't g..."

2012-07-12T21:11:24Z

Created page with "=General= In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't g..."

New page

=General=

In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command.

There are also logging statements through the OpenSimulator code, some of which are commented out. Generally these can be uncommented to provide extra useful information, though you may have to resort to adding new log statements.

=Specific areas=
However, there are a few things we say about debugging specific OpenSimulator issues. This section will be expanded as time goes on.

== Initial connection ==
This trips people up a lot because of the complex interplay between different server processes in grid mode and between simulator and viewer, with viewers on both the local LAN and over the Internet. Problems here are generally due to configuration rather than OpenSimulator code issues. See [[Troubleshooting]] for more details.

== Problems during connection ==
Here are some commands that can help you get more information is you have problems when a connection has already been established (e.g. avatars stopping in place), etc. You can get more help on any of these commands by typing "help <command-name>" on the simulator console.

* debug packet <0-256> [<first-name>] [<last-name>] - this command logs various incoming and outgoing packets depending on the packet level (0-256). This can help determine whether there is any packet flow between simulator and client at all.

* show queues [<full>] - this command shows packet in, out and resent statistics for particular clients, as well as the packets being queued. Monitoring the in and outbound packet numbers will show if packets are getting through between simulator and viewer. If resent is climbing rapidly then its likely you have connection issues since the simulator is having to resend lots of packets that aren't being acknowledged by the viewer. If the Q numbers increase without going down then clients may have extremely small bandwidth limits or the outgoing network connection from the simulator may be overwhelmed.

* show users - show users active on the simulator

* show connections - show information about individual client connections, including their IP address and whether they are active (in OpenSimulator 0.7.4 dev code only).

@@ Line 44: / Line 44: @@
 * show users - show users active on the simulator
-* show connections - show information about individual client connections, including their IP address and whether they are active (in OpenSimulator 0.7.4 dev code only).
+* show connections - show information about individual client connections, including their IP address and whether they are active.
 == Threadpool Activity ==

← Older revision		Revision as of 01:15, 26 April 2016
Line 1:		Line 1:
		+	{{Quicklinks}}
		+
	=General=		=General=

← Older revision		Revision as of 02:01, 29 November 2014
Line 2:		Line 2:

	In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command.		In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side. We won't go through general information about using debuggers here, you can find it elsewhere on the net. However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command.
		+
		+	=Thread Dumps=
		+
		+	If the simulator or some aspect of it appears to be frozen, one possibility is that threads are in deadlock for some reason. In this case, a dump of all threads can provide useful information.
		+
		+	In Mono, you can get this dump by sending the SIGQUIT unix signal to the Mono process. You can do this either with CTRL + \ on the console or by a command such as
		+
		+	kill -SIGQUIT <pid>
		+
		+	This dump may be very large. If you are reported this trace for debug purposes, it is essential that it is captured and reported accurately.
		+
		+	If at all possible, please start Mono with the --debug switch so that line numbers are added to traces (there is no major performance penalty on using this switch). This is sometimes the only way to pin down which parts of the code threads may be deadlocked on.
		+
		+	Sending SIGQUIT to Mono may make it unstable. If you do this, you are advised to restart the process in question after all useful debugging information has been captured.
		+
		+	'''Not sure how to achieve the same result on Windows, if this is possible at all. If someone knows would be great to get this section filled in.'''

	=Internal tools=		=Internal tools=

@@ Line 32: / Line 32: @@
 A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can [[General-Purpose Threadpool|enable logging that shows activity in this threadpool]].
-== HTTP Activity in Simulator and Robust ==
+== HTTP Activity ==
-Use this console command to enable logging of HTTP requests and responses in Simulators and Robust:
+Use this console command to enable logging of HTTP requests and responses in OpenSim or Robust:
   debug http in|out|all [<level>]

@@ Line 32: / Line 32: @@
 A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can [[General-Purpose Threadpool|enable logging that shows activity in this threadpool]].
-== HTTP Messages Between Simulator and Robust ==
+== HTTP Activity in Simulator and Robust ==
-Use this console command to enable logging of HTTP requests and responses between Simulators and Robust:
+Use this console command to enable logging of HTTP requests and responses in Simulators and Robust:
   debug http in|out|all [<level>]
@@ Line 57: / Line 57: @@
 </pre>
-In particular, to get a good idea of the communications turn on debug level 5 on both the simulator and Robust to see short snippets from each request and response.
+In particular, to get a good idea of the communications flow turn on debug level 5 on both the simulator and Robust to see short snippets from each request and response. This will also show HTTP requests between the viewer and the Simulator.

← Older revision		Revision as of 08:24, 25 March 2014
Line 31:		Line 31:

	A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can [[General-Purpose Threadpool\|enable logging that shows activity in this threadpool]].		A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can [[General-Purpose Threadpool\|enable logging that shows activity in this threadpool]].
		+
		+	== HTTP Messages Between Simulator and Robust ==
		+
		+	Use this console command to enable logging of HTTP requests and responses between Simulators and Robust:
		+
		+	debug http in\|out\|all [<level>]
		+
		+	Options:
		+	<pre>
		+	If in or all and
		+	level <= 0 then no extra logging is done.
		+	level >= 1 then short warnings are logged when receiving bad input data.
		+	level >= 2 then long warnings are logged when receiving bad input data.
		+	level >= 3 then short notices about all incoming non-poll HTTP requests are logged.
		+	level >= 4 then the time taken to fulfill the request is logged.
		+	level >= 5 then a sample from the beginning of the data is logged.
		+	level >= 6 then the entire data is logged.
		+	no level is specified then the current level is returned.
		+
		+	If out or all and
		+	level >= 3 then short notices about all outgoing requests going through WebUtil are logged.
		+	level >= 4 then the time taken to fulfill the request is logged.
		+	level >= 5 then a sample from the beginning of the data is logged.
		+	level >= 6 then the entire data is logged.
		+	</pre>
		+
		+	In particular, to get a good idea of the communications turn on debug level 5 on both the simulator and Robust to see short snippets from each request and response.

← Older revision		Revision as of 07:08, 25 March 2014
Line 27:		Line 27:

	* show connections - show information about individual client connections, including their IP address and whether they are active (in OpenSimulator 0.7.4 dev code only).		* show connections - show information about individual client connections, including their IP address and whether they are active (in OpenSimulator 0.7.4 dev code only).
		+
		+	== Threadpool Activity ==
		+
		+	A lot of the work in OpenSim is done in tasks that are run on a general-purpose threadpool. You can [[General-Purpose Threadpool\|enable logging that shows activity in this threadpool]].

@@ Line 17: / Line 17: @@
 This trips people up a lot because of the complex interplay between different server processes in grid mode and between simulator and viewer, with viewers on both the local LAN and over the Internet.  Problems here are generally due to configuration rather than OpenSimulator code issues.  See [[Troubleshooting]] for more details.
-== Problems during connection ==
+== Problems after connection established ==
 Here are some commands that can help you get more information is you have problems when a connection has already been established (e.g. avatars stopping in place), etc.  You can get more help on any of these commands by typing "help <command-name>" on the simulator console.

@@ Line 2: / Line 2: @@
 In general, OpenSimulator is a difficult system to debug due to its high concurrency, complexity and the lack of good tools for Mono on the Linux/Mac side.  We won't go through general information about using debuggers here, you can find it elsewhere on the net.  However, one thing to note is that the debugger must be capable of ignoring the SelfDeleteException - this is currently used by the XEngine script engine to kill scripts aborted via the llDie() LSL command.
 There are also logging statements through the OpenSimulator code, some of which are commented out.  Generally these can be uncommented to provide extra useful information, though you may have to resort to adding new log statements.
 =Specific areas=