Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002946opensim[REGION] OpenSim Corepublic2009-01-02 11:022010-09-12 10:47
Assigned ToDiva 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0002946: On an additional avatar, UDP comms stops if EstablishAgentCommunication is sent on login
DescriptionOn standalone, if I have more than 5 regions setup I can log one avatar in and move around with no problems. But if an additional avatar logs in, then all comms to *both* avatars largely stops (i.e. no movements, no pings, etc.). But strangely the stop isn't complete - a small number of packets is still sent through in bursts, with upto 15 seconds between bursts.

In addition, if there are only 5 regions then 2 avatars can login without any comms problems. But if a third avatar logs in then comms traffic falls to small bursts as described above.

This is very strange, but by a process of elimination I've deduced that this was introduced in r7708 - early revisions do not exhibit this problem. Even more specifically, if I comment out the sending of the EventQueue message EstablishAgentCommunication in SceneCommunicationService.InformClientOfNeighbourAsync(), the problem goes away.

So, is it completely necessary that this event is sent. On my limit tests, OpenSim seems fine without it.

I'd really appreciate more feedback on whether other people experience this, or more information about whether the client really expects the EAC message in this situation. However, it's very strange that sending this seems to limit messages to small bursts. I'd also greatly appreciate feedback as to whether this occurs on Windows or on different versions of Mono.
TagsNo tags attached.
Git Revision or version number
Run Mode Standalone (Multiple Regions)
Physics EngineBasicPhysics
Script Engine
EnvironmentMono / Linux32
Mono Version1.9.1
Attached Filespatch file icon mantis-2946-comment-out-eac-on-login.patch [^] (790 bytes) 2009-01-02 11:04 [Show Content]
txt file icon thread-dump-1.txt [^] (54,264 bytes) 2009-01-20 11:37 [Show Content]

- Relationships

-  Notes
justincc (administrator)
2009-01-02 11:04

Okay, attached as mantis-2946-comment-out-eac-on-login.patch is the code which simply comments out the EAC send.
justincc (administrator)
2009-01-02 11:07

Okay, so just to make things interesting I've been testing with two different clients, 1.20.15 and 1.21.6. Going to retest with all logins coming in from 1.21.6
justincc (administrator)
2009-01-02 11:15

Hmm, still happens if both clients are 1.21.6 (coming from different machines).

Perhaps by not sending EAC the EventQueue is never activated perhaps? (though border crossings still appear to work just fine). Perhaps this is something to do with thread numbers (but does this explain why everything was hunky-dory in r7007? It doesn't seem so).
justincc (administrator)
2009-01-02 11:20
edited on: 2009-01-05 06:17

The client thread count might have something to do with it since the problem seems to occur when standalone regions * users > 10. This is an induction - I've only tested

5 regions * 2 avatars - OK
5 regions * 3 avatars - UDP DROPS TO VERY TINY BURSTS WITH LARGE (5-15 sec) GAPS
6 regions * 2 avatars - UDP DROPS TO VERY TINY BURSTS WITH LARGE (5-15 sec) GAPS

However, it's still curious that r7007 was fine with the same number of threads, afaik. So I'm inclined to discount a mono bug, though information from somebody running mono 2.0.1 would be appreciated.

justincc (administrator)
2009-01-02 13:09

Some additional information - I'm specifying external host name as a number ip ( Also, the topologies I've tested have varied, though always with all the regions surrounding one 'central' region (which is what the avatars directly log in too).

Also, Diva has just reproduced this with 4 avatars in just 1 region!
Diva (administrator)
2009-01-02 13:13

Standalone in linux, mono 1.9, 1 region: the 4th avie makes things freeze.

I strongly suspect this has something to do with mono 1.9. I tried reproducing this on a windows standalone, and I can't repro it there. Several people have recommended an upgrade to mono 2.0.1, supposedly lots of things work better there.
jhurliman (manager)
2009-01-02 13:51

Unfortunately, the only decent way to get Mono 2.0.1 right now is to compile from source (and if you're going to do that, you should compile the latest Mono 2.2 release candidate). This isn't going to change until the new Debian is released.
HomerHorwitz (manager)
2009-01-02 15:22

Actually, mono 2.0.1 is installable from Debian/experimental
thomax (reporter)
2009-01-02 15:46

for those who run in in-world script compile error with mono2x: [^]

can be fixed with:
HOW TO: Compile Mono from SVN Source Code [Linux Universal] [^]
HomerHorwitz (manager)
2009-01-03 10:47

r7933, Linux32, mono 2.0.1, standalone: 1 region works nicely with 4 avatars; no problem there (well, apart of the memory usage of 4 running viewers).
justincc (administrator)
2009-01-03 10:49
edited on: 2009-01-03 10:52

I would be very interested to see if things also work okay with 2 (or 3) avatars and 6 regions if you have the opportunity to try that, Homer.

HomerHorwitz (manager)
2009-01-03 11:47

Standalone with 6 freshly created regions (after 'terrain fill 21'), 2 x 3:
  5 6
  3 4
  1 2

Logging in 3 avatars into R3. I see all the regions, show users full shows that all agents are established (3 roots, 15 children).
After logging in the third avatar, I can't move any of the three avatars. After some time, two get a forced logout from the viewer (server isn't responding anymore), after which the last one can walk again. At that point, the agents of the logged out avatars are still there (show users full), but after 0000004:0000005 minutes, they are cleaned up. Viewer is SL
Diva (administrator)
2009-01-03 12:06

Child agents now are full-blown agents -- they have everything a root agent has, including all communication channels to the viewer (EQ included). The only difference is that they're not made visible. In the case above, we have 18 agents being served by one single instance.

We may need to rethink the memory and bandwidth requirements for opensim instances according to the new status of child agents, and the consequences of having several regions in one single instance.

Unless, of course, we find a bug somewhere that's causing this :-)
HomerHorwitz (manager)
2009-01-03 12:12

That's everything happening on a local instance. Neither (local) network, nor CPU or memory is at its limit (it's 6 *emtpy* regions, after all).
Diva (administrator)
2009-01-03 12:22

Mono may have its limits wrt number of threads, for example. Each EQ is essentially a thread that's hanging from the client. You have 18 EQs securing those resources for extending periods of time. Not sure how mono handles that.

Can any of you reproduce this problem in Windows? I can't. I have a 3x3 standalone, and I tried 4 avies, 23 agents at some point; no problems there.

So this must have something to do with mono and, my best guess, threads... which is not to say that we shouldn't do something about it.
FrankNichols (reporter)
2009-01-03 12:46

I believe this is a bug, I have had it happen on occasion on my sim also and it "feels" like a threshold is reached and then it goes to a crawl. 4 av's are fine - no lag at all, then a 5th joins us and ping times go out from 30ms to 10,000ms.

Also, the number of av's to trigger it seems to vary on location - I have not been able to nail down what the trigger is yet, but it is not just the number of avs.
Diva (administrator)
2009-01-03 13:22

You're in linux, too, Frank, yes?

I just tested this again in Windows, on a 3x3 standalone. I logged in 5 avies, with a maximum of 30 agents at one point. The 5th avie slowed things a bit, but it wasn't just opensim -- my whole computer was crawling because of the 5 viewers. In any case, even with this slow down and the fan going off like crazy, opensim didn't freeze like it does on my Linux/mono 1.9 standalone with 1 region and 4 avies.

The fact that commenting out EAC messages (hence not opening child agents' EQs) makes this problem go away for justin, makes me believe that this has something to do with threads and mono.

The position of the avies should have an influence on this, since depending on where they are, there may be more or less child agents (hence EQs, hence hanging threads). And whatever else is going on that spawns threads that hang for some time should also have an influence.

Did I hear Teravus say that the EQ is a really bad idea? :D
It really is. We may get better performance sending out dummy EQ events every second instead of hanging in there with a blocking thread for 15 secs. At least while the viewer is processing the message, its EQ thread is gone from the server.

Anyway, I hope you find what's causing this. I'm out to the beach for a few days :D
justincc (administrator)
2009-01-05 06:47

Does enabling the EAC add a new EventQueue for each child, and does this mean that an extra thread is added in addition to the existing UDP child agent thread? If so, I can see where EAC would lead to a big increase in the number of threads (though if EQ threads exist they don't appear to show up with the 'show threads' command).

However, I'm not sure that there is any thread limit in mono (I've searched for this before and never turned up anything that indicates this). Of course, that's not to say that there aren't mono issues, but from Homer's comments these problems are turning up on mono 2.0.1 as well as 1.9.1. It would also be very surprised if there hadn't been a lot of information generated on the web already if the mono thread limit were so low.
justincc (administrator)
2009-01-05 06:50

And just for information, does disabling the EAC actually have any ill effects right at the moment? I'm presuming that this might fall back on the existing UDP protocol since things still appear to work without child EQ activation.

I'm not proposing actually disabling this in the trunk at this point - I'm just wondering if using the one line patch attached is a valid workaround until this is resolved somehow. Oh, hope you have a nice time on the beach, Diva :)
carlosroundel (reporter)
2009-01-08 09:51

I report the same behavior in grid mode, rev 7895 and later
I solved as suggested in commenting patch.
server involved: ovh rps server, 512 ram, linux ubuntu 8.04 mono 2.0.1
4 regions with one instance of opensim
after patch on regions enter up to 15 avatar without problems
Diva (administrator)
2009-01-19 10:47

Unfortunately, this patch makes things go back to the way they were before, i.e. no EQs in the children which has many subtle problems. Most of these problems are not visible, until things start to break silently. Here's how to see these problems very clearly:

Run a standalone, 2 or more adjacent regions. Go close to the border. Create a prim across the border. Put a script in it. Close it. Open that object's folder again, and edit the script, add a few blank lines. Click "Save".

When there's no EQ in the child (so when this patch is on) the saving process hangs. When there is EQ (as in trunk now) the saving process happens successfully.

The reason for EQs in children in the first place is to support operations like this one, i.e. to enable to do things across borders.

So, back to this freezing problem: it's definitely related to the multiple EQs and their respective hanging threads. The question is whether this is a mono problem or our problem - maybe we're locking things too much?
Diva (administrator)
2009-01-19 14:38
edited on: 2009-01-21 08:02

I'm still getting this freeze on my Linux standalone after upgrading to mono 2.0.1.

[Thread dump deleted by Diva. Not important anymore]

Diva (administrator)
2009-01-19 15:10

OK, progress! So these 2 caught my eye:

"EventQueueManagerThread_1" tid=0x0x2192b90 this=0x0xd9000:
  at (wrapper managed-to-native) System.Threading.Thread.Sleep_internal (int) <0x00004>
  at (wrapper managed-to-native) System.Threading.Thread.Sleep_internal (int) <0xffffffff>
  at System.Threading.Thread.Sleep (int) <0x00015>
  at OpenSim.Region.ScriptEngine.DotNetEngine.EventQueueThreadClass.DoProcessQueue () <0x00160>
  at OpenSim.Region.ScriptEngine.DotNetEngine.EventQueueThreadClass.EventQueueThreadLoop () <0x00057>
  at (wrapper runtime-invoke) System.Object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

"EventQueueManagerThread_0" tid=0x0x6cfcb90 this=0x0xd9690:
  at (wrapper managed-to-native) System.Threading.Thread.Sleep_internal (int) <0x00004>
  at (wrapper managed-to-native) System.Threading.Thread.Sleep_internal (int) <0xffffffff>
  at System.Threading.Thread.Sleep (int) <0x00015>
  at OpenSim.Region.ScriptEngine.DotNetEngine.EventQueueThreadClass.DoProcessQueue () <0x00160>
  at OpenSim.Region.ScriptEngine.DotNetEngine.EventQueueThreadClass.EventQueueThreadLoop () <0x00057>
  at (wrapper runtime-invoke) System.Object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

They caught my eye because I have XEngine by default, but looking closer at my OpenSim.ini I noticed that I also had the DotNetEngine enabled. I turned it off:

    Enabled = false

After I restarted, the freeze didn't happen. I tried several times to make sure, and, in fact, no freeze.

Then I asked Nebadon which script engine he uses in the plazas, because clearly they can handle a lot more than 4 avies. He said:

<diva> nebadon: question: what script engine do you use in WP?
<nebadon> Xengine
<nebadon> the plazas dont seem to like Dotnet
<nebadon> breaks down very fast
<diva> and in your OpenSim.ini do you have DotNet=false explicitly?
<nebadon> yes

Can someone who experiences the freeze please redo what I just did, i.e. turn off DotNetEngine, and report back?

Now the question is: what the heck are those threads doing in the DotNetEngine??? With that name? I don't know much about the script engines...
rtkwebman (reporter)
2009-01-19 20:04

Diva, I checked my OpenSim.ini and I am sorry to say I am already using XEngine and DotNetEngine is set to false. I am still experiencing the freezing/slow down.
    ; Choose one of the physics engines below
    ;physics = basicphysics
    ;physics = POS
    physics = OpenDynamicsEngine
    ;physics = modified_BulletX

    Enabled = false
    Enabled = true

Version: OpenSimulator Server (interface version 2)
Ubuntu 8.10 Standalone, mono 1.9.1
rtkwebman (reporter)
2009-01-20 08:21

Diva, now that I think about it; the other day I logged onto Wright Plaza and was fine until another avie logged in and I had a hard time not floating off and my response time was horrible. The same happened on Blade Plaza too. As a matter of fact the region just south of Blade froze me up completely. Can't remember what viewer I was using at the time.

Hope this helps.
Diva (administrator)
2009-01-20 09:46

It may be that not starting those two very "nervous" DotNetEngine threads (they're on a 50ms cycle) alleviates the problem, but doesn't fix it. I'm going to login to my standalone from different computers around my house, to see at which point it freezes now that the DotNetEngine is off.

But, again, this is another indicator that this freezing problem is thread-related, mono.

But clearly, Nebadon is doing something right with the plazas, which routinely host 20 avies (those occasional glitches in the plazas are perfectly normal). Maybe it's the large-heap option of mono?
justincc (administrator)
2009-01-20 10:23

Yes, the EventQueue stuff in DotNetEngine processes script events and doesn't have anything to do with the Second Life event queue. But I agree that the lower number of threads may affect this problem. I'll try to give it a go when I get a chance, though that may not be today.
justincc (administrator)
2009-01-20 10:52

Looking at the rest of the trace, nothing obvious leaps out at me.
Diva (administrator)
2009-01-20 10:54

OK, I redid the tests again, and did more tests. All in Linux/Mono 2.0.1. This time I run only a max of 3 viewers on each computer around my house (I have 3 computers). Clearly, the 4th viewer on my laptop is a bad idea, and distorts the results.

TEST 1: 1 region only. Tried both DotNetEngine enabled and disabled, results are the same. Logged in 3+3+2 viewers from my 3 computers, so 8 avies and 8 agents. No freeze whatsoever. The freeze I was seeing yesterday was probably due to having 4 viewers running on the same computer, and maybe other application I had at the time.

TEST 2: 4 adjacent regions (2x2), tried both DotNet enabled and disabled, the results are slightly better for disabled, but not that much. Logged in 3 viewers from one computer; the 4th viewer from another computer froze things up. That is, 3 avies = 12 agents was ok; 4 avies = 16 agents was not ok. The result was the same, independent on where the viewers were run (I tried several combinations). Trying to login a 5th avie does not succeed; the 5th viewer hangs on region handshake.

This really smells like thread problems, possibly even a deadlock on the 5th avie.
Diva (administrator)
2009-01-20 11:06

PS It doesn't matter where in the 2x2 the avies are, things freeze just the same. Which makes sense.
FrankNichols (reporter)
2009-01-20 11:13

Diva, another piece of info that may be unrelated on this. I am running head, 8090 on Linux/Mono 2.0.1. For me (hippo 0.4) TPs are very stable both between regions in the same instance and to regions in other instances and to HG regions and back. However, logging in a slow computer (hippo 0.4) with a weak video card results in TP failures more than 50% of the time. Ping times on the slow computer are in the neighborhood of 300ms, ping times on mine are around 30ms.

I am wondering if something time critical is not happening, ie. client is not reponding in time (or client is not seeing response from server in time) and the result is a thread lock.

Another piece of information, which seems related - ie. something not happening in time, if I transfer/copy a large number of assets from/to my inventory and prim I will get logged out by the client, but the server thinks I am still there. How this could relate to TP's, and I am just rambling here, is that following a TP many assets need to be sent to the client - ie. textures, prims, etc. Again, if the client/server connection is slow maybe something is timing out and the "recovery" is not happening correctly resulting in a lock.

These two symptoms could also be reflected by your tests above, in that more agents/clients logged in resulting in slower response times between client and server.
Diva (administrator)
2009-01-20 11:37

Repeated TEST 2 again, this time with the goal of trying to find threads in deadlock upon the 5th login via a SIGQUIT. The SIGQUIT makes the 4th avie logout; when that happens the 5th avie (which is blocked waiting for region handshake) unblocks and logs in. But the whole thing is still frozen. And the thread dump doesn't show any obvious deadlock - but I'm not sure if that's because the SIGQUIT itself kicked the offender thread(s) pertaining to avie 0000004.

Attaching the thread dump (thread-dump-1.txt).
Diva (administrator)
2009-01-20 11:53

Random observation 1: there are exactly 64 threads on the thread dump I attached, counting the number of tid=0x.... Powers of 2 are always suspicious, but maybe this is just a coincidence?

Random observation 2: there are only 3 Heartbeat threads. I was expecting to see 4 (4 regions). Maybe one died with the SIGQUIT. When I restart the sim, there are 4 Heartbeat threads.
Diva (administrator)
2009-01-20 12:55

I found out that mono has an important limit on threads, a thread pool that by default has 20 + (MONO_THREADS_PER_CPU * n_cpus) threads. In my case this amounted to 30 threads.

See here: [^]

Given this information, I set this environment variable to 15

And voila! The freeze is gone!

15 may not be enough for a popular sim, though.

Can others please do the same and report results?
Bartholomew Kleiber (reporter)
2009-01-20 14:03

Two points on this:
I am at this for some time now, and all of a sudden I have a hard time reproducing the issue with one island. Three days ago i was about to post it as 'an obvious issue because I am getting it every time with just three or four avatars'. Now I am trying to reproduce exactly that and suddenly it works on an empty island at least ... sigh.

Fortunately I saved a quite big scenario of a couple of sims which *really* always shows this issue.

The number 15 didnt do the trick but 25 gave me the 'voila' effect!!!
Diva, I'd say you are close.

I will now change it back to something smaller and see if the issue turns up again.
Bartholomew Kleiber (reporter)
2009-01-20 14:13

Changed it down to 10 threads_per_cpu.

Freeze is there again, so the plot thickens that your theory is correct.

The SL viewer spits out messages like this but I don't know if they are related to this problem:

INFO: LLXferManager::retransmitUnackedPackets: resending xfer 4597c60d-3dd9-b180-10ad-0be1099a0eb4:texture packet unconfirmed after: 3.14929 sec, packet 0
INFO: LLXferManager::retransmitUnackedPackets: resending xfer 5acb0370-8ab3-88fc-ac5c-b0d464caa3d5:texture packet unconfirmed after: 3.07317 sec, packet 1
INFO: LLXferManager::retransmitUnackedPackets: resending xfer c010e156-8cf8-08c0-f9c9-89390b517079:texture packet unconfirmed after: 3.06776 sec, packet 4
INFO: LLXferManager::retransmitUnackedPackets: resending xfer 6f9a1368-5de6-54c8-9c8f-c85d82ff55b5:texture packet unconfirmed after: 3.06556 sec, packet 8
INFO: LLXferManager::retransmitUnackedPackets: resending xfer 4597c60d-3dd9-b180-10ad-0be1099a0eb4:texture packet unconfirmed after: 3.08772 sec, packet 0

INFO: LLXferManager::retransmitUnackedPackets: resending xfer b37d7f3f-4590-6f2f-70a9-5adc8530539e:texture packet unconfirmed after: 3.01237 sec, packet 53
INFO: LLCircuitData::dumpResendCountAndReset: Circuit: resent 22 packets
INFO: LLXferManager::retransmitUnackedPackets: resending xfer b37d7f3f-4590-6f2f-70a9-5adc8530539e:texture packet unconfirmed after: 3.04672 sec, packet 53
rtkwebman (reporter)
2009-01-20 14:47

Well I did export MONO_THREADS_PER_CPU=35 and was able to login to my standalone with 6 avies all on different machines. CPU never exceeded 17% and everything flowed like water with no timeouts or bumps of any kind. I could not get past three before.
Bartholomew Kleiber (reporter)
2009-01-20 14:55

ok, here is what I've got so far:

    procnum around 238
    2 avatars can log in, third one freezes
    procnum around 252
    4 avatars can log in, fifth one freezes
    procnum around 293
    5 avatars can log in

... will test more tomorrow.
Bartholomew Kleiber (reporter)
2009-01-20 22:11

Since this seems to be a solution, I ask myself: are we addressing the cause or the symptom here? maybe the creation of these child threads goes exponentionally at some point? What would be the penalty if one wasn't able to create prims accross the border? losing attachements while border crossing?
Diva (administrator)
2009-01-20 22:25

It doesn't grow exponentially; it grows linearly. Each avie carries with it up to 9 agents right now -- but that's all.

Not having EQs across the border is a possibility for some applications, but not for all. LL changed a lot of messaging between the client and the servers to the EQ. The EQ is a really bad idea that we have to put up with.

The only way of addressing this issue is to do some smart thread management on the side of the opensim HTTP server, instead of relying blindly on mono's thread pool. This is now being considered.
dahlia (administrator)
2009-01-21 01:51

I tried it on my problem regions on osgrid (3x3) with a value of 30 per cpu and the problem disappeared, but I only tested with 3 avatars. I can try again later with more avatars if I can find several people who would be available to log in at the same time.
rtkwebman (reporter)
2009-01-21 07:30

How about settling on who, where, and when for a bunch of us to login to a simulator and test this?
Diva (administrator)
2009-01-21 08:01

Thanks for all the reporting on this.

It's pretty clear what's going on here, there's not much left to test for the purposes of opensim development. Each of us running openim in linux boxes, however, and for the time being, has to figure out what's the right number of MONO_THREADS_PER_CPU for our sims. This is not a straightforward number, because it depends on a variety of parameters -- like, for example, the number of CPUs on the machine, what else you're using that machine for, how script-heavy your sim is (I'm still to understand how the script engines work wrt to the thread-pool threads), how many regions you're running on it, how many of them are adjacent, how many other neighbors you have, how many avies you want to serve at the same time, etc etc. But clearly the default value (5) is really low and must be increased.

As a reference, Nebadon has set it to 125 on Wright Plaza. As far as I know, WP runs on a sim with just that region.

Neb has done that a long time ago, and that's why WP has been able to host the Office Hours without this problem showing up. Apparently everyone who knew about this parameter forgot to document it... But, to be fair, this parameter is now a lot more important because of the EQ threads, which are a massive clot on the mono thread pool.
Bartholomew Kleiber (reporter)
2009-01-21 08:16

Thanks to you all from me, too - this was bugging me for some time.

Just in case somebody wants to stresstest her magic number, I'd propose a meeting on a sim at

  UTC 9:30pm
which is
  London 9:30 pm
  San Francisco 1:30 pm
  New York 4:30 pm
  Cologne 10:30 pm

if I looked that up correctly.
justincc (administrator)
2009-01-21 09:47

Somewhat redundantly and just for the record, I can confirm that increasing the MONO_THREADS_PER_CPU does alleviate the problem for me as well :)
Diva (administrator)
2009-01-22 20:08

Going once, twice... gone.
This was a great mantis! Good thing to learn.
justincc (administrator)
2009-01-23 06:12

Could we keep this open until there is a solution that doesn't involve increasing MONO_THREADS_PER_CPU? Threads from this pool are not meant to be used for long running tasks (I would say that waiting for 15 seconds for input constitutes long running) and so this is unlikely to be resolved by future mono updates.
Diva (administrator)
2010-07-05 10:47

This issue has long been fixed since Teravus introduced pooling of the EQ threads.

- Issue History
Date Modified Username Field Change
2009-01-02 11:02 justincc New Issue
2009-01-02 11:02 justincc SVN Revision => 7917
2009-01-02 11:02 justincc Run Mode => Standalone (Multiple Regions)
2009-01-02 11:02 justincc Physics Engine => BasicPhysics
2009-01-02 11:02 justincc Environment => Mono / Linux32
2009-01-02 11:02 justincc Mono Version => 1.9.1
2009-01-02 11:04 justincc File Added: mantis-2946-comment-out-eac-on-login.patch
2009-01-02 11:04 justincc Note Added: 0008590
2009-01-02 11:07 justincc Note Added: 0008591
2009-01-02 11:15 justincc Note Added: 0008592
2009-01-02 11:20 justincc Note Added: 0008593
2009-01-02 13:09 justincc Note Added: 0008594
2009-01-02 13:13 Diva Note Added: 0008595
2009-01-02 13:51 jhurliman Note Added: 0008596
2009-01-02 15:22 HomerHorwitz Note Added: 0008597
2009-01-02 15:46 thomax Note Added: 0008598
2009-01-03 10:47 HomerHorwitz Note Added: 0008600
2009-01-03 10:49 justincc Note Added: 0008601
2009-01-03 10:52 justincc Note Edited: 0008601
2009-01-03 11:47 HomerHorwitz Note Added: 0008603
2009-01-03 12:06 Diva Note Added: 0008604
2009-01-03 12:12 HomerHorwitz Note Added: 0008605
2009-01-03 12:22 Diva Note Added: 0008606
2009-01-03 12:46 FrankNichols Note Added: 0008607
2009-01-03 13:22 Diva Note Added: 0008608
2009-01-05 06:17 justincc Note Edited: 0008593
2009-01-05 06:47 justincc Note Added: 0008615
2009-01-05 06:50 justincc Note Added: 0008616
2009-01-08 09:51 carlosroundel Note Added: 0008671
2009-01-08 12:36 master zephyr Note Added: 0008677
2009-01-08 12:54 master zephyr Note Edited: 0008677
2009-01-10 09:36 master zephyr Note Added: 0008698
2009-01-17 08:16 master zephyr Note Deleted: 0008698
2009-01-17 08:18 master zephyr Note Deleted: 0008677
2009-01-19 10:47 Diva Note Added: 0008873
2009-01-19 14:38 Diva Note Added: 0008880
2009-01-19 15:10 Diva Note Added: 0008881
2009-01-19 18:02 Diva Status new => confirmed
2009-01-19 20:04 rtkwebman Note Added: 0008884
2009-01-20 08:21 rtkwebman Note Added: 0008888
2009-01-20 09:46 Diva Note Added: 0008889
2009-01-20 10:23 justincc Note Added: 0008890
2009-01-20 10:52 justincc Note Added: 0008894
2009-01-20 10:54 Diva Note Added: 0008895
2009-01-20 11:06 Diva Note Added: 0008896
2009-01-20 11:13 FrankNichols Note Added: 0008898
2009-01-20 11:37 Diva File Added: thread-dump-1.txt
2009-01-20 11:37 Diva Note Added: 0008899
2009-01-20 11:53 Diva Note Added: 0008900
2009-01-20 12:55 Diva Note Added: 0008901
2009-01-20 14:03 Bartholomew Kleiber Note Added: 0008904
2009-01-20 14:13 Bartholomew Kleiber Note Added: 0008905
2009-01-20 14:47 rtkwebman Note Added: 0008906
2009-01-20 14:55 Bartholomew Kleiber Note Added: 0008907
2009-01-20 22:11 Bartholomew Kleiber Note Added: 0008908
2009-01-20 22:25 Diva Note Added: 0008909
2009-01-21 01:51 dahlia Note Added: 0008910
2009-01-21 07:30 rtkwebman Note Added: 0008913
2009-01-21 08:01 Diva Note Added: 0008914
2009-01-21 08:02 Diva Note Edited: 0008880
2009-01-21 08:16 Bartholomew Kleiber Note Added: 0008915
2009-01-21 09:47 justincc Note Added: 0008917
2009-01-22 20:08 Diva Note Added: 0008966
2009-01-22 20:09 Diva Status confirmed => resolved
2009-01-22 20:09 Diva Resolution open => fixed
2009-01-22 20:09 Diva Assigned To => Diva
2009-01-23 06:12 justincc Status resolved => feedback
2009-01-23 06:12 justincc Resolution fixed => reopened
2009-01-23 06:12 justincc Note Added: 0008976
2009-01-23 06:12 justincc Assigned To Diva =>
2009-01-23 06:12 justincc Status feedback => confirmed
2010-07-05 10:47 Diva Status confirmed => resolved
2010-07-05 10:47 Diva Resolution reopened => fixed
2010-07-05 10:47 Diva Assigned To => Diva
2010-07-05 10:47 Diva Note Added: 0015892
2010-09-12 10:47 chi11ken Status resolved => closed

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker