Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007389opensim[REGION] OpenSim Corepublic2014-12-03 13:322016-08-16 13:40
Reporterkenvc 
Assigned To 
PrioritynormalSeveritycrashReproducibilityalways
StatusconfirmedResolutionopen 
PlatformDual quad core 32 gb ramOSWindows 64 bitOS VersionServer 2012
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0007389: Teleports between 2 vars is broke for some AVs & TPs away from some vars to normal sim often causes long viewer freeze or crash.
Description1. Teleporting away from a heavy loaded var to a normal sim causes a very long Viewer freeze or crash on some AVs and not when using other AVs.
2. The Longer you stay on the var before teleporting, the longer the freeze is.
3. If you teleport very quickly back and forth between the var and normal sim, the teleports work many times with no issues and no freeze at all.
4. Teleports between 2 vars do not appear to be working at all. The TP progress bar acts as though the destination var does not exist and it eventually times out.
5. The more prims and scripts that are on the var the worse the issue is that's described in point 1.
Steps To ReproduceTeleport to a sim in OSGrid called Commonwealth Estate, stay there a while looking around, and then TP away to some other normal sim. The TP progress bar will often freeze for a long time, or the viewer will freeze a few seconds after the TP completes. The longer you stay there the longer the freeze happens. It seems to do this on about 90% of the people who go there.

Teleports between 2 vars for some AVs does not work at all and eventually results in a timeout TP failed message.
Additional InformationThis issue has existed for quite a while and is not really new, but as vars get more loaded with prims and scripts it is becoming more and more an issue.
TagsNo tags attached.
Git Revision or version numberr/25589
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
Script Engine
Environment.NET / Windows64
Mono VersionNone
ViewerLatest Singularity Alpha
Attached Fileslog file icon Commonwealth.log [^] (43,186 bytes) 2014-12-21 18:03
log file icon Commonwealth-2.log [^] (113,773 bytes) 2014-12-22 06:32
log file icon Commonwealth3.log [^] (66,808 bytes) 2014-12-23 18:19

- Relationships
related to 0007556closedUbitUmarov Teleporting in to 4096x4096 var fails 
related to 0007752new Teleports out of var regions with a lot of prims/textures often fails with viewer timeout and viewer crash. Viewer problem??? 

-  Notes
(0027059)
justincc (administrator)
2014-12-03 18:13

Have you reported this to the Singularity team?
(0027060)
kenvc (reporter)
2014-12-03 18:16

No, I have not. Should I ask the users who reported it to try other Viewers to make sure whether this is coming from the Viewer or Opensim. If so, which one do you suggest?
(0027063)
justincc (administrator)
2014-12-04 11:12

It's always best to try other viewers in these situations. I would suggest trying Firestorm.

I say this because varregions is experimental and you are running particularly large neighbouring ones. I think it most likely you have hit some viewer issue. A situation shouldn't really be allowed to arise where the simulator can ouright crash the viewer and I'm not sure how we could fix such a thing on the simulator end.
(0027069)
kenvc (reporter)
2014-12-05 08:52

Justin, I requested those who were reporting this to me to try at least one other Viewer with Firestorm being one of them. I'll report back the results when I receive them.
(0027126)
kenvc (reporter)
2014-12-21 04:39
edited on: 2014-12-21 10:41

We have been experimenting with this now for almost 2 weeks using several different browsers and from several different locations and, speeds, and types of internet connections. There seems to be no pattern to this, but in general Firestorm crashes more often than Singularity given the same conditions when tested within a few minutes of each other. Sometimes a TP from North to South works and sometimes it doesn't. The most common symptom is that the TP appears to complete, but several seconds after rezing the viewer just freezes, the screen turns a transparent white color, and stops responding.

(0027128)
kenvc (reporter)
2014-12-21 14:24
edited on: 2014-12-21 18:02

Attached is a log file that covers from the point where I login to Commonwealth North, wait a short time until the scene is fully visible, and then I teleport to Commonwealth South. In this particular instance, the teleport to South appeared to work but before everything was rezed, the viewer froze for an extended period of time. During this time, the lag meter in the viewer was to the top red and the AV could not move and the scene was frozen. A while later the screen changed to a transparent white color and stayed that way for a while, then suddenly things seemed to return to normal. Usually it does not ever recover when this happens, but in this case it eventually did recover.

Shortly after that, I logged off and the Viewer froze again during the logoff process and then windows displayed a message that the viewer was not responding and it had to be forcefully closed. I don't know if this final logoff freeze was related to the previous freeze up or not.

(0027129)
kenvc (reporter)
2014-12-21 18:15

The viewer froze for 56 seconds. This corresponds with the lapse of time between the log entries with the following dates and times:
2014-12-21 16:05:21,376 ERROR...
2014-12-21 16:06:17,614 DEBUG...

Included below is the detail of those 2 log entries with a space between them.

2014-12-21 16:05:21,376 ERROR - OpenSim.Framework.Servers.HttpServer.BaseHttpServer [BASE HTTP SERVER]: HandleRequest() threw exception
System.NullReferenceException: Object reference not set to an instance of an object.
   at OpenSim.Region.CoreModules.World.Media.Moap.MoapModule.CheckUrlAgainstWhitelist(String rawUrl, String[] whitelist) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\CoreModules\World\Media\Moap\MoapModule.cs:line 600
   at OpenSim.Region.CoreModules.World.Media.Moap.MoapModule.HandleObjectMediaNavigateMessage(String request, String path, String param, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\CoreModules\World\Media\Moap\MoapModule.cs:line 528
   at OpenSim.Framework.Servers.HttpServer.RestStreamHandler.ProcessRequest(String path, Stream request, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Framework\Servers\HttpServer\RestStreamHandler.cs:line 60
   at OpenSim.Framework.Servers.HttpServer.BaseStreamHandler.Handle(String path, Stream request, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Framework\Servers\HttpServer\BaseStreamHandler.cs:line 67
   at OpenSim.Framework.Servers.HttpServer.BaseHttpServer.HandleRequest(OSHttpRequest request, OSHttpResponse response) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Framework\Servers\HttpServer\BaseHttpServer.cs:line 559

2014-12-21 16:06:17,614 DEBUG - OpenSim.Region.CoreModules.World.WorldMap.MapSearchModule [MAPSEARCHMODULE]: search commonwealth south returned 1 regions. Flags=2
(0027131)
kenvc (reporter)
2014-12-22 06:40
edited on: 2014-12-22 07:21

Another log file has been attached from this morning. The exact same MOAP error referencing a whitelist with the long freeze is in this log just like the last one from yesterday. MOAP is working fine on all the sims that were teleported to during this log session. Media filtering is set to OFF in the viewer in case that info is helpful. Singularity and Firestorm both seem to have very similar behavior with the long freezes and crashes.

All teleports worked this session, but some with long freezes in the viewer until the very end, and then the viewer froze completely and had to be forcefully closed.

(0027135)
justincc (administrator)
2014-12-23 13:01

The Moap issue is certainly an OpenSimulator problem, though I've not seen it before so enabling the whitelist with a null list (instead of the expected empty list) must be rare. It looks like it might be possible through scripting, at least.

I've fixed this particular issue in master fc940ab so please try that. I'm surprised to hear a failure freezes the viewer (which I would actually call a viewer bug!) so I would be very interested if this now goes away. However, I expect this to be a separate issue from the boarder problem of crashes when teleporting between var regions.
(0027137)
kenvc (reporter)
2014-12-23 17:48

Justin, about 120 sims are being updated now and I hope this fixes the problem. But, the following is the only thing in any of the opensim.ini files related to MOAP, and there is no whitelist enabled here or anywhere else on any sims including the ones mentioned in this mantis:

[MediaOnAPrim]
    ; Enable media on a prim facilities
    Enabled = true;
(0027138)
kenvc (reporter)
2014-12-23 18:19
edited on: 2014-12-23 22:09

Just tested the new code. The previous error message related to MOAP appears to be gone, but there was still a significant viewer freeze a few seconds after teleports, but it did eventually recover each time without a crash. The viewer did freeze and crash when I tried to log off. The viewer freeze appears to be for a shorter period now than it was before the last code change. Was using Singularity for these tests because Firestorm overall seemed to have more issues with TP and map loading with var regions.

A new log file has been attached called Commonwealth3.log that starts with signing into one of the vars and teleporting between the North and South vars 3 or 4 times, then logging off. A Central var of the same size separates the North and South vars.

(0027139)
kenvc (reporter)
2014-12-24 20:53

Maybe should mention that the more recent tests with the viewer were done on the same LAN as where the sim servers and Robust resides, so I doubt if there was any packet dropping going on like there would be on slower internet connections.
(0027653)
kenvc (reporter)
2015-02-28 14:55

Several people from #OSGrid IRC (including Neb) helped test this issue today...

Most everyone on Windows or Linux had a freeze of varying lengths of time during the TP away from the var, but most of them eventually unfroze and the TP finished.

One Windows user (Ideapad y510p. i7 quad core, 8g ram, Nvidia 750m GPU) did not have the freeze at all.

Another person with OSX and 32 bit singularity (i5, 32GB ram and Nvidia GTX 760) also had no freezes at all at 256 draw distance. TPs for them rarely took more that 3 seconds using 256 or 1024 draw distance. With FS, this same person froze for 10 seconds, then it took another 8-10 to finish the TP at 256 draw distance.

This seems to indicate the issue may be OS and/or hardware related at the viewer end with Singularity performing MUCH better than FS, It seems to be happening when opengl is clearing all the data from the sim you are leaving.
(0027756)
kenvc (reporter)
2015-03-06 09:20
edited on: 2015-03-06 09:40

Another interesting discovery is that you cannot teleport directly from one var to another. You have to teleport to a normal 256x256 sim first, then teleport to the var. I have not tried this on a var with no prims or scripts, but the ones I tried it on had less than 5000 prims. In general var teleports are a mess. Updated mantis description to reflect this new issue.

When teleporting from one var to another, it seems that the destination sim never responds or something because the teleport progress bar never moves forward and eventually the teleport times out... but at least it does not crash the viewer like the original issue reported in this mantis.

(0027757)
BlueWall (administrator)
2015-03-06 09:41
edited on: 2015-03-06 09:41

I am able to teleport at will between any combination of region sizes. Linux x64 Singularity with a very low end and old graphics adapter. Draw distance 1024.

-- edit --
This with either local or Hypergrid user.

(0027758)
kenvc (reporter)
2015-03-06 11:40

OK here is another really odd thing. I signed in with an AV that has not been used in a long time. It has twice as many inventory items as my normal one. Wearing exactly the same clothes, logging in from the same computer with the same viewer, this AV is having no issues with teleports and no issues with lag when flying planes on the vars. It seems something about my main AV is very different than the other AV. Others have noticed this same thing.

How can this possibly be?
(0027947)
kenvc (reporter)
2015-03-26 18:52

I am still getting many reports that users are unable to teleport between 2 vars on any vars that I am hosting on OSG. Teleports normally work from a normal sim to a var with no issues at all. Teleports from a var to a normal sim are usually working if you wait long enough. Teleporting from Var to var is impossible.

Is there any setting anywhere that could explain this behavior that appears to only affect vars?
(0028300)
butcharnold (reporter)
2015-05-08 08:59

I am seeing an issue with connected vars regions of the same size.
If I connect 2 4x4 var regions.. I cannot tp a friend from one of the vars to another... when the friend receives the tp invite and clicks to tp.. they simply timeout.
Moving the var regions apart by one normal region space (256x256) then allows this to work as it should.
(0028516)
Luisillo_Contepomi (reporter)
2015-06-01 10:57

I see When avatar crash is a udp time out then I setup my OpenSim.ini

[ClientStack.LindenUDP] section in OpenSimDefaults.ini have AckTimeout = 60

I add in my OpenSim.ini
AckTimeout = 180

Now avatars feel a stop by some seconds when teleport but never crash.
(0028541)
kenvc (reporter)
2015-06-02 09:19

That's great info to know about increasing the timeout, but the real solution needs to address this long UDP freeze rather than just putting a bandaid on the problem.
(0028545)
Luisillo_Contepomi (reporter)
2015-06-02 11:30

I am agree kenvc,
The server stops receiving the client information.
Packet lost?¿ from 70 to 90 seconds.
For the server the user is not in the simulator then is a "ghost connection" and close it.
I do test with Firestorm and Singularity (last versions) in two Vars 1024x1024 totally empty. No prims no scripts and the result is the same. Server cut connection with the client after 60 sec by time out.
(0028660)
kenvc (reporter)
2015-06-12 17:58

Users are telling me the following 2 additions to the opensim.ini [ClientStack.LindenUDP] section on the sims I am hosting appear to have solved most var region TP issues:

    async_packet_handling = false
    AckTimeout = 180

The simulators are all running on Windows, so I cannot say if this will help with this issue on other platforms. If others try these settings, please report your results here!
(0028661)
Luisillo_Contepomi (reporter)
2015-06-13 00:49

Tested sucefully on a Linux Centos 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64 with mono 2.10.8

 async_packet_handling = false
    AckTimeout = 180

Note "AckTimeout" is not by default in OpenSim.ini, you must add it in [ClientStack.LindenUDP] section.

Is by default in OpenSimDefault.ini with value 60
(0028662)
kenvc (reporter)
2015-06-13 04:26

Luisillo, did this last change seem to significantly reduce the length of the long viewer freeze you were seeing like it did on mine, or did it just prevent the long freeze from causing the viewer to crash?
(0028663)
Ken_S (reporter)
2015-06-13 04:39

These last 2 settings changes appear to have also solved the issue of the viewer freezing up fairly often when trying to logoff causing the viewer to be very difficult or very slow to close.
(0028664)
kenvc (reporter)
2015-06-13 05:04

If both of these issues are solved with these 2 simple opensim.ini settings changes it will be great!!! The long viewer freeze or crash issue right after TP and right after logoff has been a nagging issue as long as I can remember.
(0028665)
cinderblocks (reporter)
2015-06-13 07:02
edited on: 2015-06-13 07:03

I have noticed a behavior where on teleport, a simulator misjudges (through calling IsOutsideView()) that the sending region has a child agent on it. Many times the simulator assumes a child agent has been created because it expects the viewer has a connection with the other region when, in fact, it does not. So once the teleport has begun, it tries to get the seed cap from the neighbor region which simply doesn't have it. This leads to a null string and a bad capability url, which can cause all sorts of delightful problems. (Failed teleports, most often, no mesh loading, no http texture loading, anything requiring a sim cap really because the viewer is trying to get caps from a bogus url!)

I don't know if this is related to this problem, but it might be.

(0028675)
Luisillo_Contepomi (reporter)
2015-06-13 10:30

kenvc with this changes in OpenSim.ini have not difference with the tps as was in the regular regions (256x256) Viewer no crash. No messages disconnecting user in log console.
(0028678)
kcozens (administrator)
2015-06-13 12:17

Thank you for the INI setting, kenvc. I made the change in a grid running OpenSim 0.8.0-post-fixes where users were often crashing after a TP from a var to a regular region. The feedback I've received is they are no longer having their viewer crash or permanently lock up after a TP from var to regular. I find my viewer seems to freeze for up to 20 seconds or so but it eventually unfreezes with no ill effects noticed.
(0028736)
kenvc (reporter)
2015-06-18 13:16

After using the 2 ini settings mentioned in a previous note, has anyone still had any of the previous TP crashes related to TPing from var regions? Please provide feedback and details either way.

I still get a freeze that is directly related to how long I stay in the var before TPing out, but at least the crashes have gone away for me.
(0028754)
Luisillo_Contepomi (reporter)
2015-06-19 13:55

async_packet_handling = false give problems with viewer when have more than one avatar. I can not explain in english because
It is a very strange behavior. Delays in the other see your animations or position.
I need do more test tomorrow morning.
By now I am using only AckTimeout = 180
(0028755)
Luisillo_Contepomi (reporter)
2015-06-20 03:35

About teleports crash in var regions:

Tested with true or false in "async_packet_handling" have not difference in teleports problem for me.

 async_packet_handling = true is the default and is good for me.
    

AckTimeout = 180 have a little delay when Teleport but is solved, no crash. but now I am detecting that the animations are not correctly sended to the other user viewers.

I am detecting other issue about udp I will open other mantis issue.
(0028817)
kenvc (reporter)
2015-06-25 15:35

I am no longer having TP crashes when teleporting from a var, but still having the terribly long viewer freeze that seems to be directly related with how much time I spent on the var. The longer I stay there before teleporting away, the longer the freeze.
(0028824)
Luisillo_Contepomi (reporter)
2015-06-26 08:04

Yes always more than 90 second.
(0028826)
kcozens (administrator)
2015-06-26 16:29

Something must be wrong if you have a 90 second delay. If I had that, I would have assumed the viewer was still locking up on a TP away from a var like it used to do. I wouldn't wait 90 seconds to see if the lock up was temporary. My experience is that the delay, if there is much of one, is under 20 seconds.
(0028841)
kenvc (reporter)
2015-06-29 06:09

kcozens, are you on Linux or Windows, and what video card are you using? We are trying to narrow down what systems have this free and which ones don't.
(0028855)
kcozens (administrator)
2015-06-30 05:59

The grid I am using Linux Mint 17 32-bit (but plan to switch to 17.2 64-bit soon). The viewer I use is mostly a fairly current version of Singularity Alpha. My video card is an nVidia GeForce 9500GT.

I recently experienced a viewer lock up for an extended period of time while I was using Firestorm Beta 4.7.1.45325 but I can't tell you if it (would have) fixed itself. The viewer was locked up for much longer than usual (upwards of a minute?). As I was going to shutdown the viewer after I reached my destination I just wound up killing my viewer while it was still frozen.
(0030538)
kenvc (reporter)
2016-06-16 05:50

This is still happening. Latest info about this seems to point to the Viewers and the way they all clear the previous scene from the display card's memory. Apparently this is a slow process when the prim count on the var is high and the AV has been on the sim long enough to load a lot of graphics info from the sim into the video card's memory.

It would be nioce to see a fix for this, but at this point it appears like it has to be done at the viewer end, and I hear it would mean a major change in the way the Viewer handles clearing the previous scene when you TP somewhere.
(0030541)
cinderblocks (reporter)
2016-06-16 05:54

It wouldn't be as much of an issue for the viewer if OpenSimulator supported distance based object culling. This would benefit not only varregion teleports, but frames per second in the viewer, and all around better scene loading as the simulator would only be sending viewable objects to the viewer, and culling them when they are out of range.
(0030804)
UbitUmarov (administrator)
2016-06-28 01:10

Yeap, but distance based object culling region side, is still far.
The solution used on aurora fork for example is now a bit harder to implement.
(0031033)
UbitUmarov (administrator)
2016-08-16 13:40

actually a crude culling by distance is now under test

ObjectsCullingByDistance = true in section [InterestManagement]

enables it
expect glitchs, etc, etc
also it is avatar position based only, not camera.
does not affect other avatars ( still )
it is a tradeoff between region cpu and network load versus viewer load.

- Issue History
Date Modified Username Field Change
2014-12-03 13:32 kenvc New Issue
2014-12-03 18:13 justincc Note Added: 0027059
2014-12-03 18:16 kenvc Note Added: 0027060
2014-12-04 11:12 justincc Note Added: 0027063
2014-12-05 08:52 kenvc Note Added: 0027069
2014-12-21 04:39 kenvc Note Added: 0027126
2014-12-21 10:41 kenvc Note Edited: 0027126 View Revisions
2014-12-21 14:24 kenvc Note Added: 0027128
2014-12-21 18:02 kenvc Note Edited: 0027128 View Revisions
2014-12-21 18:03 kenvc File Added: Commonwealth.log
2014-12-21 18:15 kenvc Note Added: 0027129
2014-12-22 06:32 kenvc File Added: Commonwealth-2.log
2014-12-22 06:40 kenvc Note Added: 0027131
2014-12-22 06:47 kenvc Note Edited: 0027131 View Revisions
2014-12-22 06:49 kenvc Summary Teleports between adjoining var regions almost always causes viewer crash. Walking across sometimes crashes too but not as often => Teleports between adjoining var regions often causes long viewer freeze or crash. Flying across borders sometimes crashes also.
2014-12-22 06:50 kenvc Description Updated View Revisions
2014-12-22 06:50 kenvc Steps to Reproduce Updated View Revisions
2014-12-22 07:21 kenvc Note Edited: 0027131 View Revisions
2014-12-23 13:01 justincc Note Added: 0027135
2014-12-23 17:48 kenvc Note Added: 0027137
2014-12-23 18:19 kenvc Note Added: 0027138
2014-12-23 18:19 kenvc File Added: Commonwealth3.log
2014-12-23 22:04 kenvc Note Edited: 0027138 View Revisions
2014-12-23 22:06 kenvc Note Edited: 0027138 View Revisions
2014-12-23 22:07 kenvc Note Edited: 0027138 View Revisions
2014-12-23 22:09 kenvc Note Edited: 0027138 View Revisions
2014-12-24 20:53 kenvc Note Added: 0027139
2015-02-12 19:13 kenvc Relationship added related to 0007428
2015-02-28 14:55 kenvc Note Added: 0027653
2015-02-28 14:56 kenvc Summary Teleports between adjoining var regions often causes long viewer freeze or crash. Flying across borders sometimes crashes also. => Teleports away from loaded var region often causes long viewer freeze or crash.
2015-02-28 15:02 kenvc Description Updated View Revisions
2015-02-28 15:02 kenvc Steps to Reproduce Updated View Revisions
2015-03-06 09:20 kenvc Note Added: 0027756
2015-03-06 09:31 kenvc Description Updated View Revisions
2015-03-06 09:31 kenvc Steps to Reproduce Updated View Revisions
2015-03-06 09:32 kenvc Description Updated View Revisions
2015-03-06 09:36 kenvc Summary Teleports away from loaded var region often causes long viewer freeze or crash. => Teleports between 2 vars is totally broke, & TPs away from loaded var to normal sim often causes long viewer freeze or crash.
2015-03-06 09:38 kenvc Relationship deleted related to 0007428
2015-03-06 09:39 kenvc Status new => confirmed
2015-03-06 09:40 kenvc Note Edited: 0027756 View Revisions
2015-03-06 09:41 BlueWall Note Added: 0027757
2015-03-06 09:41 BlueWall Note Edited: 0027757 View Revisions
2015-03-06 11:40 kenvc Note Added: 0027758
2015-03-07 10:42 kenvc Summary Teleports between 2 vars is totally broke, & TPs away from loaded var to normal sim often causes long viewer freeze or crash. => Teleports between 2 vars is broke for some AVs & TPs away from some vars to normal sim often causes long viewer freeze or crash.
2015-03-07 10:42 kenvc Description Updated View Revisions
2015-03-07 10:42 kenvc Steps to Reproduce Updated View Revisions
2015-03-07 10:42 kenvc Additional Information Updated View Revisions
2015-03-26 18:52 kenvc Note Added: 0027947
2015-05-06 20:50 kenvc Relationship added related to 0007556
2015-05-08 08:59 butcharnold Note Added: 0028300
2015-06-01 10:57 Luisillo_Contepomi Note Added: 0028516
2015-06-02 09:19 kenvc Note Added: 0028541
2015-06-02 11:30 Luisillo_Contepomi Note Added: 0028545
2015-06-12 17:58 kenvc Note Added: 0028660
2015-06-13 00:49 Luisillo_Contepomi Note Added: 0028661
2015-06-13 04:26 kenvc Note Added: 0028662
2015-06-13 04:39 Ken_S Note Added: 0028663
2015-06-13 05:04 kenvc Note Added: 0028664
2015-06-13 07:02 cinderblocks Note Added: 0028665
2015-06-13 07:03 cinderblocks Note Edited: 0028665 View Revisions
2015-06-13 10:30 Luisillo_Contepomi Note Added: 0028675
2015-06-13 12:17 kcozens Note Added: 0028678
2015-06-18 13:16 kenvc Note Added: 0028736
2015-06-19 13:55 Luisillo_Contepomi Note Added: 0028754
2015-06-20 03:35 Luisillo_Contepomi Note Added: 0028755
2015-06-25 15:35 kenvc Note Added: 0028817
2015-06-26 08:04 Luisillo_Contepomi Note Added: 0028824
2015-06-26 16:29 kcozens Note Added: 0028826
2015-06-29 06:09 kenvc Note Added: 0028841
2015-06-30 05:59 kcozens Note Added: 0028855
2015-11-27 08:16 kenvc Relationship added related to 0007752
2016-06-16 05:50 kenvc Note Added: 0030538
2016-06-16 05:54 cinderblocks Note Added: 0030541
2016-06-28 01:10 UbitUmarov Note Added: 0030804
2016-08-16 13:40 UbitUmarov Note Added: 0031033


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker