Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008518opensim[REGION] Scripting Enginepublic2019-04-13 07:232019-05-05 06:34
Reportermike.chase 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformLinuxOSUbuntu ServerOS Version18.04
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008518: Out of heap space with ZHAO AO under YEngine
DescriptionWhen running YEngine a ZHAO AO (<ZHAO-II-core7/29.1) will fairly quickly run out of heap space. See notes below but it appears the AO is repeatedly doing a list intersection operation that causes a memory leak under YEngine. I instrumented the intersection operation and can see heap slowly dropping until I get the OutOf memory exception.
Steps To ReproduceWear a ZHAO AO and enter a region running YEngine. Generally within 30 secs you'll get a message like:

[06:47] ZHAO AO Male Carl Bento AO: [YEngine]: Exception while running 6c37d80a-08d0-487b-92bd-ccc75631ffb0
[06:47] ZHAO AO Male Carl Bento AO: OutOfHeapException: oldtotal=262063, newtotal=262207, limit=262144
[06:47] ZHAO AO Male Carl Bento AO: Prim: <ZHAO AO Male Carl Bento AO>, Script: <ZHAO-II-core7/29.1>, Location: Utopia Skye Welcome <128,128,22>
[06:47] ZHAO AO Male Carl Bento AO: Script must be Reset to re-enable.
[06:47] ZHAO AO Male Carl Bento AO: at OpenSim.Region.ScriptEngine.Yengine.XMRInstAbstract.UpdateHeapUse (System.Int32 olduse, System.Int32 newuse) [0x0006f] in <3c821c2957904485b22e58fcda9b1b0a>:0
[06:47] ZHAO AO Male Carl Bento AO: at OpenSim.Region.ScriptEngine.Yengine.HeapTrackerList.Save (OpenSim.Region.ScriptEngine.Shared.LSL_Types+list lis) [0x00008] in <3c821c2957904485b22e58fcda9b1b0a>:0
[06:47] ZHAO AO Male Carl Bento AO: at (wrapper dynamic-method) System.Object.hasIntersection(list,list)(XMRInstanceSuperType,OpenSim.Region.ScriptEngine.Shared.LSL_Types/list,OpenSim.Region.ScriptEngine.Shared.LSL_Types/list)
[06:47] ZHAO AO Male Carl Bento AO: at (wrapper dynamic-method) System.Object.checkAndOverride()(XMRInstanceSuperType)
[06:47] ZHAO AO Male Carl Bento AO: at (wrapper dynamic-method) System.Object.default timer(OpenSim.Region.ScriptEngine.Yengine.XMRInstAbstract)
[06:47] ZHAO AO Male Carl Bento AO: at OpenSim.Region.ScriptEngine.Yengine.XMRInstAbstract.CallSEH () [0x00050] in <3c821c2957904485b22e58fcda9b1b0a>:0
[06:47] ZHAO AO Male Carl Bento AO: at OpenSim.Region.ScriptEngine.Yengine.XMRInstance.StartEx () [0x00009] in <3c821c2957904485b22e58fcda9b1b0a>:0
Additional InformationThe offending code is below. The printFreeMemory call is the instrumentation
I added. When running with the first function you can steadily see the memory declining until it runs out.

// Find if two lists/sets share any elements in common
integer hasIntersection( list _list1, list _list2 )
{
    printFreeMemory();

    list bigList;
    list smallList;
    integer smallListLength;
    integer i;

    if ( llGetListLength( _list1 ) <= llGetListLength( _list2 ) ) {
        smallList = _list1;
        bigList = _list2;
    }
    else {
        bigList = _list1;
        smallList = _list2;
    }
    smallListLength = llGetListLength( smallList );

    for ( i=0; i<smallListLength; i++ ) {
        if ( llListFindList( bigList, llList2List(smallList,i,i) ) != -1 ) {
            return TRUE;
        }
    }

    return FALSE;
}

Interesting to note... Recoding this as:

// Find if two lists/sets share any elements in common
integer hasIntersection( list _list1, list _list2 )
{
    printFreeMemory();

    list lz = []; integer x;
    for (x = 0; x < llGetListLength(_list2); x++) {
        if (~llListFindList(_list1,llList2List(_list2,x,x))) {
            lz = lz + llList2List(_list1,x,x);
        }
    }

    if (llGetListLength(lz) > 0)
        return TRUE;
    else
        return FALSE;
}

does not exhibit the issue so its something in the order of operations in the first case that triggers it. SO there is a script workaround but I dare say its impossible to change all the ZHAO AO's floating around the hypergrid.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim)
Physics EngineubODE
EnvironmentMono / Linux64
Mono VersionOther
ViewerFirestorm 6.0.1
Attached Files

- Relationships

-  Notes
(0035087)
BillBlight (developer)
2019-04-13 07:33

Easy, increase the stacksize/heapsize ..

[YEngine]
    Enabled = true
    ScriptStackSize = 512
    ScriptHeapSize = 512
    UseSourceHashCode = true
(0035088)
BillBlight (developer)
2019-04-13 07:37

I was being mildly sarcastic, sorry for that, yes it does appear they may be a memory issue, in YEngine where big lists are concerned , found this myself, but was able to work around it by increasing the stack size. This does not use a noticeable amount of extra memory but does allow the engine to have some headroom, until it recovers, which it does.
(0035089)
mike.chase (reporter)
2019-04-13 08:24

No it doesn't always recover. I've played with stack settings and it still sometimes fails though less reliably. And NO I'm not going to hack around it by increasing stack. The GC isn't running reliably or isnt collecting memory in some cases and thats a bug. Good to know there's a workaround if you just want to run YEngine. I did the report because I want to see the bug get fixed. I'm going to try and debug it in a test system. I already have narrowed it down to list operations so I just need to find the call. It's VERY easy to repro so it shouldn't be hard. But maybe other like uBit have some ideas.
(0035090)
BillBlight (developer)
2019-04-13 08:34

Ok, just saying I have zhao all over the place and they don't crash, but I agree the bug should be corrected ..

But then again what do I know.
(0035094)
mike.chase (reporter)
2019-04-13 10:10

Again that's helpful to note but there is still a bug in list handling that needs to get fixed. Hence the bug report.
(0035095)
BillBlight (developer)
2019-04-13 10:12

Yes as I said I agree it needs to be fixed, I'll have to roll back my heap to defaults and see if I can make it happen again, because I just tried to replicate it and can't on my current config, so will have to roll over to a testing environment, to help track this down.
(0035096)
UbitUmarov (administrator)
2019-04-13 12:20

ZHAO and similar AOs, is something people should just delete.
replacing by things that do use server side AO (like the one Bill is actually using)

but that piece of code can be anywhere, so ill try to understand what is going on...
(0035097)
UbitUmarov (administrator)
2019-04-13 15:27
edited on: 2019-04-13 15:28

ok some debug results.

 smallList = _list1 adds to memory usage. It should not.

memory usage is released, but that is done by GC on a class finalizer.
on current runtimes, that can mean AGES.
gc will allocate fast, and deallocate very slow, and finalizers a lot slower.

this are original flaws on xmr, not easy to improve now :(

this is not a issue on X, just because X as no memory usage control

(0035098)
UbitUmarov (administrator)
2019-04-13 15:44
edited on: 2019-04-13 16:10

a test case:
list bb = ["ASD",8];

default
{
    state_entry()
    {
        llSay(0, "Script running");
    }
    touch_start(integer jjj)
    {
        integer mem = llGetUsedMemory();
        llOwnerSay("s mem: "+(string)mem);
        //list aa = ["ASD",8];
        list aa = bb;
        integer mem2 = llGetUsedMemory();
        llOwnerSay("e mem: "+(string)mem2+" "+(string)(mem2 -mem));
    }
}

result doing touchs:
[15:37] Object: s mem: 7
[15:37] Object: e mem: 21 14
...
[15:38] Object: e mem: 609 14
[15:38] Object: s mem: 7
[15:38] Object: e mem: 21 14


the time to "release" can be a lot more, and it is totally async, when gc feels like..

(0035099)
UbitUmarov (administrator)
2019-04-13 15:47

of course, this is true memory usage, since the memory is only free when GC kicks in, a problem we have all over opensim code with crap GC doing this kind of pseudo memory leaks (and even real ones).
(0035100)
BillBlight (developer)
2019-04-13 16:35
edited on: 2019-04-13 16:41

I don't really see this happen, I can MAKE it happen if I just start copying lists like a mad man ..

I do use the following on linux ..

export MONO_GC_PARAMS="nursery-size=16m,promotion-age=10,minor=split,major=marksweep,no-lazy-sweep,alloc-ratio=30"

In my startup script...


And I mean the heap exception, not the excessive memory usage, that is very strange.
No help to windows users though

(0035101)
BillBlight (developer)
2019-04-13 16:48
edited on: 2019-04-13 16:49

Ok to better clear up what I said in my last comment,


Yes I see the GC not cleaning the memory, but even that seems a little faster with the GC_PARAMS I have set .

Can't make it give an exception, in that case ..

Uses more memory than it should.

So still an issue, but since this was about the heap exception, I am reporting that I can't make that exception happen at this point.

Just because I can't break it does not mean it is not a bug that needs to be fixed, just fishing for ways that stop it from happening.

Since XEngine did not have memory management, I can fully understand why this would happen with that particular script and why I always saw those AOs cause crazy script usage with XEngine because they just ran away with memory.

(0035103)
mike.chase (reporter)
2019-04-14 07:16

So this is sort of what I expected. Short term memory pressure and the GC doesnt kick in quickly enough. The problem is anything with lists that creates short term memory demand that can't be met could trigger this. uBit is it not possible to manually call GC and then retry the allocation when the error occurs? I'm just not at all familiar with how the heaps are managed in YEngine
(0035104)
UbitUmarov (administrator)
2019-04-15 04:45

I have a workaround, just needs a little retouch before I commit
(0035105)
UbitUmarov (administrator)
2019-04-15 04:53

in fact that call to gc.collect has been there, but gc can't be used like that. it is just a request, not guaranteed to happen or when, and may have nasty side effects
(0035106)
mike.chase (reporter)
2019-04-15 06:08

So I expect each thread gets a private "heap" used for the stack and real heap for the script. There is some tie in between memory allocated in that heap and real GC and because real heap memory is stuck waiting for a finalizer to run its showing in use in the managed heap for the script? Is that on target at all? If so can you point me to the code that is managing that private heap?
(0035107)
UbitUmarov (administrator)
2019-04-15 16:43

made several changes on master, using a pseudo free, since GC just can't be used.
this is still very far from good memory accounting, only considers a limited set.
(0035108)
mewtwo0641 (reporter)
2019-04-15 23:51

As of master, a lot of my scripts are now crashing with errors that look like this: [23:41] Object: OutOfHeapException: oldtotal=1048453, newtotal=1048804, limit=1048576

where previously these scripts worked just fine
(0035109)
BillBlight (developer)
2019-04-16 00:13

@mewtwo0641, XEngine or YEngine, example script please ...
(0035110)
BillBlight (developer)
2019-04-16 00:15

I'm hoping you are using YEngine as the latest changes did not touch Xengine
(0035111)
mewtwo0641 (reporter)
2019-04-16 00:28

@Bill - Sorry, I figured YEngine was implied since this mantis pertains to YEngine, but yes, I am using YEngine.

This is the simplest script that I could find that triggers the crash. It keeps track of visitors to the region. On default settings it will crash around 45 mins to an hour of run time.


float refreshTime = 0.2;

list visitorList = ["Test User"];

init()
{
    llSetTimerEvent(refreshTime);
}

checkAvs()
{
    list agents = llGetAgentList(AGENT_LIST_REGION, []);
    
    integer i = 0;
    
    for(i; i < llGetListLength(agents); i++)
    {
        key curAgent = llList2Key(agents, i);
        integer find = llListFindList(visitorList, [llKey2Name(curAgent)]);
        
        if(find == -1 && !osIsNpc(curAgent))
        {
            //If avatar is found then do stuff here
        }
    }
}

string getTime(integer seconds)
{
    // timeFormat list must be constructed in the order days, hours, minutes, seconds
    list timeFormat = ["d, ", "h, ", "m, ", "s"];
    
    string timeDays = (string)(seconds/86400) + llList2String(timeFormat, 0);
    string timeHours = (string)((seconds%86400)/3600) + llList2String(timeFormat, 1);
    string timeMinutes = (string)(((seconds%86400)%3600)/60) + llList2String(timeFormat, 2);
    string timeSeconds = (string)(((seconds%86400)%3600)%60) + llList2String(timeFormat, 3);
    
    if(seconds / 86400 > 0) return timeDays + timeHours + timeMinutes + timeSeconds;
    else if(((seconds % 86400) / 3600) > 0) return timeHours + timeMinutes + timeSeconds;
    else if(((seconds % 86400) % 3600) / 60) return timeMinutes + timeSeconds;
    else if((((seconds % 86400) % 3600) % 60) >= 0) return timeSeconds;
    else return timeDays + timeHours + timeMinutes + timeSeconds;
}

default
{
    state_entry()
    {
        init();
    }
    
    timer()
    {
        llSetText("Region Uptime: " + getTime((integer)llGetTime()), <0,1,0>, 1.0);
        checkAvs();
    }
    
    changed(integer change)
    {
        if(change & CHANGED_REGION_START)
        {
            llResetTime();
            init();
        }
    }
}
(0035112)
BillBlight (developer)
2019-04-16 00:31

what is your YEngine settings .. Just out of curiostity


example:
[YEngine]
    Enabled = true
    ScriptStackSize = 256
    ScriptHeapSize = 256
(0035113)
mewtwo0641 (reporter)
2019-04-16 00:33

@Bill - My YEngine settings are set like this:

[YEngine]
    Enabled = true
    ScriptStackSize = 1024
    ScriptHeapSize = 1024

I made a slight mistake on my previous post that on default settings it would crash around 45 mins to an hour of script runtime, but that time frame is actually for the settings listed above. At default settings it would likely crash sooner than that.
(0035114)
BillBlight (developer)
2019-04-16 00:50
edited on: 2019-04-16 00:55

I'm currently running this script at a .02 timer interval, up to 3000 items in the list (modded it real quick so it detects me and adds me to the list each time)

And it has failed to crash, but I can see if it was a runaway and not actually doing what it is supposed to do it would crash ..

managed to crash my viewer from filling up chat script was still going ..

So I'm not sure of the conditions causing it to crash for you all of a sudden ..
 

Guess we will have to wait for Ubit to take a look ..


BlightBill BlightBill BlightBill BlightBill BlightBill BlightBill BlightBill BlightBill BlightBill BlightBill BlightBill Bligh
[00:53] Object: List Length 4035
 and Memory is 993133

and still not crashed

(0035115)
BillBlight (developer)
2019-04-16 00:56

Ohhhh I bet I know what it is ...

from the commit

"fix memory usage on reset. This changes will only take effect on new compiles"


You need to delete your script cache ..
(0035116)
BillBlight (developer)
2019-04-16 01:02
edited on: 2019-04-16 01:04

If that is not it I'm really confused, I'm adding to the list every .02 seconds, and in 7 minutes am at [01:01] LISTTEST: List Length 4000
 and Memory is 925172

and free mem is still 925172 so for you to get to 1048804 that script is eating a lot more mem than making a list ...

(I set my heap to the same as your current for testing, which is why I wanted to know what it was)

[01:04] LISTTEST: List Length 5600
 and Memory is 938948

(0035117)
mewtwo0641 (reporter)
2019-04-16 01:25

I just tried clearing my script folders but the scripts are still crashing for me and also tried recompiling the scripts from within the viewer but still ends up crashing eventually. If I set the ScriptStackSize and ScriptHeapSize to some insanely high number like 9999999 then it seems to not crash but I suspect doing that is only delaying it rather than fixing the issue. I noticed that modifying the script to toggle the timer off for a while and then toggling it back on (without a script reset or starting/stopping the script entirely) doesn't seem to give things a chance to "breathe".

It seems to me that it almost acts if memory is accumulating (I can't say for sure whether on the stack or the heap) during the operations (or not being released fast enough before more is added), but is never released after losing scope of its function, so it adds on more and more until script crash. This is only speculation though.
(0035118)
BillBlight (developer)
2019-04-16 01:29

Might help to throw some debug in there and see if the list is just growing and growing ..

Using your example, I got [01:05] LISTTEST: List Length 6400
 and Memory is 944996 and that is free mem, so I'm really not sure where that is running out of mem for you, at that rate it would have to be a list over 100k to be running out of mem.

(and not debating the problem, just trying to figure out where it is coming from)
(0035119)
mewtwo0641 (reporter)
2019-04-16 01:41

I understand, it's proving to be quite a tricky one lol.

Added some lines in the script to see what the list sizes look like

llSay(0, "Free Mem = " + (string)llGetFreeMemory());
llSay(0, "Visitor List Length = " + (string)llGetListLength(whitelist));
llSay(0, "Agents In Region = " + (string)llGetListLength(agents));

with the results of

[01:38] Visitor List: Free Mem = 261799
[01:38] Visitor List: Visitor List Length = 3
[01:38] Visitor List: Agents In Region = 4
[01:38] Visitor List: Free Mem = 261799
[01:38] Visitor List: Visitor List Length = 3
[01:38] Visitor List: Agents In Region = 4
[01:38] Visitor List: Free Mem = 261799
[01:38] Visitor List: Visitor List Length = 3
[01:38] Visitor List: Agents In Region = 4
[01:38] Visitor List: Free Mem = 261799
[01:38] Visitor List: Visitor List Length = 3
[01:38] Visitor List: Agents In Region = 4
[01:38] Visitor List: Free Mem = 261799
[01:38] Visitor List: Visitor List Length = 3
[01:38] Visitor List: Agents In Region = 4

... etc

So it doesn't appear that the lists are growing.

Just another guess but perhaps it's not directly related to the amount of data going into something and rather, maybe it's holding multiple copies of the object being made that garbage collection isn't getting to (either in a timely manner or not at all), but only referencing one? Which would explain why you never see unintended multiple list entries while the script still crashes and might also explain why stopping the timer never seems to reset memory consumption?
(0035120)
BillBlight (developer)
2019-04-16 01:47
edited on: 2019-04-16 01:48

not sure if (string)llGetListLength(whitelist)); is the actual visitor list, whitelist normally implies a security whitelist not a total list ..


Not that it matters without being able to duplicate the actual error with the exact script, hard to track down ..

(0035121)
mewtwo0641 (reporter)
2019-04-16 01:59

Ah sorry, I made those changes and tested on a different script.Here's the updated results on the correct script:

llSay(0, "Free Mem = " + (string)llGetFreeMemory());
llSay(0, "Visitor List Length = " + (string)llGetListLength(visitorList));
llSay(0, "Agents In Region = " + (string)llGetListLength(agents));

[01:55] Visitor List: Free Mem = 261987
[01:55] Visitor List: Visitor List Length = 1
[01:55] Visitor List: Agents In Region = 4
[01:55] Visitor List: Free Mem = 261987
[01:55] Visitor List: Visitor List Length = 1
[01:55] Visitor List: Agents In Region = 4
[01:55] Visitor List: Free Mem = 261987
[01:55] Visitor List: Visitor List Length = 1
[01:55] Visitor List: Agents In Region = 4
[01:55] Visitor List: Free Mem = 261987
[01:55] Visitor List: Visitor List Length = 1
[01:55] Visitor List: Agents In Region = 4

No fundamental change in results though.
(0035122)
UbitUmarov (administrator)
2019-04-16 02:21

well this does show the kind of bad scripts people have.. ;)
but well no data to debug the issue.. example code shows no "leak"
(0035123)
mewtwo0641 (reporter)
2019-04-16 02:39
edited on: 2019-04-16 02:43

Problem is, I don't know what's "bad" about the script I provided. There's nothing too crazy going on with it besides gathering a list of agents in region every 0.2 seconds which is probably admittedly not the best practice in the world, but I have previously never had crashing issues with it on YEngine either up until recent.

(0035124)
BillBlight (developer)
2019-04-16 02:40

Maybe if you gave us the actual script instead of one "Like it" we could actually debug the issue ...
(0035125)
mewtwo0641 (reporter)
2019-04-16 02:53

The script that I provided is the actual script that is crashing on me although it is a work in progress script; that is why it appears to be incomplete.

In the area here in checkAvs():

if(find == -1 && !osIsNpc(curAgent))
{
    //If avatar is found then do stuff here
}

I only had a llWhisper(0, llKey2Name(curAgent)); for debugging there but took it out to avoid potentially spamming everyone who tries this script to test.

Other than that small change the script I provided is the same one that is crashing.
(0035126)
BillBlight (developer)
2019-04-16 02:57
edited on: 2019-04-16 02:58

code does not do anything, it does not even make a list so not sure how it is even using any memory .. other than with one user on it ..

It does not add anybody to a list, sooo I'm curious to how it would even grow in memory, when tested just like it is, it just sits there

(0035127)
mewtwo0641 (reporter)
2019-04-16 03:01

That is why I'm so confused about this. I've looked over the script multiple times and can't see anything that would suggest to me that memory use is uncontrollably growing. I have not made any changes to the script recently either since it wasn't having issues crashing; it's just sort of been sitting on my region collecting dust and I had almost forgotten about it until I saw all the crashing messages today which is what lead me to believe that this issue has only very recently started.
(0035128)
BillBlight (developer)
2019-04-16 03:06

I have a much more complicated visitor script that keeps many lists, makes notecards , calls http, sends IM's and it has been running since the new master dropped ... So without seeing what your "Work in Progress" is, this base script does not break anything.
(0035129)
BillBlight (developer)
2019-04-16 03:12

Again not debating the fact there is an issue, very well could be, and you are seeing it, just trying to replicate it ..
(0035130)
mewtwo0641 (reporter)
2019-04-16 03:13
edited on: 2019-04-16 03:30

I call script I posted is a work in progress yes, but it's just a stalled work in progress. Might have been better if I had called it a "stalled development script" instead of a work in progress lol. What I have posted in this mantis though is the complete script at the point in time where I stopped development of it, minus the debugging I mentioned in last post. I realize it doesn't make sense... doesn't make a bit of sense to me either, but I can't deny the messages I've been seeing coming from it today.

(0035131)
mike.chase (reporter)
2019-04-16 04:55

I have started to read through the patch and it looks like it enrolls additional data types in tracking. I haven't had time to look into it in depth but I thought I saw it adding pointers to allocated items to a list. Unless these are weak references that will hold on to memory rather than release it. It's very likely I have this wrong though since there is a fair lot of code in XMR and I'm only just starting to get familiar with it.

BTW, this might be a really good example of why developing on HEAD is a bad idea. Anyone pulling this tree now is going to get things with the regressions being pointed out above. I would think iterating this on a branch until the issue is addressed and then merging that would limit exposure for people running YEngine who update frequently.
(0035132)
tampa (reporter)
2019-04-16 06:26

Seeing through the history I would actually agree it may be time to once again move "testing" into its own branch rather than just "hope for the best" on master. At the same time the nature of the project is as such that master is very much a development branch and not a stable or whatever, we have branches for that stuff too. Up to Ubit how much he trusts his changes not to cause fires or worse ;)
(0035135)
UbitUmarov (administrator)
2019-04-16 08:37

master IS a test branch
(0035148)
UbitUmarov (administrator)
2019-04-17 12:51

original script still has this problem?
other scripts still have problems?

well and the memory currently accounted is only a fraction of real one :(
(0035194)
ZauberParacelsus (reporter)
2019-05-05 06:34

I've seen a report from a user that this issue is occuring at the TAG grid as well.

- Issue History
Date Modified Username Field Change
2019-04-13 07:23 mike.chase New Issue
2019-04-13 07:33 BillBlight Note Added: 0035087
2019-04-13 07:37 BillBlight Note Added: 0035088
2019-04-13 08:24 mike.chase Note Added: 0035089
2019-04-13 08:34 BillBlight Note Added: 0035090
2019-04-13 09:53 mike.chase Note Added: 0035091
2019-04-13 10:00 mike.chase Note Deleted: 0035091
2019-04-13 10:05 BillBlight Note Added: 0035093
2019-04-13 10:07 BillBlight Note View State: 0035093: private
2019-04-13 10:07 BillBlight Note Deleted: 0035093
2019-04-13 10:10 mike.chase Note Added: 0035094
2019-04-13 10:12 BillBlight Note Added: 0035095
2019-04-13 12:20 UbitUmarov Note Added: 0035096
2019-04-13 15:27 UbitUmarov Note Added: 0035097
2019-04-13 15:28 UbitUmarov Note Edited: 0035097 View Revisions
2019-04-13 15:44 UbitUmarov Note Added: 0035098
2019-04-13 15:47 UbitUmarov Note Added: 0035099
2019-04-13 16:10 UbitUmarov Note Edited: 0035098 View Revisions
2019-04-13 16:35 BillBlight Note Added: 0035100
2019-04-13 16:41 BillBlight Note Edited: 0035100 View Revisions
2019-04-13 16:48 BillBlight Note Added: 0035101
2019-04-13 16:49 BillBlight Note Edited: 0035101 View Revisions
2019-04-14 07:16 mike.chase Note Added: 0035103
2019-04-15 04:45 UbitUmarov Note Added: 0035104
2019-04-15 04:53 UbitUmarov Note Added: 0035105
2019-04-15 06:08 mike.chase Note Added: 0035106
2019-04-15 16:43 UbitUmarov Note Added: 0035107
2019-04-15 23:51 mewtwo0641 Note Added: 0035108
2019-04-16 00:13 BillBlight Note Added: 0035109
2019-04-16 00:15 BillBlight Note Added: 0035110
2019-04-16 00:28 mewtwo0641 Note Added: 0035111
2019-04-16 00:31 BillBlight Note Added: 0035112
2019-04-16 00:33 mewtwo0641 Note Added: 0035113
2019-04-16 00:50 BillBlight Note Added: 0035114
2019-04-16 00:52 BillBlight Note Edited: 0035114 View Revisions
2019-04-16 00:55 BillBlight Note Edited: 0035114 View Revisions
2019-04-16 00:56 BillBlight Note Added: 0035115
2019-04-16 01:02 BillBlight Note Added: 0035116
2019-04-16 01:04 BillBlight Note Edited: 0035116 View Revisions
2019-04-16 01:04 BillBlight Note Edited: 0035116 View Revisions
2019-04-16 01:25 mewtwo0641 Note Added: 0035117
2019-04-16 01:29 BillBlight Note Added: 0035118
2019-04-16 01:41 mewtwo0641 Note Added: 0035119
2019-04-16 01:47 BillBlight Note Added: 0035120
2019-04-16 01:48 BillBlight Note Edited: 0035120 View Revisions
2019-04-16 01:59 mewtwo0641 Note Added: 0035121
2019-04-16 02:21 UbitUmarov Note Added: 0035122
2019-04-16 02:39 mewtwo0641 Note Added: 0035123
2019-04-16 02:40 BillBlight Note Added: 0035124
2019-04-16 02:43 mewtwo0641 Note Edited: 0035123 View Revisions
2019-04-16 02:53 mewtwo0641 Note Added: 0035125
2019-04-16 02:57 BillBlight Note Added: 0035126
2019-04-16 02:58 BillBlight Note Edited: 0035126 View Revisions
2019-04-16 03:01 mewtwo0641 Note Added: 0035127
2019-04-16 03:06 BillBlight Note Added: 0035128
2019-04-16 03:12 BillBlight Note Added: 0035129
2019-04-16 03:13 mewtwo0641 Note Added: 0035130
2019-04-16 03:30 mewtwo0641 Note Edited: 0035130 View Revisions
2019-04-16 04:55 mike.chase Note Added: 0035131
2019-04-16 06:26 tampa Note Added: 0035132
2019-04-16 08:37 UbitUmarov Note Added: 0035135
2019-04-17 12:51 UbitUmarov Note Added: 0035148
2019-05-05 06:34 ZauberParacelsus Note Added: 0035194


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker