Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007904opensim[MISC] Compiling / Buildingpublic2016-05-11 05:102018-10-06 19:34
ReporterSnoots Dwagon 
Assigned To 
PriorityurgentSeverityblockReproducibilityalways
StatusnewResolutionopen 
PlatformPCOSWindows OS VersionAll
Product Version 
Target VersionFixed in Version 
Summary0007904: Slow Object CONTENTS loading blocks building
DescriptionThis problem was discovered on Inworldz and fixed there. It still exists on OsGrid. The more CONTENTS added to an object the more the contents listing lags whenever updating. Once one gets past 100 or so items (such as if making a music player or texture picker), updating contents lags to the point it becomes non-updateable. This presents a BLOCK situation to building. If Contents work at all the listing is so extremely slow as to frustrate builders in the extreme.

I am listing this as URGENT because it is a block situation, preventing builders from creating-- one of the core functions of OpenSim.
Steps To Reproduce* Create a prim
* For testing purpose, add 50 random full-perm inventory items to the prim (so nothing is lost from inventory by being non-copyable)
** Note how long it takes for the items to show up in CONTENTS
* Now add another 50. Make same observation.
* Continue adding 50 at a time. It should become quickly unusuable.

Now, imagine being a scripter and having to go through that wait every time a script is updated, added or removed.

As mentioned, Inworldz located and fixed the bug and indicated it was fairly easy to fix once located. They may be willing to discuss what causes the bug so that it can be quickly fixed on OSgrid.
Additional InformationCurrently this particular bug is driving me crazy, as I'm attempting to create a massive city and this particular bug is causing me to spend an HOUR or more trying to do something that should be done in one minute or less. That is a no-exaggeration illustration of how severe this bug is.
TagsNo tags attached.
Git Revision or version number
Run ModeStandalone (1 Region)
Physics EngineBulletSim
EnvironmentMono / Windows
Mono VersionNone
ViewerFirestorm-- and all
Attached Filesjpg file icon object_inventory_update.jpg [^] (439,877 bytes) 2016-05-13 13:42

- Relationships
related to 0007907new Large-primcount items create long delay in right-click editing 
related to 0007922new Contents of prims fail to load when edited 
related to 0008391new Object Contents failing to load promptly 

-  Notes
(0030292)
melanie (administrator)
2016-05-11 08:35

Are you seeing this in 0.9.0? We do know the issue exists in 0.8.2 but no more fixes will be done for 0.8 series code.
(0030293)
Snoots Dwagon (viewer)
2016-05-11 09:48

Ahhh... I honestly don't know. I will test and see. Thank you!
(0030294)
Snoots Dwagon (viewer)
2016-05-11 09:52

I just checked the server I was on. This is the version:

OpenSim 0.9.0.0 Dev OSgrid 0.9.0.0 (Dev) be43fc2: 2016-03-06 (Win/.NET)
(0030295)
Snoots Dwagon (viewer)
2016-05-11 09:58

Verified problem exists at Sandbox Plaza II.
(0030296)
Mata Hari (reporter)
2016-05-11 10:23

Also worth noting, whatever is causing the block is also locking anything else related to it. For example, add a script with:

default
{
    touch_start(integer num)
    {
        llDialog(llDetectedKey(0),"Waiting...",["OK"],-123);
    }
}

When the item's inventory is slow enough loading to be a real pain (2+seconds) add one more generic item to it and then quckly touch it. The dialog doesn't appear until *after* the inventory has been reloaded, even if you don't have the contents tab open.

It is my impression that the length of time taken to reload inventory is longer under 0.9 than it is under 0.8.2; but considering all of the possible factors involved (there's considerable variance in times when you repeat identical tests under identical conditions...something can take 3-4 seconds to load on one attempt and then take 10-15 seconds the next time with no changes) so it's hard to be certain.
(0030297)
Snoots Dwagon (viewer)
2016-05-11 13:50

As an example in time factors:

I created a music player with some 250 sound clips in the contents. The player works just fine. But if it is edited and the contents viewed-- in excess of three minutes later one is still waiting.

On very rare occasion the contents will pop right up (as it should do all the time), indicating this is possibly a severe bottleneck issue.
(0030308)
Mata Hari (reporter)
2016-05-13 13:42
edited on: 2016-05-13 13:49

What's really mystifying about this is that it appears to be a massive sudden use of CPU for no apparent reason.

Attached is usage graph by Opensim.exe as I add a single animation to an object in the region that already has approximately 400 animations in it. Immediately prior to making the change, CPU usage is very close to 0%. As soon as I drop the animation into the object, CPU rapidly escalates and fluctuates between 40% and 50% for more than 30 seconds. Finally, that drops off, there's a little burst of network traffic (sending the update to my viewer I assume) and it drops back down to using close to 0% of the CPU again.

I can't imagine why adding a single item to an object's inventory would need to use 50% of a quad-core i7's horsepower...

EDIT: removing the item from inventory again produces the exact same set of events

(0030309)
Snoots Dwagon (viewer)
2016-05-13 15:02

I've noticed exactly the same symptoms Mata. Just today I was adding a single sound file to an object that contained just six items. It took a full 20 to 30
seconds (didn't time it precisely) for that item to register in the contents. During that time nothing could be done with the item and even the sim itself seemed to lag. Active sandboxes must be a nightmare.
 
There is something else I've noticed that may or may not be related to this issue. I'm going to post it in another mantis, but mentioning it here because of the possible relativity to this issue:

* Create an item of say, 900 prims, and link it all together.
* Remove focus from that item (exit the edit box).
* Now right click and edit the item. It takes AGES for the edit box to become functional.

* Now rez a single prim by that 900 prim object.
* Right click and edit the single prim. Of course it is immediately editable.
* Without exiting the edit box, shift focus to the 900 prim object.

Result: Editing of the 900 prim object is IMMEDIATE, as it should be.

This too points to a serious bottleneck somewhere-- in this case though one that can be bypassed if the trick is known. But since the large-prim object can be immediately edited by shifting from a smaller object, it should also be immediately editable from a right click. It's not-- which points to some kind of logic issue in the building processor.
 
The source of the problem may be the same issue, closely related, or not related at all. But it manifests similarly, which may provide a hint.
(0030310)
Snoots Dwagon (viewer)
2016-05-13 15:13

Changed Category from "Grid/ Robust Serving" to "Compiling/Building".
(0030311)
Snoots Dwagon (viewer)
2016-05-13 15:18
edited on: 2016-05-13 15:20

Follow-up: Opinion

Fixing this issue as well as the large-primcount editing issue may be one of the best things that could happen to OpenSim. Nothing discourages new users more than excessive lag. As a builder and scripter with some 12+ years experience I can verify that of all the bugs currently on the Mantis, these two bug me the most. (Sorry for the pun. Well, not really.)

Were these two issues fixed, the vast majority of building / scripting problems I've faced (namely, updating builds in any way) would be gone. Little else could better improve the new builder experience. It's not that other bugs aren't serious. It's that these two particular bugs are so visible and invasive to the building / updating process.

(0030471)
melanie (administrator)
2016-06-12 15:30

Object content loading is done by the viewer via the LLVFS. The object contents are dumped into a "file" and the "file" is then sent to the viewer via UDP. This is where the issue lies, something as important to fluid building as loading contents should be run via a CAP but LL have failed to allow that.

Once object inventory is changed, the viewer sets the object's status to invalid. Until the inventory has been refreshed from the server, something which happens regardless of whether or not the contents tab is currently shown, the prim is dead to touches etc. They are blocked and held viewerside.

There is nothing we can do about the viewer behavior but the download should certainly go faster!
(0030473)
Mata Hari (reporter)
2016-06-12 15:35

@Melanie: any idea why it cause such a massive CPU hit on the region server? Is it continually processing something in a loop somehow relating to that data? There doesn't seem to be any change at all in network traffic during the entire period of time when the CPU is being maxed out.
(0030474)
melanie (administrator)
2016-06-12 15:36

I'm not sure. I only looked at that code once when I made changes to the "file" to disallow asset IDs to be sent to anyone other than the owner for content protection. I didn't do any more reading there because this has always worked for me. My workflow is much different from most people's.
(0030475)
Diva (administrator)
2016-06-12 16:08
edited on: 2016-06-12 16:11

This code that came from avination is very different from what we had in 0.8.2. The lock scheme is completely different. I don't know if that's a good thing or a bad thing. I see that it takes about 3 secs to get an object's inventory with about 100 items (local test grid, no latency).

(0030476)
Mata Hari (reporter)
2016-06-12 16:17

@Diva: the same issue exists under 0.8.2.1 as well, although not to quite as severe a degree. It, too, makes massive use of the region server's CPU throughout the period where the fetch is happening.
(0030477)
melanie (administrator)
2016-06-12 16:21

The lock scheme is a great improvement overall because using ReaderWriterLocks instead of Monitor locks allows read concurrency. The issues with actually loading it are elsewhere.
(0030478)
Diva (administrator)
2016-06-12 16:23

As far as I can tell, once the file is created, the download is really fast. (the file gets cached while nothing changes) It's the file generation that is causing the long delay and, apparently, it's worse in 0.9 than in 0.8.2.
(0030479)
melanie (administrator)
2016-06-12 16:24

Well, when I worked with it I had the code instrumented and actually creating the file itself was no time at all.... I had always assumed it's the UDP that messes it up. Strange.
(0030481)
Diva (administrator)
2016-06-12 21:08

After looking at this for a few hours, my conclusion is that this is pretty unsolvable at this point. Sending the inventory via UDP is very slow. It's slower than it ought to be, though, but I cannot pinpoint the reason for the slowness.
(0030482)
Snoots Dwagon (viewer)
2016-06-12 21:27

I am not an expert in this area, but I seem to remember from long ago that the UDP implementation used by LL originally was flawed. Any error in the packet sending requires the entire packet to be sent again. It would be thought this would be minor, but in tests ran by an experienced tech it was discovered such resends could number in the hundreds for a single packet, causing extreme lag and in some cases even loss of data (thus the regular cry on SL of damaged inventory).

The solution was to use HTTP instead, which proved considerably faster and more reliable. Of course this would have to be implemented both server and viewer side.
(0030488)
Gavin Hird (reporter)
2016-06-13 05:02

I have not looked at the code yet, but does not any objects like this get encoded into / decoded from XML when stored in / retrieved from the asset server that would increasingly add to the processing as the object has more content?

The asset you are working on in-world it will have changes flushed to the asset server at regular intervals (configurable) and the cache invalidated presumably so it will have to be re-read from the asset server?

This can be configured in OpenSim.ini i the [Startup] section:

    ;# {MinimumTimeBeforePersistenceConsidered} {} {Time before un-changed object may be persisted} {} 60
    ;; Objects will be considered for persistance in the next sweep when they
    ;; have not changed for this number of seconds.
     MinimumTimeBeforePersistenceConsidered = 10

    ;# {MaximumTimeBeforePersistenceConsidered} {} {Time before changed objects may be persisted?} {} 600
    ;; Objects will always be considered for persistance in the next sweep
    ;; if the first change occurred this number of seconds ago.
     MaximumTimeBeforePersistenceConsidered = 100
(0030489)
Diva (administrator)
2016-06-13 06:57

FYI, I just flipped my test sim from 0.9 dev to 0.8.2 and the time it takes to load the prims' inventory with 100 items is exactly the same: about 4 seconds.

(My test grid is a highly controlled environment that does not account for latency or packet loss. This is a good thing and a bad thing at the same time -- it's good because experiments and tests can usually be reproduced consistently; it's bad because it does not represent performance in production, so the 4 seconds I'm measuring here will be a higher number in production)
(0030492)
Snoots Dwagon (viewer)
2016-06-13 07:49

That's great Diva. It's always good to have a controlled environment.

Under standard conditions with ping times, latency and packet loss, that 4 seconds becomes minutes, or never, or crash the viewer, or make the sim inoperable.

I conducted some experiments last night, and sadly discovered that while the number of items is highly relevant, it's not necessarily correlative. I worked on an object with four items in contents and it exhibited the same performance. I actually had to reset the entire sim.

This indicates to me that the function itself has a logic bottleneck that can seriously impact sim performance.
(0030495)
Diva (administrator)
2016-06-13 08:59

@Snoots are you testing in 0.9 dev or 0.8.2? At this point I'd like to focus on regressions, i.e. things that worked reasonably well in 0.8.2 and stopped working in 0.9. Is your lock situation in 0.8.2?
(0030496)
Mandarinka Tasty (reporter)
2016-06-13 09:41

Hello:)

I'd like to add my observation here and what was taken under consideration in

Second Life, when such problems were met.

When You edit the object with option: Show Selection Outlines disabled in

the viewer: ctrl + alt + H ( in firestorm )

then speed of loading things drastically increases itself.

So please try to disable it and then make measures.

Maybe it has certain connection with those issues here too.
(0030501)
Snoots Dwagon (viewer)
2016-06-13 11:39

Hello Diva. No, I'm using .9 But my friend on the same server computer (different instance) is using .8.2 so we can make any comparison you want. Just let me know what you need and I'll do my best to compare the two.
(0030502)
Snoots Dwagon (viewer)
2016-06-13 11:40

Oh, thanks Mandarinka. I will definitely try that on both editing large-prim-count objects and in checking contents.
(0030504)
christy lock (reporter)
2016-06-13 12:28
edited on: 2016-06-13 12:55

On further testing on my standalone ( this is the 5/23/16 dload currently on osgrid dload page. Win 7 64bit with MySql ) I found that any new avatar that tried to look at contents could do so 4-8 times before it stopped loading for them. Making no changes but logging in an avatar that has never tried to see the contents would work again for 4-8(aprox) then the Loading contents would be ruined for them. While they could see the contents the previous avatars couldn't at the same time.

(0030506)
Diva (administrator)
2016-06-13 14:16

@Snoots, I would like to know if the long delays/locks you are seeing are also present in 0.8.2
(0030514)
Snoots Dwagon (viewer)
2016-06-13 20:28

@Diva: ran test on both 0.8.2 and 0.9.0 this evening. Same results.
(0030515)
Diva (administrator)
2016-06-13 20:30

ok, thanks. Then this is not a regression.
(0030516)
Snoots Dwagon (viewer)
2016-06-13 20:32

Something else I discovered this evening which may be related: it seems the content handling routines are messed up as well. I tried loading a prim with 52 sound files. It failed to load sound file 14 and instead listed it as sound file 5-1. Nothing I could do would fix the problem.

I totally re-created the object on another sim entirely... and it did exactly the same thing. I tried relogging; did the same thing.

So from what I'm seeing, the object content handler has some severe issues that go beyond a basic bottleneck.
(0030517)
Diva (administrator)
2016-06-13 20:34

What do you mean "tried loading a prim with 52 sound files"? What exactly did you do?
(0030518)
Snoots Dwagon (viewer)
2016-06-13 20:44
edited on: 2016-06-13 20:45

I was creating a sound player. Dragged 52 sound clips from inventory to the prim. Tried it several times, both as one batch, and another time in two batches of 26 files each. Tried it on two different sims as well, one of them the OpenSim East anniversary sim.

This issue surprised me of course. Could be some kind of specific glitch that only happens when using 52 sound clips, dunno. Haven't tested it beyond tonight. I'm going to give it another day or two, then try to reproduce the issue. Was most frustrating this evening; a project that should have taken about 3 minutes wound up taking over an hour.

(0030519)
Snoots Dwagon (viewer)
2016-06-13 20:47
edited on: 2016-06-13 20:48

@Mandarinka: I tried the outlined and no-outlined method suggested, using a 900 prim object as a test. Got very irregular results ranging from 4 seconds to well over 2 minutes. Results were statistically invalid, both for outlined and non-outlined methods, indicating either a random issue or irregularity in the process.

(0030663)
christy lock (reporter)
2016-06-22 09:59

With standalone in loopback - not connected to the internet the contents load every time. Change to external dns and it fails as usual.
(0030664)
Snoots Dwagon (viewer)
2016-06-22 11:12

Good observation Christy. Thanks for adding that note.
(0030810)
UbitUmarov (administrator)
2016-06-28 03:29

this is not (necessary) related to mantis 7907
(0030812)
Snoots Dwagon (viewer)
2016-06-28 05:56

I agree Ubit. However it's worth that someone noted it *just in case* they're related items. If not, the relationship can later be cancelled. It's good that you noted that this isn't an absolute.
(0030878)
UbitUmarov (administrator)
2016-07-07 09:54

object contents download is slow by design.
try doing "Steps To Reproduce" at SL
(0030880)
Snoots Dwagon (viewer)
2016-07-07 12:02

Not this slow. And SL is not the best example to offer. Try Inworldz instead. They fixed that issue some time ago. There IS a bottleneck in this matter.
(0030920)
UbitUmarov (administrator)
2016-07-14 03:51

As I said, contents is slow by design. It maximum bandwidth is limited by udp throttling (to around 20-30KBps and this is required) and worse protocol demands that new packets are sent on the reception of ack of previous one (with 250ms ping that means less than 4KBps).
A 100 items simple test needed about 70KB of data, that's 17seconds with that ping, if all goes well.
Viewers used with opensim are mainly tested at SL, and we already have enough problems with that.
I did however made a few changes to contents sending.
Please note that in same cases viewers incorrectly display "No Contents" and always ask for a file copy on a new click. that seems to be a viewers issue since singularity dev versions do not do it.
(0030921)
Snoots Dwagon (viewer)
2016-07-14 06:16

I appreciate what you're saying. I also think this needs re-thunk. Because on Inworldz I can load an item with 400 contents in about 7-10 seconds. Evidently they're doing it right.

Consider: 70KB of data is a spit in a bucket on systems that can download 10mbps or more. If it takes 17 seconds to send across 70K, something may need totally re-written. But the devs at Inworldz said there was a "bottleneck" in that system. It might be good to contact Zauber or Tranquility and find out what that bottleneck was. I'm sure they'd share; very friendly people.

Same goes for textures. They found a *severe* bottleneck in the texture loading system... and fixed it (at least as much as they could. It still didn't fix the viewer cache issues that are causing so many problems). Balpien and I told LL about the texture bottleneck as far back as 2005 (when I first notified them) and then in 2007 (when Balpien pointed out the specific symptoms). Inworldz fixed that pretty quickly.

Mind you, not comparing Inworldz to OpenSim. But... if any grid locates a problem, fixes it, and is willing to share info on what was wrong (and perhaps even the fix itself)... those are good friends to have. : )
(0030922)
Snoots Dwagon (viewer)
2016-07-14 06:18

BTW... I do appreciate all the hard work you folks are doing. I'm nothing more than a hound dog pointing out the quarry. You folks are the ones with the marksmanship. Your efforts are greatly appreciated by all whom you benefit. I know it's often a thankless job.
(0030923)
UbitUmarov (administrator)
2016-07-14 06:45

Thanks for your appreciation :)

yes I do hate the time it takes to see contents :(
The changes I made are as far I can go at this point.
Except of course fixing bugs :)
The fact most viewers loose contents cache and sometimes show "no contents" is irritating me, but don't find a region side cause :(
(0030924)
Snoots Dwagon (viewer)
2016-07-14 07:11

Ubit, if you have time contact ZAUBER PARACELSUS in Inworldz. He's a dragon and a great guy. Ask him about texture handling; he may be able to point to the issues involved and give you a handle on this. If it's primarily Viewer side at this point, I need to take this over to the Firestorm Mantis.
(0030925)
UbitUmarov (administrator)
2016-07-14 07:14

this mantis issue is about prim contents. Textures is another and more complex issue.
(0030926)
Snoots Dwagon (viewer)
2016-07-14 07:32

Doi, my brain went sideways. I meant contact him about contents handling.
(0030927)
Mata Hari (reporter)
2016-07-14 09:38

There is also something that has changed in the not too distant past.

I used to routinely work with objects that had 400-500 items in them and while not fast, the inventory was loading much more rapidly than it does now with 0.8.2 and is even slower with 0.9 dev. I don't know if this is caused by changes to the viewer side of things, the Opensim code, or perhaps operating system changes.

Thanks to recent ISP hardware upgrades my bandwidth and ping times are now more than twice as fast as they used to be, yet object inventories that used to load in 30-60 seconds a couple years ago now take 4-10 minutes. Also, I don't ever recall seeing the huge hit on the region server's CPU during the object inventory loading time in the past whereas it's now creating massive issues, freezing *every single script* in the region for the entire duration.
(0031180)
Mata Hari (reporter)
2016-10-03 13:57

I did a test with this using both current Firestorm (4.7.9) and Singularity (1.8.6).

Under Firestorm an object with 750 animations in its contents will take between 3-5 minutes and during that time the Opensim instance for that region uses 100% CPU, effectively making the sim incredibly lagged for that time and also making any other software on the system grind to a halt.

Selecting the identical object with Singularity took between 35-50 seconds to load the inventory and the region CPU usage never exceeded 5% and was mostly <4%.

I've added that into to an existing Firestorm's Jira since it would appear to be a viewer-specific issue.
(0031181)
Snoots Dwagon (viewer)
2016-10-03 14:49

Excellent findings Mata.

When one considers, for builders and scripters even 35-50 seconds can be a very long time to wait for data to load so they can update an item. Inworldz got this down to loading in a third of that time... so it could be partially an asset or sim server software issue as well. I am wondering if Inworldz would be willing to share their findings in this (not their code, but the cause of the problem). They're friendly enough to possibly even share code on this basic issue.
(0031182)
UbitUmarov (administrator)
2016-10-03 15:03

Yes FS and all viewers that just pasted in recent LL changes have several problems, not only on opensim but also at SL, just have more visible impact at opensim. But i had to give up talking with FS team.
using singu or other more reliable viewer, object contents should be a bit faster on 0.9, performance with reduce to 0.8 level as more requests are open at same time. Keep in mind the used protocol is slow by design not expected to support more than 256 items. We can only improve more changing the protocol.
(0031183)
Mata Hari (reporter)
2016-10-03 15:10

Supporting only 256 items is something that is rather impractical considering many actual in-world use cases where there is a need to be able to handle more.

When making beds, sofa, etc it's not at all uncommon to have >1000 objects and there are many in SL with 3000+
(0031184)
UbitUmarov (administrator)
2016-10-03 15:32

I did wrote about sl contents download timings elsewhere and you Mata did saw it. Object contents download to viewers is slow.. uses a protocol a lot slower that the old udp one for textures, etc. Expect any fix from LL imported to FS to make it even more incompatible with opensim. The changes in 0.9 are what is possible until we have a viewer team willing to add a protocol change.
(0031185)
Snoots Dwagon (viewer)
2016-10-03 16:07

I have to agree with you there Ubit. Until OpenSim cuts the SL apron strings, it will always be limited to Linden Lab's vision. Whether OpenSim *wants* to cut those strings is another matter entirely. But some progress has been made in the form of special osFunction script commands, physics and other concepts.

Frankly, when OpenSim started I wish the devs had started from scratch and made a system that's better than SL, rather than duplicating (and in some cases worsening) Linden Lab arbitrary decisions. VARs is one of the greatest achievements of OpenSim. Now the dev community needs to take that "beyond the LL box" concept about a thousand steps further... and start breaking away from SL limitations.

Slightly off topic though. Doesn't have much to do with this Mantis. Just my opnion... one I've voiced before. When we think about it, the basics of SL-based VR really haven't advanced all that much in the last 13 years. Even mesh uses old, existing technology. Which is why, I think, none of the VR worlds have gained widespread acceptance. The child has failed to grow and is not really what the public wants or needs.

It's those apron strings.

- Issue History
Date Modified Username Field Change
2016-05-11 05:10 Snoots Dwagon New Issue
2016-05-11 08:35 melanie Note Added: 0030292
2016-05-11 09:48 Snoots Dwagon Note Added: 0030293
2016-05-11 09:52 Snoots Dwagon Note Added: 0030294
2016-05-11 09:58 Snoots Dwagon Note Added: 0030295
2016-05-11 10:23 Mata Hari Note Added: 0030296
2016-05-11 13:50 Snoots Dwagon Note Added: 0030297
2016-05-13 13:42 Mata Hari Note Added: 0030308
2016-05-13 13:42 Mata Hari File Added: object_inventory_update.jpg
2016-05-13 13:49 Mata Hari Note Edited: 0030308 View Revisions
2016-05-13 15:02 Snoots Dwagon Note Added: 0030309
2016-05-13 15:13 Snoots Dwagon Note Added: 0030310
2016-05-13 15:13 Snoots Dwagon Category [GRID] Robust Server => [MISC] Compiling / Building
2016-05-13 15:18 Snoots Dwagon Note Added: 0030311
2016-05-13 15:19 Snoots Dwagon Note Edited: 0030311 View Revisions
2016-05-13 15:20 Snoots Dwagon Note Edited: 0030311 View Revisions
2016-06-12 14:39 Mata Hari Relationship added related to 0007922
2016-06-12 15:30 melanie Note Added: 0030471
2016-06-12 15:35 Mata Hari Note Added: 0030473
2016-06-12 15:36 melanie Note Added: 0030474
2016-06-12 16:08 Diva Note Added: 0030475
2016-06-12 16:11 Diva Note Edited: 0030475 View Revisions
2016-06-12 16:17 Mata Hari Note Added: 0030476
2016-06-12 16:21 melanie Note Added: 0030477
2016-06-12 16:23 Diva Note Added: 0030478
2016-06-12 16:24 melanie Note Added: 0030479
2016-06-12 21:08 Diva Note Added: 0030481
2016-06-12 21:27 Snoots Dwagon Note Added: 0030482
2016-06-13 05:02 Gavin Hird Note Added: 0030488
2016-06-13 06:57 Diva Note Added: 0030489
2016-06-13 07:49 Snoots Dwagon Note Added: 0030492
2016-06-13 08:59 Diva Note Added: 0030495
2016-06-13 09:41 Mandarinka Tasty Note Added: 0030496
2016-06-13 11:39 Snoots Dwagon Note Added: 0030501
2016-06-13 11:40 Snoots Dwagon Note Added: 0030502
2016-06-13 12:28 christy lock Note Added: 0030504
2016-06-13 12:32 christy lock Note Edited: 0030504 View Revisions
2016-06-13 12:55 christy lock Note Edited: 0030504 View Revisions
2016-06-13 14:16 Diva Note Added: 0030506
2016-06-13 20:28 Snoots Dwagon Note Added: 0030514
2016-06-13 20:30 Diva Note Added: 0030515
2016-06-13 20:32 Snoots Dwagon Note Added: 0030516
2016-06-13 20:34 Diva Note Added: 0030517
2016-06-13 20:44 Snoots Dwagon Note Added: 0030518
2016-06-13 20:45 Snoots Dwagon Note Edited: 0030518 View Revisions
2016-06-13 20:47 Snoots Dwagon Note Added: 0030519
2016-06-13 20:48 Snoots Dwagon Note Edited: 0030519 View Revisions
2016-06-21 09:51 Diva Relationship added related to 0007907
2016-06-22 09:59 christy lock Note Added: 0030663
2016-06-22 11:12 Snoots Dwagon Note Added: 0030664
2016-06-28 03:29 UbitUmarov Note Added: 0030810
2016-06-28 05:56 Snoots Dwagon Note Added: 0030812
2016-07-07 09:54 UbitUmarov Note Added: 0030878
2016-07-07 12:02 Snoots Dwagon Note Added: 0030880
2016-07-14 03:51 UbitUmarov Note Added: 0030920
2016-07-14 06:16 Snoots Dwagon Note Added: 0030921
2016-07-14 06:18 Snoots Dwagon Note Added: 0030922
2016-07-14 06:45 UbitUmarov Note Added: 0030923
2016-07-14 07:11 Snoots Dwagon Note Added: 0030924
2016-07-14 07:14 UbitUmarov Note Added: 0030925
2016-07-14 07:32 Snoots Dwagon Note Added: 0030926
2016-07-14 09:38 Mata Hari Note Added: 0030927
2016-10-03 13:57 Mata Hari Note Added: 0031180
2016-10-03 14:49 Snoots Dwagon Note Added: 0031181
2016-10-03 15:03 UbitUmarov Note Added: 0031182
2016-10-03 15:10 Mata Hari Note Added: 0031183
2016-10-03 15:32 UbitUmarov Note Added: 0031184
2016-10-03 16:07 Snoots Dwagon Note Added: 0031185
2018-10-06 19:34 UbitUmarov Relationship added related to 0008391


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker