Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008516opensim[REGION] OpenSim Corepublic2019-04-09 03:312019-09-07 18:32
Assigned ToUbitUmarov 
PlatformPCOSWindowsOS Version10
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0008516: Missing Objects and/or Terrain Parts on Teleport between Grids
DescriptionThis is a Mantis issue to act as a place to note observations as the recent code changes related to SupportViewerObjectsCache are tried out more.

With recent dev master code on teleport there can be loss of objects and/or terrain chunks. They cannot be made visible by clicking in the area where the lost objects or terrain is known to be (which can occur already with some objects in viewers). So we are not able to get them back without moving away and back or relogging.

Sending a "force update" from the Server console also has no effect.

The recent commits suggest turning down the viewer bandwidth if this sort of loss occurs. I turned the viewer bandwidth down to 750KBps and that did seem to improve things a lot... but I still do see some items go AWOL occasionally even at 750Kbps. I have yet to identify the specifics of such a situation as its not all the time now.

I have also seen a couple of viewer crashes on teleport even at 750KBps bandwidth but again cannot be specific yet.

I do note a blip in packet loss in the stats tool (ctrl+shift+1) when the issues arises... Packet loss is on zero but as the loss occurs there is a sudden spoke a few seconds after arriving on a region where object/terrain loss can be observed.
Steps To ReproduceTeleport back and forth between grids running latest dev master code and observe if any objects or terrain patches are missing. Try viewer bandwidths such as 1500KBps (default for Firestorm on my setup) and 750KBps.
Additional InformationUbitUmarov and myself have discussed this quite a bit and done some cross grid hypergrid testing with a range of viewer bandwidths rather than my usual (the default I think) 1500Kbps I use on Firestorm (on a 100Mbps link at home and gigabit networking at work).

This occurred with both with SupportViewerObjectsCache set to false and true (true now being the default in master code). Ubit indicated the mechanism changed even if the parameter was set to false.

Ubit indicates that we should be resilient to reasonable packet loss. I have seen moderate packet loss on Second Life and on earlier OSGrid releases without any problems of such object loss. But here the visual loss of objects does seem to be associated with an observation in the stats console of a sudden spoke in packet loss as it occurs.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script Engine
Environment.NET / Windows64
Mono VersionNone
Viewerdev master (2019-04-09)
Attached Filesjpg file icon 2019-04-09-Terrain-Loss-1.jpg [^] (89,528 bytes) 2019-04-09 08:10

jpg file icon 2019-04-09-Prim-Loss-1.jpg [^] (182,032 bytes) 2019-04-09 08:22

- Relationships
related to 0008515resolvedUbitUmarov Mesh disappears 
related to 0008528assignedUbitUmarov Random mesh pieces do not rez 

-  Notes
aiaustin (developer)
2019-04-09 08:07

The way the stats for ping loss show it is as if the cause of the packet loss is occurring just after the arrival of the avatar at the destination region. Not that packet loss that just happens to occur causes the loss if the objects or terrain.

Packet loss can be registering zero, and seems to stay at zero whenever a successful TP takes place and no object or terrain is lost. But I think it is the case that each time I see packet loss if I look round I find there are missing prims or missing chunks of the terrain that are around 16m X 16m.
aiaustin (developer)
2019-04-09 08:21
edited on: 2019-04-09 08:23

Now I am observing something I find weird... note I am TPing between AILand grid Ailand region (avatar logged in natively there) and Openvue grid Openvue region (across Hypergrid). These are the two main regions on each grid. I am on latest dev master on each (2019-04-09 12:05). I am going slow on TP rate, leaving 20 seconds between TPs.

Once I get the prims missing (in this case three ground level 10x10 prims in the cobblestone area on Openvue as shown in attached image 2019-04-09-Prim-Loss-1.jpg), I can TP back home and return to Openvue and the SAME prims are missing. I can even relog, do the same and they can be missing.

BUT, and this is the odd thing... but might hold a clue to where the prims are getting lost... if when on the foreign grid Openvue region I TP via the map to an adjacent region (Sandbox) and the n return via the map to Openvue all on the same grid, the prims show.

BUT, then I go home to AiLand and TP back to Openvue again (all leaving 20 seconds between jumps) and the SAME prims have gone again. Its as if a totally different mechanism is being used to refresh the contents for inter-grid to inter-region jumps. I told you it was weird.

UbitUmarov (administrator)
2019-04-09 08:45

been there via HG with 1500kbps fs and had no issues
aiaustin (developer)
2019-04-09 08:47
edited on: 2019-04-09 08:53

Good :-) I am watching the logs as you hop about Ubit.

All looks normal. This showed once...

16:53:03 - [LLUDPSERVER]: Received a resend of already processed packet 0000047, type RegionHandshakeReply from Ubit.Umarov

mewtwo0641 (reporter)
2019-04-09 22:03

I see this issue still on master with grid set up as not HG. Have to constantly issue 'force update' command in the console to get the missing pieces to show.
aiaustin (developer)
2019-04-10 02:04
edited on: 2019-04-10 02:13

I updated to (2019-04-09 23:27) for both grids and observe essentially the same issues. At 750Kbps it works most of the time, but at 1500Kbps it fails most of the time. And after I have prims and terrain missing even if I wait 20 seconds between a TP out and back the same bits are still missing.

As before a small packet loss just once was associated with the first appearance of the missing prims/terrain chunks and after that it sticks even with nno further reported pin loss. It looks like once lost its.

BUT.. and I still find this peculiar and perhaps a hint as to what might be happening... a TP to an adjacent region (Sandbox) on the second grid and back to the region on that grid with missing prims/terrain (Openvue) makes everything show fine, but TP back to home grid (AiLand) and back to Openvue and the same bits are missing again.

I have double checked, and "force update" has no effect on the missing prims or terrain chunks for me on Openvue/AiLand grid testing. I think MeMewToo0641 may have a separate issue as in all my testing I have never been able to make any prim or terrain chunk that goes missing reappear.

aiaustin (developer)
2019-04-10 02:16
edited on: 2019-04-10 02:19

Ohoh… now the missing chunks and prims are persisting across relogs. I logged out and back in and after all settled did a jump to Openvue grid and the same bits are missing there.

In two separate relogs, I logged in to AiLand and let everything settle down. then I did my first jump to Openvue grid.. and the parts missing on previous logins were still missing. Same parts.

aiaustin (developer)
2019-04-10 09:09
edited on: 2019-04-10 09:10

I still have the same prims missing on FIRST teleport from AiLand grid to Openvue grid after updating from yesterday's git master (966 - 2019-04-09 23:27) to today's (967 - 2019-04-10 13:01) and hence restarting the servers.

Is this possible? I thought the object cache got refreshed after server restart?

UbitUmarov (administrator)
2019-04-10 09:19
edited on: 2019-04-11 09:41

no direct relation to objects cache.
as I told several times, viewer just does not rez things it did received.
most likely because it did overload, leaving things in memory in broken state.
the effect ranges from totally missing, just invisible that show up on touch and even just a face or two visible.
What prims, depends just on order they did arrive to viewer and how the issue was triggered.

similar for terrain.

viewer bandwidth is not only about network, is also about viewer capability to process things.

aiaustin (developer)
2019-04-10 10:03

If it is in memory Ubit, why does it persist over relogs/viewer restarts, and even after server clean upgrades and restarts?

Also, how come the objects show when moving off to an adjacent region and back on the destination grid, yet the same missing parts are still missing after return to home grid and back. Is there a different mechanism for updates in these two cases?

Clearly something serious has gone wrong.. we have not seen issues like this before have we? Its a very different issue to the occasional non-rez of objects that show when the area they are in is clicked on.
BillBlight (developer)
2019-04-10 10:19

@aiaustin, the same could be said, for WHY am I and others on my grid, on osgrid, and on 2 test grids unable to replicate your issue ..

Stomping your foot and saying it's broken does not help the issue.

I can give you 4 cases where it is not broken.

Not trying to say it is not an issue somewhere but two people having an issue does not point to a systemic issue.
aiaustin (developer)
2019-04-10 11:03

Bill.. I am not “stomping my foot”. I am reporting a clear change in behaviour on my test grids. I am delighted that you don’t see any issue on your specific test environments. This Mantis Issue is here to collect information as and when or if anyone else starts to see the issues, so we can pin down the problem.
aiaustin (developer)
2019-04-13 01:00
edited on: 2019-04-13 01:01

Updated Openvue and AiLand grids to (2019-04-11 23:19) and set SupportViewerObjectsCache = false explicitly in OpenSim.ini (default now being true from openSimDefaults.ini).

Testing over the last 2 days in two viewer environments one of which is on a fast link on the same subnet as the servers and the other is over a home ISP accessing the servers remotely. using Firestorm (Windows 10 Pro), default bandwidth of 1500Kbps.

I do not see any missing prims or terrain chunks on teleports between these grids with SupportViewerObjectsCache = false. I even tested TPs back and forth 5 or 6 times at under 5 seconds delay. All looks fine. No packet loss blip occurs when using the stats console.

I then restarted the two grids with SupportViewerObjectsCache = true on both and then see a few missing prims when doing TPs. The prims this time differed a little on each TP, but most were apparently sub-parts of a linked object.

I did these tests after seeing the commit (2019-04-10 20:28) which may have fixed the problem with the issue occurring even when SupportViewerObjectsCache = false as reported earlier.

BillBlight (developer)
2019-04-13 06:25

Just for information purposes, FS default bandwidth is 500Kbps not 1500.
aiaustin (developer)
2019-04-13 09:57
edited on: 2019-04-13 10:08

Ah, that’s good to know Bill. I was just concerned to make sure we work well with defaults, as many users will never change such settings. I guess I set mine to “cable” speed a long time ago and did not realise that continued to be set.

aiaustin (developer)
2019-06-05 11:19

This issue now seems to be fixed.

Openvue and AiLand grids now use SupportViewerObjectsCache = true (the default as set in openSimDefaults.ini).

Issue is addressed on latest dev master - sometime between 24th March 2019 and 14th May 2019. So OSgrid version as of 14th May 2019 is good too.
aiaustin (developer)
2019-06-11 01:55
edited on: 2019-06-11 02:00

As at OSGrid 2019-05-14 I do see missing prims at initial login. Not TP or HG related. Simple cubes missing. Different ones in same linkset for example none after several logins. As before, right clicking or trying to operate on the affected object does not make the missing piece rez. A relog is needed.

UbitUmarov (administrator)
2019-06-11 22:31
edited on: 2019-06-12 00:32

let me remark that viewer capability to handle prims may depend on fps
try to keep the viewer maximized and selected during all login/tps

mewtwo0641 (reporter)
2019-07-31 01:45

Still see this issue on master and have had reports of it from people who use the grid. Setting SupportViewerObjectsCache = false doesn't seem to make a difference. I have suggested to the people reporting it to me to try the old zoom out and ESC zoom in trick; sometimes that works and sometimes it doesn't. Toggling Render > Volume in the Advanced menu (CTRL + ALT + SHIFT + 9) off and then back on again sometimes helps too, but typically I have to resort to sending the 'force update' command in the OpenSim console when they tell me that nothing is working on their end to fix it.
mike.dickson (reporter)
2019-08-20 06:46

We're also seeing this issue on fairly current master. Turning bandwidth down does appear to help but not eliminate the problem. It affects both mesh and regular prims.

I understand that people feel this might be a viewer capacity issue. However the issue DOESN'T happen on SL so I'm not convinced "bandwidth" is the fix. My viewer is rendering 100fps and with network lowered to 750 I still get dropouts sometime. Originally I thought kills were being sent but the prims are "there" (a physics shape often exists) but they aren't rendering.
mike.dickson (reporter)
2019-08-23 06:48

Interesting to note. If I enable the GetAssets cap handler this problem seems to essentially go away. Of course that means less data being shoved over UDP so that makes some sense. The ViewerAssets cap has been in place in the viewers for some time and UDP asset fetching is totally off in SL so its very possible the viewers don't expect to get this data over UDP and because of that stuff breaks.
UbitUmarov (administrator)
2019-08-23 12:06

Thanks for the information.
Meshes and Textures should already be out of udp, and those should be the bulk of assets data at arrival.
Never the less i did add the ViewerAssets cap, not only because it makes sense to send all assets via tcp(http) but also to prevent potencial future issues in viewers side code merging.
But it the problem is related to bandwidth. If got worse after a few changes i made improving lludp i had to reduce the effciency of udp packet usage (not using all MTU bytes) to mitigate it
Now im using a suicidal 3000Kbps (not recomended) and see no problems either on my test grid or osgrid. (in fact most regions i visit are on master with that cap active)
UbitUmarov (administrator)
2019-08-23 12:10
edited on: 2019-08-24 04:08

It may happen it is the extra latency of http that gives viewers a little more breathing time..

aiaustin (developer)
2019-08-23 13:21
edited on: 2019-08-23 13:25

On the assumption that users are directed to use Firestorm 6.0.2 can I check what we should set then? Do we just uncomment this OpenSimDefaults.ini line ...

    ; Cap_GetAsset = "localhost" DO not ucoment this line. Some popular viewers still dont do it right for opensim. Here to easy testing

Is the capability you mention for ViewerAssets just in your test code at present Ubit and not in dev master?

UbitUmarov (administrator)
2019-08-24 02:17

Yes think fs 6.0.2 has it broken
It only asks it on first region, then keeps using the same url
that of course may fail.
aiaustin (developer)
2019-08-24 04:08
edited on: 2019-08-24 08:10

Oh, that is quite serious. Does "localhost" map to whichever server the OpenSim.exe region(s) are running on, or does it resolve to ONE specific server on first use and then stick to that?

If its the latter, then that means it should really not be used except for grids that use a single server for everything.

I see it is indeed reported in the Firestorm JIRA and was corrected by Beq Janus back on 28 Feb 2019 in the Firestorm code... the problem being of course that this is after the current Firestorm 6.0.2 release for OpenSim. [^]

aiaustin (developer)
2019-08-24 08:06

Is the OpenSim server side Cap_GetAsset capability the very same as the “ViewerAssets” capability or are they different?
UbitUmarov (administrator)
2019-08-24 13:36

It is the same
Total Sorbet (reporter)
2019-08-24 13:43
edited on: 2019-08-24 13:52

On login to OSG i have 25% of prims missing.. If i then jump to an alternate region viewer will show nearly all prims (100% on a good day :p) I have tried various bandwidth settings and although i would say this definitely has an effect im unable to find a setting that fixes this.

My default login procedure recently is to login to somewhere i don't want to be then jump to somewhere I do! That way I get to see most prims :)

UbitUmarov (administrator)
2019-08-24 15:04

Total forgot to mention she uses Linux Firestorm
UbitUmarov (administrator)
2019-08-24 15:58
edited on: 2019-08-24 16:04

Think i still had a silly bug on object cache support
Can you pls test master?
Clear viewer cache!!
Issues related to that bug should happen on second relog after that (or returning to a region already in viewer cache)

UbitUmarov (administrator)
2019-08-24 16:37

About the ViewerAsset fs problem. That is still not that relevant, old LLUDP( and http getTexture nd getMesh*) methods are still there, and used by most viewers, some not having that cap.

i did add it, because it does make sense to use http on those, and since reference viewers code had lludp methods removed, less viewers code merging issues to worry about.
mike.dickson (reporter)
2019-08-24 17:34

Had a quick chat with Beq who confirmed there was a bug fixed around ViewerAssets and the behaviour (cap not getting refreshed on a transfer) in FS. It's in 6.2 but not yet released for OpenSim.
aiaustin (developer)
2019-08-25 09:56

I updated Openvue and AiLand grids to latest dev master, cleared my viewer(s) cache(s) and will now watch for any issue,
aiaustin (developer)
2019-08-27 03:53
edited on: 2019-08-27 03:57

Just to report back... As well as Openvue and AiLand grids, I updated my OSGrid addon regions to the latest dev master also for testing. To date after the update to the latest dev master at 2019-08-25 and revisiting already visited (cached) regions, moving between the three grids, and relogs I have not yet seen any missing prims or mesh (or indeed terrain holes). I will continue to monitor carefully and report if I do see anything after this update.

It might be useful to encourage the OSGrid folks to update their distribution for add on regions so we better test this. I know people can compile the latest dev master and add the bin folder from that over the OSGrid version to update (as I have done), but possibly not many will do that.

UbitUmarov (administrator)
2019-08-27 10:53

Yes, code is just in "burnin" on some plazas before Dan makes a release
Thanks again for your testing and feedback
Total Sorbet (reporter)
2019-08-28 08:38

I updated and happy to report no missing prims past 2 days :)
mike.dickson (reporter)
2019-08-31 05:04

Just a quick update. I've rolled out to a few troublesome regions a build based on master with the suggested changes. I can report that for the updated regions this issue seems mostly addressed. I did have an issue with one region with a large mesh build (linkset) and a number of adjacent regions. When camming well back from the object when I zoomed back to my current position the building I was standing on was invisible. Edit, touching, etc, didn't bring it back. I don't think this was a case of the server sending a kill because the object was still physically there. Maybe something with interest lists. But the phenomenon was the same as the missing prims before this fix which I thought was interesting.
UbitUmarov (administrator)
2019-08-31 06:37

Server side culling does not suport camera position, only avatar position.
SO it may cause what you report.
maybe you have it active? (don't remember def as i type)

camera position support would be insane heavy...
mike.dickson (reporter)
2019-09-01 05:43

ObjectsCullingByDistance is false in the code and no default is set in the config or defaults files. But yes its acting that way. It's especially prevalent with Mesh and in some cases you get the invisible polygon that happens when a shape isn't loaded. In those cases edit with cause it to pop in. In others it's just gone. and there's no shape there to edit. So yeah these might be viewer generated kills because something is outside DD. I'll do some more experimenting. Overall better but performance is slow and this camera issue is a real hit for perceived reliability.
UbitUmarov (administrator)
2019-09-01 06:58

Maybe that mesh just has bad LODs?
mewtwo0641 (reporter)
2019-09-02 18:52

Have been testing master for past week; the issue seems to be much improved although I still see it on occasion and have to issue the 'force update' command to fix it. I am still testing with SupportViewerObjectsCache = false though. Should I be testing with it set to true, or does it matter?
aiaustin (developer)
2019-09-03 02:29
edited on: 2019-09-03 02:30

As others have said, I think we are seeing a range of issues when objects don't show.

For the original problem reported in this mantis "force update" had no effect, and there were no outline (physics style wireframes), or ability to touch any area and make the object, prim, terrain piece or mesh appear. I have not see that problem since Ubit made the latest changes. Running with 1500kbps and SupportViewerObjectsCache = true.

mewtwo0641 (reporter)
2019-09-07 18:32
edited on: 2019-09-07 18:32

@aiaustin - That is possible, I have been seeing a high number of log messages that look like this: "[LLUDPSERVER]: Received a resend of already processed packet (Number), type (Type) from (Avatar Name)" upon login of an account. The types vary from General, RegionHandleRequest, GenericMessage, AgentHeightWidth, AgentUpdate, AgentAnimation, EconomyDataRequest, and probably more. I've always seen these messages from time to time but they have never been this frequent or numerous as they have been in the past couple weeks of testing master and the high frequency of them seems to coincide with seeing missing objects upon login or it could just be anecdotal. It may also be entirely unrelated to this issue and/or the issue I described in my previous post. I am just unsure at this point in time.

- Issue History
Date Modified Username Field Change
2019-04-09 03:31 aiaustin New Issue
2019-04-09 03:31 aiaustin Status new => assigned
2019-04-09 03:31 aiaustin Assigned To => UbitUmarov
2019-04-09 03:33 aiaustin Relationship added related to 0008515
2019-04-09 03:35 aiaustin Additional Information Updated View Revisions
2019-04-09 08:07 aiaustin Note Added: 0035073
2019-04-09 08:09 aiaustin File Added: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:09 aiaustin File Deleted: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:10 aiaustin File Added: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:21 aiaustin Note Added: 0035074
2019-04-09 08:22 aiaustin File Added: 2019-04-09-Prim-Loss-1.jpg
2019-04-09 08:23 aiaustin Note Edited: 0035074 View Revisions
2019-04-09 08:45 UbitUmarov Note Added: 0035075
2019-04-09 08:47 aiaustin Note Added: 0035076
2019-04-09 08:53 aiaustin Note Edited: 0035076 View Revisions
2019-04-09 22:03 mewtwo0641 Note Added: 0035077
2019-04-10 02:04 aiaustin Note Added: 0035078
2019-04-10 02:04 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:08 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:10 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:12 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:13 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:16 aiaustin Note Added: 0035079
2019-04-10 02:19 aiaustin Note Edited: 0035079 View Revisions
2019-04-10 09:09 aiaustin Note Added: 0035080
2019-04-10 09:10 aiaustin Note Edited: 0035080 View Revisions
2019-04-10 09:19 UbitUmarov Note Added: 0035081
2019-04-10 10:03 aiaustin Note Added: 0035082
2019-04-10 10:19 BillBlight Note Added: 0035083
2019-04-10 11:03 aiaustin Note Added: 0035084
2019-04-11 09:41 aiaustin Note Edited: 0035081 View Revisions
2019-04-13 01:00 aiaustin Note Added: 0035085
2019-04-13 01:01 aiaustin Note Edited: 0035085 View Revisions
2019-04-13 06:25 BillBlight Note Added: 0035086
2019-04-13 09:57 aiaustin Note Added: 0035092
2019-04-13 10:07 aiaustin Note Edited: 0035092 View Revisions
2019-04-13 10:08 aiaustin Note Edited: 0035092 View Revisions
2019-05-14 09:06 aiaustin Relationship added related to 0008528
2019-06-05 11:19 aiaustin Note Added: 0035288
2019-06-11 01:55 aiaustin Note Added: 0035381
2019-06-11 01:57 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 01:57 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 02:00 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 22:31 UbitUmarov Note Added: 0035392
2019-06-12 00:32 aiaustin Note Edited: 0035392 View Revisions
2019-07-31 01:45 mewtwo0641 Note Added: 0035517
2019-08-20 06:46 mike.dickson Note Added: 0035595
2019-08-23 06:48 mike.dickson Note Added: 0035596
2019-08-23 12:06 UbitUmarov Note Added: 0035597
2019-08-23 12:10 UbitUmarov Note Added: 0035598
2019-08-23 13:21 aiaustin Note Added: 0035599
2019-08-23 13:25 aiaustin Note Edited: 0035599 View Revisions
2019-08-24 02:17 UbitUmarov Note Added: 0035601
2019-08-24 04:08 aiaustin Note Added: 0035602
2019-08-24 04:08 aiaustin Note Edited: 0035598 View Revisions
2019-08-24 04:11 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 04:14 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 04:20 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 08:06 aiaustin Note Added: 0035603
2019-08-24 08:09 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 08:10 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 13:36 UbitUmarov Note Added: 0035604
2019-08-24 13:43 Total Sorbet Note Added: 0035605
2019-08-24 13:44 Total Sorbet Note Edited: 0035605 View Revisions
2019-08-24 13:44 Total Sorbet Note Edited: 0035605 View Revisions
2019-08-24 13:52 Total Sorbet Note Edited: 0035605 View Revisions
2019-08-24 15:04 UbitUmarov Note Added: 0035606
2019-08-24 15:58 UbitUmarov Note Added: 0035607
2019-08-24 16:00 UbitUmarov Note Edited: 0035607 View Revisions
2019-08-24 16:04 UbitUmarov Note Edited: 0035607 View Revisions
2019-08-24 16:37 UbitUmarov Note Added: 0035608
2019-08-24 17:34 mike.dickson Note Added: 0035609
2019-08-25 09:56 aiaustin Note Added: 0035610
2019-08-27 03:53 aiaustin Note Added: 0035612
2019-08-27 03:54 aiaustin Note Edited: 0035612 View Revisions
2019-08-27 03:57 aiaustin Note Edited: 0035612 View Revisions
2019-08-27 10:53 UbitUmarov Note Added: 0035614
2019-08-28 08:38 Total Sorbet Note Added: 0035622
2019-08-31 05:04 mike.dickson Note Added: 0035625
2019-08-31 06:37 UbitUmarov Note Added: 0035626
2019-09-01 05:43 mike.dickson Note Added: 0035631
2019-09-01 06:58 UbitUmarov Note Added: 0035633
2019-09-02 18:52 mewtwo0641 Note Added: 0035656
2019-09-03 02:29 aiaustin Note Added: 0035658
2019-09-03 02:30 aiaustin Note Edited: 0035658 View Revisions
2019-09-07 18:32 mewtwo0641 Note Added: 0035676
2019-09-07 18:32 mewtwo0641 Note Edited: 0035676 View Revisions

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker