Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007657opensim[GRID] Hypergridpublic2015-07-28 16:092015-08-23 08:02
ReporterFerd Frederix 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionfixed 
PlatformWindowsOSWindows 7OS Version7 Pro
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0007657: Mitigating db corruption caused by issue 7514
DescriptionSince October 2014, objects serialized on a MONO grid suffer a corruption that make it useless on a Windows grid. See Mantis 7514. The XML Name Space identifier "xmlns:" gets repeated as "xmlns:xmlns:", which causes the serializer on Windows machines to abort with "Deserialization of xml failed. Exception System.Xml.XmlException: The ':' character, hexadecimal value 0x3A, cannot be included in a name". This error is due to the second colon in "xmlns:xmlns:".

Diva's r/25949 supposedly fixes the original bug that creates these entries at the originating machine. However, it does nothing for the bug reported here, which is failure to decode the corrupted assets from the db, or from the hypergrid.

It seems unreasonable to redefine the XML standard, and patch the XML Parser to fix a bug that is in a database. And I cannot imagine core devs approving anything that would change an asset in the database. But it seems reasonable to ask for a core patch inside the serializer's try-catch. This patch would be inactive and consume no resources unless the catch occurs. We could thus correct the extra xmlns on-the-fly in the catch, then retry the serialization. if it still fails, then the problem lies elsewhere and the final catch would occur. Or perhaps a wrapper class could be implemented. The key thing would be to make it so the serializer accepts this highly specific XML error and then continue. This would repair a very long list of broken assets that are spreading more and more about the hypergrid, and permanently stop the spreading of bad assets by OAR, IAR, and Hypergridding.

Even so, it is a hack that will raise concerns. But it's worthwhile to be discussed how this damage can be mitigated..

Additional InformationThere is some good news. The db can be fixed (for a while) by an external Perl utility, which I am attaching to this Mantis. Jeff Kelley wrote the original version, I've hardened it and added some fail-safes, and we have both run it successfully on our grids. We have permanently repaired over 3,000 bad assets in two small grids of 80K assets total. Jeff Kelley has also managed to fix an OAR by using an untar, sed and tar. The regex to fix the db from the Perl is : s/(xmlns:){2,}+/xmlns:/g

However, Perl and sed are not readily available to Windows users, and such users are not likely to be able to repair databases, even with Perl, cygwin, or other tools. And until every machine is patched and every asset is repaired in every database, the objects will still not stop scribbling red marks on consoles and black marks on the Opensimulator name. The bad assets will continue to spread and will eventually infect everything. The only cure is to periodically run the Perl script to keep up with the incoming bad data. And that is only temporary until someone brings in another broken asset.

Patching every asset in my database still does not seem to fix everything. I believe some assets are fetched from remote servers when a rez attempt is made by HGInventoryAccessModule::RezObject, which eventually calls SceneObjectSerializer::FromOriginalXmlFormat, which will croak,and those will fail.

Example of an object that fails when rezzed. The uuid c69659ae-* is not located when searching my db table assets, or any other table. If this is true, how can an asset in my inventory be trusted to rez? The remote DB, if there is one, could be offline. Or maybe this this another bug, as the UUID in table assets is certainly not c69659ae-*, as is shown in this error log.

2015-07-28 13:02:35,273 DEBUG (STP:Util:53) - OpenSim.Region.CoreModules.Framework.InventoryAccess.HGInventoryAccessModule [HGScene]: RezObject itemID=c69659ae-0738-40c2-95d3-b0bc1aac012a fromTaskID=00000000-0000-0000-0000-000000000000
2015-07-28 13:02:35,287 ERROR (STP:Util:53) - OpenSim.Region.Framework.Scenes.Serialization.SceneObjectSerializer [SERIALIZER]: Deserialization of xml failed. Exception System.Xml.XmlException: The ':' character, hexadecimal value 0x3A, cannot be included in a name. Line 1, position 57.
   at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
   at System.Xml.XmlTextReaderImpl.ParseAttributes()
   at System.Xml.XmlTextReaderImpl.ParseElement()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlCharCheckingReader.Read()
   at System.Xml.XmlReader.ReadToFollowing(String name)
   at OpenSim.Region.Framework.Scenes.Serialization.SceneObjectSerializer.FromOriginalXmlFormat(XmlReader reader) in g:\HyperGrid\dev\opensim\OpenSim\Region\Framework\Scenes\Serialization\SceneObjectSerializer.cs:line 85

TagsNo tags attached.
Git Revision or version number
Run Mode Standalone (Multiple Regions)
Physics EngineBulletSim
Script Engine
Environment.NET / Windows64
Mono VersionNone
ViewerAny
Attached Files? file icon mantis7514.pl [^] (8,391 bytes) 2015-07-28 16:09
? file icon mantis7514 Full Rev B.pl [^] (12,649 bytes) 2015-07-30 11:51
? file icon mantis7514 Smallest Rev C.pl [^] (8,309 bytes) 2015-07-30 13:59

- Relationships
related to 0007514closedDiva Can't rez items from OSGrid on my own grid. 
related to 0007668new object description xmlns:xmlns: gets changed to xmlns: when transfered via HG 

-  Notes
(0029005)
Seth Nygard (reporter)
2015-07-28 19:21

I would suggest that anyone "fixing" any assets should also flush any local cache in their simulators after running the script. You want to make sure there is only one, copy of the asset available, and that is the fixed one. So by flushing all the sim caches that forces everything to be pulled from the asset server the first time, ensuring a "fixed" asset is retrieved.


IMHO a filter should really to be added to the incoming HG asset handler to clean up any bad assets before they get stored in the local asset store. This is probably the only long term solution considering how many grids and potential "bad" assets there are out there.
(0029006)
Seth Nygard (reporter)
2015-07-28 20:20

I ran a dry run on the main grid asset store and out of 110942 assets scanned the script found 9754 corrupt xmlns objects.
(0029007)
Gavin Hird (reporter)
2015-07-29 01:15

Found 255 corrupt out of 18247 objects on dry run. They all originated from other grids.
(0029008)
Gavin Hird (reporter)
2015-07-29 03:44
edited on: 2015-07-29 03:45

In Postgres I can do a query to find the corrupt records:

SELECT ID, (SELECT regexp_matches(ENCODE(data, 'escape'), '(xmlns:xmlns:)')) FROM assets WHERE "assetType" = '6';

I will write a query that use the regex_replace function to update the corrupt records, and if that works, create a trigger or rule that performs the query on inserted or updated records.

(0029009)
Gavin Hird (reporter)
2015-07-29 07:04

As far as I can see this single statement will fix the issue on the records corrupted:

SELECT ID, (SELECT DECODE(regexp_replace(ENCODE(data, 'escape'), '((xmlns:){2,})+', '(xmlns:)' , 'g'),'escape')) FROM assets WHERE "assetType" = '6';

Actually you don't need the SELECT ID part, and there is a bit of conversion going on here because the data is stored in the PostgresSQL bytea binary format. If you peel that off you end up with a statement that is

SELECT regexp_replace('((xmlns:){2,})+', '(xmlns:)' , 'g')) FROM assets WHERE "assetType" = '6';

I have not run this on the production database yet, but only on a copy of it. I need to make a good backup of the production DB first! :-O
(0029010)
Seth Nygard (reporter)
2015-07-29 07:22

@Gavin, while I agree that Postgresql allows for some other ways to address this issue, likely >99% of OpenSim installs are not running Postgresql.
  

The attached script, or any other such database trigger based solution, are really only hacks to help work around a problem that resulted in asset corruption, due to a commit dating back to October and has affect a very large number of users/grids since then. The Bad assets come from other grids that are not updated with fixes to correct the initial problem.

A filter for this really does belong in the core, to check assets before they ever get stored. There needs to be a solution that works for everyone as this is going to be an ongoing problem that could exist forever.
(0029011)
Gavin Hird (reporter)
2015-07-29 07:24

> A filter for this really does belong in the core, to check assets before they ever get stored

Absolutely agree. Does not MySQL have the regex_replace function?
(0029012)
Seth Nygard (reporter)
2015-07-29 07:32

@Gavin Unless things have changed recently the only way to get regex_replace functionality in MySQL is through an add-on. MariaDB does have it built-in I believe however, and I know some people are using that in place of MySQL with OpenSim.
(0029013)
Gavin Hird (reporter)
2015-07-29 07:33

OK, Sorry I did not realize that as most databases have it.
(0029015)
Ferd Frederix (reporter)
2015-07-29 09:29

Received a email report of an ubuntu box with 178 bad out of 16,248 assets. I assume that a Linux box having bad assets is proof that it is spreading. Those assets must come from Windows boxes that got them from Linux boxes. Probably caused by sending objects by drag and drop, as the assets could not be rezzed on Windows to be taken.

The same report says that you may need to comment out two lines in the Perl. This only needs to be done if the modules are not part of your core Perl.

49 #use Try::Tiny; # error trapper
50 #use XML::Simple; # error checker for final XML

I had removed the code that required these two modules. The code was very slow and was not supporting this fix. It tested every asset by unserializing the data into an object in Perl. I was able to find a few weird assets with it, including one asset with no data. Removing the extra code also removed the need for the Try/Catch surrounding it. It is harmless to leave these lines in, or to comment them out. If your Perl fails to find them, you can comment these lines out.
(0029016)
Neo Cortex (reporter)
2015-07-29 10:55

I found another way to quickly reproduce the issue:
- logged in to my Linux grid (mono 2.10.8, robust 0.8.0.1, sim's under 0.8.1.0)
- hypergrid jumped to a windows sim (opensim 0.8.1)
- created a prim and took it into my inventory, still standing on that remote sim
- ran check script on my grid and found that new item corrupt.
(0029017)
Gavin Hird (reporter)
2015-07-29 11:02

Has anyone investigated the issue to find out if it is only on create (insert) the corruption happens, or could it be corrupted on update even if the initial record was correct or has been fixed? – Like someone taking an object to a Windows based sim, updates it there and brings back home.
(0029018)
JeffKelley (reporter)
2015-07-29 11:40

Cleaned up 1705 corruptions on a public grid (46k objects since running). Avatars can now hg to a Windows grid and rez object they could not rez yesterday.

The reverse is not true. Foreign avis being given an object from inventory to inventory cannot rez. As noted by Fred, there is something during HG assets transfer we don't catch yet. The grid was restarted after clearing all region caches.
(0029019)
Gavin Hird (reporter)
2015-07-29 11:55

The final query became:

UPDATE assets SET data = DECODE(regexp_replace(ENCODE(data, 'escape'), '((xmlns:){2,})+', '(xmlns:)' , 'g'),'escape') WHERE "assetType" = '6';

It executed scan and replace of 18274 records in 77.8 sec so running quite efficient.

There are regex functions for MySQL at this repository it might be worthwhile looking into. https://github.com/hholzgra/mysql-udf-regexp [^]
(0029021)
aiaustin (developer)
2015-07-30 02:41
edited on: 2015-07-30 02:44

To help other Windows users who don't have Perl installed I used

     ActivePerl... http://www.activestate.com/activeperl [^]

and simply installed it using the .msi installer on Windows. But when I first tried to run the script it reported that a necessary module was not installed by default.... DBD-MySQL is required as described in this MySQL documentation.

     https://dev.mysql.com/doc/refman/5.0/en/activestate-perl.html [^]

Run the package manager to see what is installed via C:\Perl\bin\ppm.bat or C:\Perl64\bin\ppm.bat but initially it only shows the currently installed modules, NOT all available packages. Use the "View" menu and select "All Packages" and then DVD-MySQL" shows as an option to install. Right click on the entry and select Install and then use the green right arrow to run the marked action.

After that you should just be able to run the mantis7512.pl script above in a command window after you have inserted your own DB details where its marked.

(0029022)
aiaustin (developer)
2015-07-30 03:01

http://strawberryperl.com [^] as the alternative Perl for Windows users seems to be inaccessible at the moment. may be temporary?

When it comes to actually updating the database, I set UPDATE = 1 but the comment says XMLCHECK must also be set ON...

# for safety, we will never update a record unless the UPDATE and XMLCHECK flags are on.

Where is that done?
(0029030)
Ferd Frederix (reporter)
2015-07-30 12:06

Whether you use strawberry or activestate is a personal preference. And strawberry.com is down for me, too. You can get it from Cnet at http://download.cnet.com/Strawberry-Perl-Portable-64-bit/3000-2212_4-75808035.html [^]


I've used both. Strawberry is more closer to the Linux Perl and has CPAN support. Activestate may be a bit easier to use. Strawberry is free, and ActivePerl is free for non-commercial or non-production use. They charge $1000 for a corporate license if it is "used on the Internet".
(0029031)
Ferd Frederix (reporter)
2015-07-30 12:06
edited on: 2015-07-30 13:40

I uploaded two newer versions of the scanner. The "mantis7514 Smallest Rev B.pl" is basically the same as before but with the XMLCHECK comment and the extra, unused modules removed, so it has has the smallest dependency list in Perl and should run even on older Perls.

The XMLCHECK check is only done in the second program, "mantis7514 Full Rev B.pl", which can be set to scan every record of type 6 for any XML failures to decode. I used it to to find two other odd XML's - one of them totally null but for a UUID, the other type have a tag that has an illegal null embedded in them. I also scanned by db of all types not just 6 (objects) and found no other xmlsns: errors.

(0029032)
aiaustin (developer)
2015-07-30 13:27

Thanks Ferd... I can do the scan with UPDATE = 0 and that reports the number of records that need fixing.

I can then delete the .ini file and corrupt folder created by that run and run again with UPDATE = 1 and that reports the same numbers.

But if I them do the UPDATE = 0 scan again it still reports the same number of corrupt entries.

There must be some trick to forcing it to update? I had assumed it was because I had not set some required flag for XMLCHECK in the original script, but given your new uploaded versions that does not seem so.

I must be doing something stupid. I am using ActivePerl and will try with Strawberry Perl when I can get a copy to see if that actually dies the UPDATE = 1 fixes.
(0029035)
Ferd Frederix (reporter)
2015-07-30 14:02

Oops. My bad. See Rev C smallest. I had deleted the UPDATE call due to the paranoia brought on by running it on live db.
(0029036)
aiaustin (developer)
2015-07-30 14:33

Excellent... that works.. thanks a lot Ferd and Jeff.

But Windows users need to delete /tmp/ in this line... or set it to a suitable Windows file location...

use constant MANTIS7514 => '/tmp/mantis7514.ini'; # path to a temp file to keep track of each run. Delete the file to start at the beginning
(0029049)
Richardus Raymaker (reporter)
2015-08-01 10:53
edited on: 2015-08-01 10:55

That |XML error looks like soemthing i see soemtimes. last one a few days ago at startup. plenty of them. Offcorse on windows system. Now i need to read all above postings. Can opensim not implement some database checkup to fix problems like this with console comand.

Am still a bit puzzled what todo right now.

(0029050)
Diva (administrator)
2015-08-01 10:57

There is a new release of 0.8.1, called 0.8.1.1, that has this bug fixed. Unfortunately, the damage to the data has been done during this period, and many assets will be silently broken until you try to rez them.

I highly recommend using this Perl tool to clean them up.
(0029051)
Richardus Raymaker (reporter)
2015-08-01 11:31

It's failing on my windows system with region database.

perl.exe mantis7514_Smallest_Rev_C.pl
DBD::mysql::db selectall_arrayref failed: You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right syn
tax to use near '-astro.assets WHERE assetType=6 and create_time > 0 order by cr
eate_time asc' at line 1 at mantis7514_Smallest_Rev_C.pl line 112.
DBD::mysql::db selectall_arrayref failed: You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right syn
tax to use near '-astro.assets WHERE assetType=6 and create_time > 0 order by cr
eate_time asc' at line 1 at mantis7514_Smallest_Rev_C.pl line 112.
(0029052)
JeffKelley (reporter)
2015-08-01 11:36

> It's failing on my windows system with region database.

$robustDB should contain the name of your ROBUST database.
(0029053)
Richardus Raymaker (reporter)
2015-08-01 11:38

So, this tool is not for regions. But that nobody told.
But i expected something like that when i did read $robust.
Pushing the problem to the grid owner :O
(0029054)
Ferd Frederix (reporter)
2015-08-01 17:11

I just picked up some things and they didn't rez. Then I remembered Seth's advice: delete the contents of your bin/assetcache folder after running this program so that the changes it makes will into effect. That fixed it. The cache gets rebuilt automatically.

The location may have been moved in your installation. See bin/config_include/FlotsamCache.ini for details on the location of your cache. It is usually a line that looks like this:
 
 CacheDirectory = ./assetcache
(0029055)
Seth Nygard (reporter)
2015-08-01 18:28

Any grid running SRAS, FSAssets, or other similar asset store will need a different way to clean up their assets.

I still strongly believe the need for a filter to be added in the core code to "fix" any asset before it is stored or updated. Unless we can guarantee 100% coverage of all assets in every grid along with 100% coverage of updates to fix the cause at the same time these bad assets and going to continue to be an ongoing problem across the hypergrid.
(0029056)
Ferd Frederix (reporter)
2015-08-01 18:44
edited on: 2015-08-01 19:58

Diva sent a note today to opensim-dev@opensimulator.org :

There is a new release of 0.8.1, numbered 0.8.1.1, that includes the fix for the bug reported here http://opensimulator.org/mantis/view.php?id=7657 [^]

The download links available from the Wiki have been updated.

Note that this does not incorporate the improvements done so far in the development branch. The new release is just 0.8.1 with that nasty bug fixed.

Diva Canto
--------------------------
Thank you Diva!


Edit: This section below is not applicable.
Is there an easy way compile that fix into the latest dev branch? (I am a git idiot) All I know to do is to "git update" and compile. And I don't see the patch listed in the src listing at http://opensimulator.org/viewgit/?a=shortlog&p=opensim [^]

When we get confirm this fixes the bug, then we can close this issue.


   ~~~ Ferd

(0029057)
Mata Hari (reporter)
2015-08-01 19:51

@Ferd: all Diva's fix does is prevent creation of *new* corrupted assets. It doesn't clean or fix any existing corrupted ones. That fix is already in git master branch (and has been for a month or two). Unfortunately there are a *huge* number of corrupt assets out there already and they continue to be distributed. Until/unless every database is cleaned simultaneously on all grids they will continue to plague us for a long time; hence I tend to support Seth's suggestion for a filter to check and fix any incoming asset (in addition to running this Mantis' python script to fix existing corrupted assets).
(0029058)
Ferd Frederix (reporter)
2015-08-01 19:57
edited on: 2015-08-02 19:04

Thankxx, I read it as fixing this issue. That's why there is no patch for it, then, and thus no need to recompile dev master. She merged it into 0.8.1. Thanks!


Edit: Looks it does fix the issue, except for osGrid. The cleanup of your database still needs to be done.

(0029059)
Diva (administrator)
2015-08-01 20:30

[18:58] <cia-opensim> opensim: diva * re5a1243abc04 OpenSim (4 files in 4 dirs):
[18:58] <cia-opensim> Mantis 0007657 and 0007514. This should alleviate the problem of bad object assets being passed around via HG and archives. No guarantees that all the leaks have been found, but at least it detects and fixes these bad assets upon:
[18:58] <cia-opensim> (1) storing and getting assets over HG -- assuming the core HG asset service is being used (not the case with OSGrid!)
[18:58] <cia-opensim> (2) importing assets via OAR and IAR
[18:58] <cia-opensim>
[18:58] <cia-opensim> Instantiation of bad assets now should also work, instead of producing an exception, but the bad assets themselves aren't being fixed in the DB. That should be done with a cleaning tool -- see Perl script in Mantis 0007657.

Also
[20:27] <cia-opensim> opensim: diva * rc8cd2f18f5af / (2 files in 2 dirs):
[20:27] <cia-opensim> Mantis 0007657: similar fixes for FSAssets.
(0029060)
Mata Hari (reporter)
2015-08-01 20:36

@Diva: thank you!
(0029061)
Seth Nygard (reporter)
2015-08-01 21:04

@Diva Thank You. This filter combined the the cleanup script should prove to be very helpful.
(0029062)
Gavin Hird (reporter)
2015-08-01 22:11

@Diva. Thanks a lot!

It would be fantastic if the last update could be put in a 0.8.1.2 update (if not already done).
(0029066)
aiaustin (developer)
2015-08-02 01:45
edited on: 2015-08-02 02:53

We actually have a relatively stable version in the dev master branch just now... just wondering if its time to do a 0.8.2.0 instead of making a patched version of 0.8.1?

(0029067)
JeffKelley (reporter)
2015-08-02 02:34

Thanks Diva. The post-fixes release will prevent new corruptions to occur and should be deployed asap by grid owners running 0.8.1. Your last fix will make its way to 0.8.2 and should be the definitive answer to this issue. Everyone running master should see no more corruptions. Ever.
(0029069)
aiaustin (developer)
2015-08-02 04:20
edited on: 2015-08-02 04:31

@Diva... a small issue in the commit ... the summary comment says "Oct. 20" in every case (4 instances) rather than "Oct. 2014"... plus 3 more instances in FSAssets vesion...

+ /// Sanitation for bug introduced in Oct. 20 (1eb3e6cc43e2a7b4053bc1185c7c88e22356c5e8)

(0029092)
aiaustin (developer)
2015-08-04 11:20
edited on: 2015-08-05 14:18

Another thought... the fix I think scans in the changes made by @Diva in the OpenSim core code to scan every incoming object via HG or OARs/IARs adds considerably to the time taken to do the operations. I think Ferd at some point suggested just doing the rewrite when the XML parser threw an exception...which would trigger only on the few ( and hopefully reducing) number of affected assets. Just thinking aloud.

(0029359)
Ferd Frederix (reporter)
2015-08-23 08:02

Since this is fixed and released in 0.8.1.1, I am closing it.

- Issue History
Date Modified Username Field Change
2015-07-28 16:09 Ferd Frederix New Issue
2015-07-28 16:09 Ferd Frederix File Added: mantis7514.pl
2015-07-28 19:21 Seth Nygard Note Added: 0029005
2015-07-28 20:20 Seth Nygard Note Added: 0029006
2015-07-29 01:15 Gavin Hird Note Added: 0029007
2015-07-29 03:44 Gavin Hird Note Added: 0029008
2015-07-29 03:45 Gavin Hird Note Edited: 0029008 View Revisions
2015-07-29 07:04 Gavin Hird Note Added: 0029009
2015-07-29 07:22 Seth Nygard Note Added: 0029010
2015-07-29 07:24 Gavin Hird Note Added: 0029011
2015-07-29 07:32 Seth Nygard Note Added: 0029012
2015-07-29 07:33 Gavin Hird Note Added: 0029013
2015-07-29 09:29 Ferd Frederix Note Added: 0029015
2015-07-29 10:55 Neo Cortex Note Added: 0029016
2015-07-29 11:02 Gavin Hird Note Added: 0029017
2015-07-29 11:40 JeffKelley Note Added: 0029018
2015-07-29 11:55 Gavin Hird Note Added: 0029019
2015-07-30 02:41 aiaustin Note Added: 0029021
2015-07-30 02:44 aiaustin Note Edited: 0029021 View Revisions
2015-07-30 03:01 aiaustin Note Added: 0029022
2015-07-30 11:51 Ferd Frederix File Added: mantis7514 Smallest Rev B.pl
2015-07-30 11:51 Ferd Frederix File Added: mantis7514 Full Rev B.pl
2015-07-30 12:06 Ferd Frederix Note Added: 0029030
2015-07-30 12:06 Ferd Frederix Note Added: 0029031
2015-07-30 12:08 Ferd Frederix Note Edited: 0029031 View Revisions
2015-07-30 13:27 aiaustin Note Added: 0029032
2015-07-30 13:40 aiaustin Note Edited: 0029031 View Revisions
2015-07-30 13:56 Ferd Frederix File Deleted: mantis7514 Smallest Rev B.pl
2015-07-30 13:59 Ferd Frederix File Added: mantis7514 Smallest Rev C.pl
2015-07-30 14:02 Ferd Frederix Note Added: 0029035
2015-07-30 14:33 aiaustin Note Added: 0029036
2015-08-01 10:53 Richardus Raymaker Note Added: 0029049
2015-08-01 10:55 Richardus Raymaker Note Edited: 0029049 View Revisions
2015-08-01 10:57 Diva Note Added: 0029050
2015-08-01 11:31 Richardus Raymaker Note Added: 0029051
2015-08-01 11:36 JeffKelley Note Added: 0029052
2015-08-01 11:38 Richardus Raymaker Note Added: 0029053
2015-08-01 17:11 Ferd Frederix Note Added: 0029054
2015-08-01 18:28 Seth Nygard Note Added: 0029055
2015-08-01 18:44 Ferd Frederix Note Added: 0029056
2015-08-01 18:46 Diva Relationship added related to 0007514
2015-08-01 19:51 Mata Hari Note Added: 0029057
2015-08-01 19:57 Ferd Frederix Note Added: 0029058
2015-08-01 19:58 Ferd Frederix Note Edited: 0029056 View Revisions
2015-08-01 20:02 Ferd Frederix Note Edited: 0029058 View Revisions
2015-08-01 20:30 Diva Note Added: 0029059
2015-08-01 20:36 Mata Hari Note Added: 0029060
2015-08-01 21:04 Seth Nygard Note Added: 0029061
2015-08-01 22:11 Gavin Hird Note Added: 0029062
2015-08-02 01:45 aiaustin Note Added: 0029066
2015-08-02 02:34 JeffKelley Note Added: 0029067
2015-08-02 02:51 aiaustin Note Edited: 0029066 View Revisions
2015-08-02 02:53 aiaustin Note Edited: 0029066 View Revisions
2015-08-02 04:20 aiaustin Note Added: 0029069
2015-08-02 04:31 aiaustin Note Edited: 0029069 View Revisions
2015-08-02 19:04 Ferd Frederix Note Edited: 0029058 View Revisions
2015-08-04 10:25 JeffKelley Relationship added related to 0007668
2015-08-04 11:20 aiaustin Note Added: 0029092
2015-08-04 12:19 JeffKelley Note Added: 0029093
2015-08-04 12:29 aiaustin Note Added: 0029094
2015-08-04 12:30 aiaustin Note Edited: 0029092 View Revisions
2015-08-04 12:43 JeffKelley Note Deleted: 0029093
2015-08-05 14:18 aiaustin Note Deleted: 0029094
2015-08-05 14:18 aiaustin Note Edited: 0029092 View Revisions
2015-08-23 08:02 Ferd Frederix Note Added: 0029359
2015-08-23 08:02 Ferd Frederix Resolution open => fixed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker