MantisBT - opensim
View Issue Details
0008423opensim[GRID] Asset Servicepublic2018-12-10 09:542019-01-23 15:38
kenvc 
 
highcrashalways
newopen 
HP dual Xeon CPUs 32BG ramWindowsServer 2012
0.9.0.1 
 
Latest Dev Master as of 12/10/2018
Grid (Multiple Regions per Sim)
ubODE
.NET / Windows64
None
N/A
0008423: Invalid formatted GUID in the DB causes the instance to crash on startup. Log file does not show the bad GUID so cannot fix it
Invalid formatted GUID in the DB causes the instance to crash on startup. I have 2 instances now that have this problem. I run a repair on the MySQL database and that did not help. These instances are connected to OSGrid.

The log file does not show the bad GUID so cannot manually find and fix or delete the asset with the bad GUID from the database.

Please show the bad GUID in the log file so the item can be found and manually fixed in the database.
2018-12-10 11:39:07,094 INFO [SCENE]: Loading objects from datastore
2018-12-10 11:39:07,718 DEBUG [REGION DB]: Loaded 5000 prims...
2018-12-10 11:39:08,264 DEBUG [REGION DB]: Loaded 10000 prims...
2018-12-10 11:39:08,778 DEBUG [REGION DB]: Loaded 15000 prims...
2018-12-10 11:39:09,324 DEBUG [REGION DB]: Loaded 20000 prims...
2018-12-10 11:39:09,855 DEBUG [REGION DB]: Loaded 2601 objects using 23531 prims
2018-12-10 11:39:10,791 FATAL Fatal error: System.FormatException: Guid should contain 32 digits with 4 dashes (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).
   at System.Guid.GuidResult.SetFailure(ParseFailureKind failure, String failureMessageID, Object failureMessageFormatArgument, String failureArgumentName, Exception innerException)
   at System.Guid.TryParseGuidWithDashes(String guidString, GuidResult& result)
   at System.Guid.TryParseGuid(String g, GuidStyles flags, GuidResult& result)
   at System.Guid..ctor(String g)
   at MySql.Data.Types.MySqlGuid.MySql.Data.Types.IMySqlValue.ReadValue(MySqlPacket packet, Int64 length, Boolean nullVal)
   at MySql.Data.MySqlClient.ResultSet.ReadColumnData(Boolean outputParms)
   at MySql.Data.MySqlClient.ResultSet.NextRow(CommandBehavior behavior)
   at MySql.Data.MySqlClient.MySqlDataReader.Read()
   at OpenSim.Data.MySQL.MySQLSimulationData.LoadItems(SceneObjectPart prim) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Data\MySQL\MySQLSimulationData.cs:line 582
   at OpenSim.Data.MySQL.MySQLSimulationData.LoadObjects(UUID regionID) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Data\MySQL\MySQLSimulationData.cs:line 551
   at OpenSim.Services.SimulationService.SimulationDataService.LoadObjects(UUID regionUUID) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Services\SimulationService\SimulationDataService.cs:line 99
   at OpenSim.Region.Framework.Scenes.Scene.LoadPrimsFromStorage(UUID regionID) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\Framework\Scenes\Scene.cs:line 2256
   at OpenSim.OpenSimBase.CreateRegion(RegionInfo regionInfo, Boolean portadd_flag, Boolean do_post_init, IScene& mscene) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\Application\OpenSimBase.cs:line 489
   at OpenSim.OpenSimBase.CreateRegion(RegionInfo regionInfo, Boolean portadd_flag, IScene& scene) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\Application\OpenSimBase.cs:line 387
   at OpenSim.ApplicationPlugins.LoadRegions.LoadRegionsPlugin.PostInitialise() in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\ApplicationPlugins\LoadRegions\LoadRegionsPlugin.cs:line 130
   at OpenSim.OpenSimBase.StartupSpecific() in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\Application\OpenSimBase.cs:line 285
   at OpenSim.OpenSim.StartupSpecific() in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\Application\OpenSim.cs:line 211
   at OpenSim.Framework.Servers.BaseOpenSimServer.Startup() in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Framework\Servers\BaseOpenSimServer.cs:line 198
No tags attached.
patch primitemscatch.patch (1,397) 2018-12-28 23:06
http://opensimulator.org/mantis/file_download.php?file_id=4813&type=bug
? MySQLSimulationData.cs (119,110) 2019-01-21 10:33
http://opensimulator.org/mantis/file_download.php?file_id=4826&type=bug
Issue History
2018-12-10 09:54kenvcNew Issue
2018-12-15 20:37Ferd FrederixNote Added: 0033635
2018-12-16 10:39kenvcNote Added: 0033638
2018-12-16 10:41BillBlightNote Added: 0033639
2018-12-19 17:02kenvcNote Added: 0033654
2018-12-20 20:03kenvcNote Added: 0033655
2018-12-28 23:06BillBlightFile Added: primitemscatch.patch
2018-12-28 23:06BillBlightNote Added: 0033685
2018-12-28 23:12BillBlightNote Edited: 0033685bug_revision_view_page.php?bugnote_id=33685#r7470
2018-12-29 08:29UbitUmarovNote Added: 0033686
2018-12-29 18:19kenvcNote Added: 0033689
2018-12-29 18:21kenvcNote Edited: 0033689bug_revision_view_page.php?bugnote_id=33689#r7474
2019-01-21 10:32kenvcNote Added: 0033890
2019-01-21 10:33kenvcFile Added: MySQLSimulationData.cs
2019-01-21 12:40BillBlightNote Added: 0033891
2019-01-21 14:44kenvcNote Added: 0033893
2019-01-21 14:46kenvcNote Edited: 0033893bug_revision_view_page.php?bugnote_id=33893#r7616
2019-01-21 14:50kenvcNote Edited: 0033893bug_revision_view_page.php?bugnote_id=33893#r7617
2019-01-21 15:15BillBlightNote Added: 0033894
2019-01-21 15:47kenvcNote Added: 0033895
2019-01-21 16:02BillBlightNote Added: 0033896
2019-01-21 16:08kenvcNote Added: 0033897
2019-01-23 08:47paela argusNote Added: 0033918
2019-01-23 08:50paela argusNote Edited: 0033918bug_revision_view_page.php?bugnote_id=33918#r7637
2019-01-23 15:38kenvcNote Added: 0033923

Notes
(0033635)
Ferd Frederix   
2018-12-15 20:37   
I second this. I have had to spend many hours to track down a bad UUID.
(0033638)
kenvc   
2018-12-16 10:39   
I have had 2 sims completely down for the last few weeks for this same reason.

Just adding the bad GUID to one of the error message lines would be enough of a clue to allow us to find and fix the source of the problem.
(0033639)
BillBlight   
2018-12-16 10:41   
weird, I see this on linux once in a while but does not crash, just keeps chugging along ..
(0033654)
kenvc   
2018-12-19 17:02   
When running under Windows it always crashes the entire instance on startup when loading the prims and it immediately exits.
(0033655)
kenvc   
2018-12-20 20:03   
I'm sure this is not the right way to handle this issue, but I added a try catch around the part of the code that was causing the crash when a bad UUID was encountered. I also added a log entry indicating that a prim had a bad UUID.

The instance started normally after this was done without the unhandled exception and the log file indicated it had 4 prims with bad UUIDs. The bad prim's UUID and name is blank at this point in the code so I still cannot show it in the log, but at least the instance is no longer crashing on startup.
(0033685)
BillBlight   
2018-12-28 23:06   
(edited on: 2018-12-28 23:12)
I have attached a patch that will print the corrupt uuid to the console, if it is in the primitems table, which I suspect ..

But this should actually spot them on any item load, I think ...

Not for production use, but may get you going ..

(0033686)
UbitUmarov   
2018-12-29 08:29   
oops
I did change code to just ignore returing uuid.zero
btw don't like use of console there
(0033689)
kenvc   
2018-12-29 18:19   
(edited on: 2018-12-29 18:21)
Ubit, I reverted to the latest dev master code, compiled and started up all the sims, but the 2 sims with prims that have bad UUIDs are still crashing the entire instance they are in.

(0033890)
kenvc   
2019-01-21 10:32   
Ubit,

I apparently do not know how to create a patch in Windows, but I have had to keep making the same code change in the MySQLSimulationData.cs code to keep several of my instances from crashing on startup. I pulled the latest dev master code a few minutes ago and added the changes to it and attached it to this mantis. Is there any way to get these lines of new code into the dev master to fix this issue for those of us that are having it?
(0033891)
BillBlight   
2019-01-21 12:40   
@kenvc , it would be a really bad idea to include permanent code in master that skips corrupt UUID's ..

The patch I attached will print out the uuid's of the bad prims, I'd suggest using it and then deleting those prims from the DB , most likely in the PrimItems table.

Then going back to master code.
(0033893)
kenvc   
2019-01-21 14:44   
(edited on: 2019-01-21 14:50)
I applied your patch back when it was first added here and the 2 instances with this problem still crashed on startup. If this patch really did work on your system that has a bad GUID without crashing, what OS are you using?

The best solution would be something that still allows the sim to run without crashing, but that also puts the details about the bad prim(s) in the log file so they could be removed from the DB later.

(0033894)
BillBlight   
2019-01-21 15:15   
That is exactly what my code does for me, it does not crash, prints the uuid and moves on ..
(0033895)
kenvc   
2019-01-21 15:47   
What OS are you running it on?
(0033896)
BillBlight   
2019-01-21 16:02   
As I said earlier I run on Linux, Ubuntu to be more accurate ..

The issue is you seem to have something really corrupt, and it would be better to find and fix it in your DB than to have the code ignore it ..

Not sure why it just outright crashes for you, if you try my patch check the logs and see if it gives you the UUID to the log or crashlog ..
(0033897)
kenvc   
2019-01-21 16:08   
That's the problem. As you can see from other Windows user comments, a bad GUID causes an instance crash on startup when running under Windows, but it does not cause a crash when running on Linux.

If it was not crashing, I would have not worried about the bad GUID and never even mentioned this as a problem.
(0033918)
paela argus   
2019-01-23 08:47   
(edited on: 2019-01-23 08:50)
its place is not in log or debug display opensim is already hugely polluting display debug much too much.
The problem of Mistletoe must be solved directly with the simulator he finds it then he removes it in the database it is not more complicated and it is clean at work!

(0033923)
kenvc   
2019-01-23 15:38   
Its very difficult and time consuming to figure out which prim(s) are causing the issue when the instance is running under Windows OS and crashes before the suggested patch can record or display which prim(s) caused the problem.

I and running 200+ sims and simply do not have time to go through the MySQL prim table line by line until a GUID is found that does not look right (missing a dash or whatever). If someone could come up with a MySQL script that could find a GUID that was not perfectly formed in the prim table (I assume that's most likely where the bad GUIDs are located), I would be happy to try that.