Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008960opensim[MISC] Compiling / Buildingpublic2022-01-12 15:272022-01-14 15:20
ReporterSnoopy 
Assigned To 
PriorityhighSeveritycrashReproducibilityalways
StatusnewResolutionopen 
PlatformIntelOperating SystemUbuntuOperating System Version21.04
Product Version0.9.2.1Dev 
Target VersionFixed in Version 
Summary0008960: MySql.Data crashes in combination with MariaDB
DescriptionOne one of my servers MySql.Data suddenly started failing, causing segmentation faults. The server (amd64 architecture) uses Ubuntu 21.04 with Mono 6.12.0.122, MariaDB 10.5.13 and the latest OpenSim version.

GDB debugging shows that an exception seems to be thrown when MySql.Data.MySqlClient.MySqlConnectAttrs:InitFramework is executed when the connection to the database is established. The problem might be related to communications issues between MySql.Data and the database or maybe caused by thread pools issues of the Mono runtime environment.

As mentioned before, these issues did suddenly start, without having done any updates or configuration changes. Other database connections using the "mysql" command line tool or using PHP scripts still work without any problems. Another server using the same setup runs since many months without problems. A check of the hardware was done, but without identifying any hardware issues on that server.

I did reinstall MariaDB, Mono and OpenSim on that server, not changing the result. Everything works with the exception of the MySql.Data crashes mentioned before.

Does anybody have an idea how to identify the reason for these issues? Might it be one of the MySql.Data related issues we need to expect when using MariaDB?

Any help or ideas are welcome. Thanks in advance.
TagsNo tags attached.
Git Revision or version number
Run ModeStandalone (1 Region)
Physics EngineBulletSim
Script EngineXEngine
EnvironmentMono / Linux64
Mono Version6.x
Viewer
Attached Fileslog file icon OpenSim.log [^] (4,960 bytes) 2022-01-13 13:14
txt file icon gdb.txt [^] (6,192 bytes) 2022-01-13 13:14 [Show Content]
txt file icon syslog.txt [^] (369 bytes) 2022-01-13 13:14 [Show Content]
txt file icon mariadb.txt [^] (177 bytes) 2022-01-13 13:15 [Show Content]

- Relationships

-  Notes
(0038337)
tampa (reporter)
2022-01-13 12:49

Can we get some actual logs with the errors?
(0038338)
Snoopy (administrator)
2022-01-13 13:17
edited on: 2022-01-13 13:35

I have attached an OpenSim log and a GDB session showing the stack when the segmentation fault happens as well as corresponding syslog and MariaDB error.log information.

Do you have an idea why MySql.Data crashes and how this can be fixed?

The crash happens when myConnection.Open() is executed.

(0038339)
tampa (reporter)
2022-01-14 14:46

These logs look odd, did you setup OpenSim to write a pid file? Also from the looks of things that's not something directly with OpenSim, but the system itself.

Check clocksource, if disk is full and perhaps do some system diagnostics, these errors sound more like faulty hardware.
(0038340)
Snoopy (administrator)
2022-01-14 14:59

Thank you for your answer. I did further analyse the problem by comparing two very similar servers, using the same software versions. One server runs smoothly since many months and the server mentioned here suddenly started having issues.

Beside this I did narrow down the problem, by writing a small program, independent of OpenSim, that also uses MySql.Data and causes the same segmentation fault. The problem is that MySql.Data always triggers an exception while connecting to the database server. This exception is usually properly handled by an exception handler. On the server having issues the software crashes instead.

On the server working properly the trace looks as follows, while on the faulty server the crash happens right between the two lines shown (before the "EXCEPTION" line).

    [...]
    [0x7f783bb1e780: 0.54442 12] ENTER:c MySql.Data.MySqlClient.MySqlConnectAttrs:InitFramework ()()
    [0x7f783bb1e780:] EXCEPTION handling: System.NullReferenceException: Object reference not set to an instance of an object
    EXCEPTION: catch found at clause 0 of MySql.Data.MySqlClient.MySqlConnectAttrs:InitFramework ()
    [...]

For me this is an indication that there must be some hardware problem affecting exception handling, a low level functionality of the processor.

Besides this problems started suddenly without having done any software or configuration changes. All did run well for weeks. Then one day I did restart a number of process without problems, until suddenly software using the MySql.Data library (triggering exceptions internally), like OpenSim, did not start anymore.

My analysis and the symptoms described bring me to the conclusion that the processor or the mainboard must have some hardware issue. I will replace the server hardware and the chances are high that after this replacement OpenSim will work properly again.
(0038341)
tampa (reporter)
2022-01-14 15:09

Yeah a simple test program with nothing else in it failing straight up is not a good sign. Since you already tried everything else, only other thing I would think of would be a bad update somewhere leaving incompatible files behind in the database install or install of mono. Doing full apt purge remove, but depending on the server itself it might be easier just to refresh the hardware at this point then spending a ton of time debugging with no clear end in sight.

Feel free to report back if the issue persists, but it is starting to look very much like not an OpenSim bug.
(0038342)
Snoopy (administrator)
2022-01-14 15:20
edited on: 2022-01-14 15:22

Thanks for your very quick reply. I did already purge and reinstall mono and the database multiple times, even trying different mono versions. Including manual checks that there really were no left overs before reinstalling. All mono versions behave exactly the same in respect to this segmentation fault caused by an exception within MySql.Data.

Additionally I did use my two servers to run the mono program on one of the servers and connecting to the database on the other server and vice versa. This did show that the database is not the trouble maker. It is definitively that single point within the MySql.Data library, that always uses an exception when connecting to the database.

I really tried everything and the only thing left is a hardware problem. Everything points this direction. I will post how finally this issue could be fixed. It is definitively not an OpenSim bug, but it affects OpenSim usage on that server.


- Issue History
Date Modified Username Field Change
2022-01-12 15:27 Snoopy New Issue
2022-01-13 12:49 tampa Note Added: 0038337
2022-01-13 13:14 Snoopy File Added: OpenSim.log
2022-01-13 13:14 Snoopy File Added: gdb.txt
2022-01-13 13:14 Snoopy File Added: syslog.txt
2022-01-13 13:15 Snoopy File Added: mariadb.txt
2022-01-13 13:17 Snoopy Note Added: 0038338
2022-01-13 13:35 Snoopy Note Edited: 0038338 View Revisions
2022-01-14 14:46 tampa Note Added: 0038339
2022-01-14 14:59 Snoopy Note Added: 0038340
2022-01-14 15:09 tampa Note Added: 0038341
2022-01-14 15:20 Snoopy Note Added: 0038342
2022-01-14 15:20 Snoopy Note Edited: 0038342 View Revisions
2022-01-14 15:21 Snoopy Note Edited: 0038342 View Revisions
2022-01-14 15:21 Snoopy Note Edited: 0038342 View Revisions
2022-01-14 15:22 Snoopy Note Edited: 0038342 View Revisions


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker