Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008707opensim[REGION] OpenSim Corepublic2020-05-24 00:462020-05-25 15:16
ReporterManwa Pastorelli 
Assigned To 
PriorityhighSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformLinuxOSUbuntuOS Version20.04
Product Version0.9.1.0 
Target VersionFixed in Version 
Summary0008707: MySql8 Compatibility issues mb3 not working as expected
DescriptionCharacter Set / Collate issues when using MySql8.

I have tried this with the same results in 0.911 and 0.92, but the product version above doesn't list them.

This is probably actually an issue with MySql; however, since they have marked mb3 as deprecated for later removal, I doubt they will fix this.

This only affects certain items in opensim when used in particular ways. I have only found an issue using Satyr Farm Animals. However, the potential exists for many more.

I have created tables in the following ways:

CREATE DATABASE mydatabase CHARACTER SET utf8mb3

CREATE DATABASE mydatabase CHARACTER SET utf8mb3 COLLATE utf8_general_ci;

CREATE DATABASE mydatabase CHARACTER SET utf8mb3 COLLATE utf8mb3_general_ci;

**mb4 was done as a test since the error string looks like it should be mb4
CREATE DATABASE mydatabase CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

All of which generate the same results. They all work for most use cases, but if certain characters are used errors are generated. When this occurs the items appear to work in the world; however, errors can be seen in the console. When the simulator is restarted, the affected items do not save—leaving users without their items.

MySql8 documentation stats that UTF8 is now an alias for UTF8mb4 and will be removed later. It looks as though mb3 doesn't work as expected either.
Steps To Reproduce1. Create a simulator running on a database created with MySql8 and any of the above creation methods.

2. Collect the Satyr Farm products from hg.osgrid.org:80:OpenSimWorld

3. Rez any of the animals.

4. Quit the simulator and look at the console to see the errors.


Additional Information2020-05-24 09:06:30,653 ERROR [REGION DB]: MySQL error in ExecuteNonQuery: Incorrect string value: '\xE2\x99\x80\x0A0 ...' for column 'Text' at row 1
2020-05-24 09:06:30,656 ERROR [SCENE]: Storing of SF Elephant, dc5e01a2-6acd-4a63-8831-c6089855220e in Testing-0920-256 failed with exception Incorrect string value: '\xE2\x99\x80\x0A0 ...' for column 'Text' at row 1 at MySql.Data.MySqlClient.MySqlStream.ReadPacket () [0x00096] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.NativeDriver.GetResult (System.Int32& affectedRow, System.Int64& insertedId) [0x00026] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.Driver.GetResult (System.Int32 statementId, System.Int32& affectedRows, System.Int64& insertedId) [0x00000] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.Driver.NextResult (System.Int32 statementId, System.Boolean force) [0x00023] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.MySqlDataReader.NextResult () [0x00205] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00315] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader () [0x00000] in <0004ab8b375b422f9000ac25a68089d9>:0
  at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery () [0x00033] in <0004ab8b375b422f9000ac25a68089d9>:0
  at OpenSim.Data.MySQL.MySQLSimulationData.ExecuteNonQuery (MySql.Data.MySqlClient.MySqlCommand c) [0x00024] in <44a4f7ec66f548ab8c3c1e4870853388>:0
  at OpenSim.Data.MySQL.MySQLSimulationData.StoreObject (OpenSim.Region.Framework.Scenes.SceneObjectGroup obj, OpenMetaverse.UUID regionUUID) [0x00078] in <44a4f7ec66f548ab8c3c1e4870853388>:0
  at OpenSim.Services.SimulationService.SimulationDataService.StoreObject (OpenSim.Region.Framework.Scenes.SceneObjectGroup obj, OpenMetaverse.UUID regionUUID) [0x00000] in <8e9d48cb71f0426294098b5529543b2f>:0
  at OpenSim.Region.Framework.Scenes.SceneObjectGroup.ProcessBackup (OpenSim.Region.Framework.Interfaces.ISimulationDataService datastore, System.Boolean forcedBackup) [0x00341] in <3614d381280e4d09950777f021d0c4bf>:0
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script EngineXEngine
EnvironmentMono / Linux64
Mono Version6.x
ViewerFirestorm 6.3.10 (58211)
Attached Files

- Relationships

-  Notes
(0036486)
tampa (reporter)
2020-05-24 00:56

Let me guess, using special chars? That is something one should avoid if possible regardless of whether the DB barfs them back out or not. Between operating systems and keyboards you get small differences in type and some work fine, others break all the things.

I recently installed that farm system from the source and had no such issues in MariaDB however, which I suspect is because they have retained some greater exceptions to charset rules or whatever.

A similar issue with collation and charset has already been reported, also with the note to avoid special chars as much as possible. Likely the same fix to switching to a more universal version of utf-8. You can either switch the databases manually or remove the special chars. Can't tell you if/when a fix to the DB setup may come.
(0036488)
UbitUmarov (administrator)
2020-05-24 02:46

this is reported several times
but if the error shows utf8, then
'\xE2\x99\x80\x0A0.. is a 3 bytes char followed by ilegal a0 ???
(0036493)
Manwa Pastorelli (reporter)
2020-05-24 05:09
edited on: 2020-05-24 05:10

Should I read that as the recommended way to go for the time being as switching to MariaDB?

I fixed it for my single affected customer by taking out a VPS using an older version of Ubuntu, and as such Mysql. I don't consider this to be a good solution looking forwards.

According to Maria's migration documentation, once you go above MySql5, migration to MariaDB is a one-way trip. I will do it if that is the direction Opensimulator will go in.

(0036494)
UbitUmarov (administrator)
2020-05-24 05:13

there was no such recomendation so far.
also some of our tables (one of our groups modules) will blow up unless mariaDB is also set to utf8mb3
(unless the max index size did changed recently)
(0036501)
Manwa Pastorelli (reporter)
2020-05-25 05:22

@ Tampa, can I ask you which charset and collate you are using? I just installed MariaDB to try, and I am getting exactly the same results. I should probably ask which version of MariaDB as well I guess.

mysql Ver 15.1 Distrib 10.3.22-MariaDB is the one I just tried.
(0036502)
tampa (reporter)
2020-05-25 05:29

I use the older stable of MariaDB 10.1 still since that is the version the distro supports. The collation and charset depends on the table, but most of them are either utf-8 general ci and latin1 swedish ci with the charsets for full unicode support.

It would probably make some sense to explore which charset and collation make the most sense and perhaps adjust migrations in regards to the DB used, but that comes down to a lot of factors so not as easy as it sounds.

Question is still if the this is a bug or misuse though.
(0036503)
Manwa Pastorelli (reporter)
2020-05-25 08:34

It is miss-use, but it is a form of miss-use which worries me. I first became aware of this because I noticed my server memory use was dramatically higher than usual (15gb above normal after 24 hours of uptime).

I don't know if it is an opensim memory leak or a MySQL memory leak, but the repetition of this error would cause problems quickly with a large number of items rezzed. In my case, my customer had around 20 items out.
(0036504)
tampa (reporter)
2020-05-25 12:51

That memory leak may be due to something else, which version of mono and libgdiplus are you running?
(0036505)
Manwa Pastorelli (reporter)
2020-05-25 14:52

Libgdipluss 6.0.4+dfsg-2

Mono JIT compiler version 6.8.0.105
(0036506)
tampa (reporter)
2020-05-25 15:05

You are likely experiencing a problem with libgdiplus then, because that release is broken in conjunction with dynamic textures. You can switch to the mono preview builds which have libgdiplus 6.0.5 that fixes these leaks. OpenSim runs fine on mono 6.10 as well.
(0036507)
Manwa Pastorelli (reporter)
2020-05-25 15:16

thankyou, I will do that then.

- Issue History
Date Modified Username Field Change
2020-05-24 00:46 Manwa Pastorelli New Issue
2020-05-24 00:48 Manwa Pastorelli Category [MISC] DSG => [REGION] OpenSim Core
2020-05-24 00:48 Manwa Pastorelli Description Updated View Revisions
2020-05-24 00:56 tampa Note Added: 0036486
2020-05-24 02:46 UbitUmarov Note Added: 0036488
2020-05-24 05:09 Manwa Pastorelli Note Added: 0036493
2020-05-24 05:10 Manwa Pastorelli Note Edited: 0036493 View Revisions
2020-05-24 05:13 UbitUmarov Note Added: 0036494
2020-05-25 05:22 Manwa Pastorelli Note Added: 0036501
2020-05-25 05:29 tampa Note Added: 0036502
2020-05-25 08:34 Manwa Pastorelli Note Added: 0036503
2020-05-25 12:51 tampa Note Added: 0036504
2020-05-25 14:52 Manwa Pastorelli Note Added: 0036505
2020-05-25 15:05 tampa Note Added: 0036506
2020-05-25 15:16 Manwa Pastorelli Note Added: 0036507


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker