Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008919opensim[GRID] Other Servicepublic2021-07-20 10:342021-07-21 17:09
ReporterCasperWarden 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformOperating SystemOperating System Version
Product Version0.9.1.0 
Target VersionFixed in Version 
Summary0008919: OpenSim does not support MariaDB 10.6
DescriptionOn http://opensimulator.org/wiki/Configuration, [^] the following is stated:

"You need to make sure the selected charset is utf8mb3 (ie 3 bytes)"

However, MariaDB 10.6 (and possibly other flavours of MySQL) changes the advertisement of this character set from "utf8" to "utf8mb3".

The Mysql.Data.dll connector from Oracle does not recognise this character set.

To fix this, a new version of Mysql.Data.dll must be built with the following addition to CharSetMap.cs:

_mapping.Add("utf8mb3", new CharacterSet("utf-8", 3));

I have attached my compiled binary (this also updates the connector to version 8), if it is useful.

Steps To Reproduce* Use MariaDB 10.6.3 with OpenSim with the recommended character set of utf8mb3
* Start OpenSim
TagsNo tags attached.
Git Revision or version number
Run ModeStandalone (1 Region) , Standalone (Multiple Regions) , Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script EngineXEngine
EnvironmentMono / Linux64
Mono Version6.x
Viewer
Attached Fileszip file icon MySQL Data Connector.zip [^] (698,516 bytes) 2021-07-20 10:34

- Relationships

-  Notes
(0037886)
tampa (reporter)
2021-07-20 10:47

Another option is to change the migrations from utf8 to utf8mb4, which both resolves this and potentially ends up more compatible with special chars, given apparently some have taken to using them in prim names and descriptions.

To me that seems both easier, since you first need to check what else has changed in the connector and could break compatibility, and more usable across database providers, given utf8 alone is a bit ambiguous and could be seen as a lot of things, mb3, mb4 or whatever else is out there.
(0037887)
UbitUmarov (administrator)
2021-07-20 11:00

opensim has no reference to uft8mb3 or utf8mb4, just utf8.

Database use 3 of 8 bytes for utf8, according to their configuration options

Some opensim tables where made when utf8 did mean 3 bytes by default, so some fields are just too large if 4 bytes is used.

Meanwhile some of those tables where already fixed at the creation time, so 4 bytes variant can be used.

Problem remains on current tables, so 3 bytes need to be used or the tables changed by hand. Some fields where set utf8 when they should be latin1.
Note that this does matter, because some are indexes and on those full 3/4 bytes are used to store each char.

Hope this helps
(0037888)
BillBlight (developer)
2021-07-20 11:10

According to their page ..

"Character SetsĀ¶
The utf8 character set (and related collations) is now by default an alias for utf8mb3 rather than the other way around. It can be set to imply utf8mb4 by changing the value of the old_mode system variable (MDEV-8334)"

https://mariadb.com/kb/en/changes-improvements-in-mariadb-106/ [^]
(0037889)
UbitUmarov (administrator)
2021-07-20 11:14

Problem is that some distros, by pure religion, change that to the 4 bytes variant
(0037890)
tampa (reporter)
2021-07-21 01:46

So far I have changed RegionStore, EstateStore and the vast majority of grid-side tables to mb4, nothing has complained about field size being too large yet. Is that really still a limit or was that a limit in older versions of mysql?
(0037891)
UbitUmarov (administrator)
2021-07-21 10:05

Ok seem more information about this issue, and did compile mysql.data.dll version 6.9.11 with the extra mapping as suggested.

Can't still jump to version 8.

Can you please test? note: do runprebuild and compile again, or the version change can cause issues.. I forgot to mention it on the commit..
(0037892)
aiaustin (developer)
2021-07-21 12:14

Will this change affect people running MySQL 5.6 or 5.7?
(0037893)
UbitUmarov (administrator)
2021-07-21 12:21

should not
(0037894)
aiaustin (developer)
2021-07-21 12:29

Thx Ubit. Diva's Wifi makes use of a custom Diva.Data.MySQL.dll. I wonder if she spots this update and if it affects WiFi for when its prepared for 0.9.2.0? I will draw her attention to this.

    StorageProvider = "Diva.Data.MySQL.dll"
(0037895)
CasperWarden (reporter)
2021-07-21 14:06

What's the issue with version 8 of the connector? It compiles and runs fine for me?
(0037896)
UbitUmarov (administrator)
2021-07-21 16:09

Versions 8 are now .Net Core and .Net5 oriented. Older frameworks support is less and less reliable (kids do not care about old code)
Their current install adds more dlls with specific version that may collide with future opensim use (like google protobuf).
Because the use of Nuget they are just a total pain to control, with possible stupid blind updates.

Are supposed to be signed, but only oracle owns the key, ofc.

They seem to add nothing useful for opensim. 6.9.x works with servers 8.x hopeful now with maria 10.6 also
(0037897)
CasperWarden (reporter)
2021-07-21 16:28
edited on: 2021-07-21 16:30

The extra dlls are unnecessary for the .net 4.8 build (only Ubiety.Dns.Core.dll is required). Protobuf is not needed.

All the other crap is only required for the .net core version. Idk why.

(0037898)
UbitUmarov (administrator)
2021-07-21 17:09

actually if you look the case 4.5.2 and possible 4.8 even adds parts of .net core support

in core they add protobuf, ssl (tls) support dlls, etc
.net 5 includes some like protobuf, so not there..

Of the new things protobuf could be only interesting feature, improving the comms to server using it.
But seems not in use on our case, even if we do add it.

So my decision to stay on 6.9.x with your fix, for now, just hope it works

- Issue History
Date Modified Username Field Change
2021-07-20 10:34 CasperWarden New Issue
2021-07-20 10:34 CasperWarden File Added: MySQL Data Connector.zip
2021-07-20 10:47 tampa Note Added: 0037886
2021-07-20 11:00 UbitUmarov Note Added: 0037887
2021-07-20 11:10 BillBlight Note Added: 0037888
2021-07-20 11:14 UbitUmarov Note Added: 0037889
2021-07-21 01:46 tampa Note Added: 0037890
2021-07-21 10:05 UbitUmarov Note Added: 0037891
2021-07-21 12:14 aiaustin Note Added: 0037892
2021-07-21 12:21 UbitUmarov Note Added: 0037893
2021-07-21 12:29 aiaustin Note Added: 0037894
2021-07-21 14:06 CasperWarden Note Added: 0037895
2021-07-21 16:09 UbitUmarov Note Added: 0037896
2021-07-21 16:28 CasperWarden Note Added: 0037897
2021-07-21 16:30 CasperWarden Note Edited: 0037897 View Revisions
2021-07-21 17:09 UbitUmarov Note Added: 0037898


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker