Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007123opensim[GRID] Other Servicepublic2014-04-19 13:002014-05-21 19:11
Reportersmxy 
Assigned Toorenh 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Versionmaster (dev code) 
Target VersionFixed in Versionmaster (dev code) 
Summary0007123: V2 Offline Messaging: retrieval is broken.
DescriptionSaving offline messages works fine. Retrieving them is broken. See exception.
Steps To ReproduceObserve exception at console when avatar, with offline messages waiting to be retrieved, logs in.
Additional Information15:57:58 - [OfflineIM.V2]: Retrieving stored messages for 5adfe612-116c-4146-8f0e-ae839e018615
15:57:58 - [LLCLIENTVIEW]: Caught exception while processing OpenMetaverse.Packets.RetrieveInstantMessagesPacket for Cabana Boy, Cannot cast from source type to destination type. at OpenSim.OfflineIM.OfflineIMServiceRemoteConnector.GetMessages (UUID principalID) [0x0007c] in /home/ste/grid/src/coreOS/build/build-r24620/OpenSim/Addons/OfflineIM/Remote/OfflineIMServiceRemoteConnector.cs:88
  at OpenSim.OfflineIM.OfflineIMRegionModule.RetrieveInstantMessages (IClientAPI client) [0x00027] in /home/ste/grid/src/coreOS/build/build-r24620/OpenSim/Addons/OfflineIM/OfflineIMRegionModule.cs:172
  at OpenSim.Region.ClientStack.LindenUDP.LLClientView.HandleRetrieveInstantMessages (IClientAPI sender, OpenMetaverse.Packets.Packet Pack) [0x00056] in /home/ste/grid/src/coreOS/build/build-r24620/OpenSim/Region/ClientStack/Linden/UDP/LLClientView.cs:11538
  at OpenSim.Region.ClientStack.LindenUDP.LLClientView.ProcessSpecificPacketAsync (System.Object state) [0x0001c] in /home/ste/grid/src/coreOS/build/build-r24620/OpenSim/Region/ClientStack/Linden/UDP/LLClientView.cs:738
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
EnvironmentMono / Linux64
Mono Version3.2
Viewer
Attached Files

- Relationships
related to 0006973closedorenh Offline IM's Fail 

-  Notes
(0025810)
smxy (reporter)
2014-04-20 09:53

After commenting out the try ... catch code, around line 738 of LLClientView.cs, the exception report changes, though not significantly:

12:37:43 - [UTIL]: Util STP threadfunc 22 terminated with error System.InvalidCastException: Cannot cast from source type to destination type.
  at OpenSim.OfflineIM.OfflineIMServiceRemoteConnector.GetMessages (UUID principalID) [0x0007c] in /home/ste/grid/src/coreOS/build/build-r24627/OpenSim/Addons/OfflineIM/Remote/OfflineIMServiceRemoteConnector.cs:88
  at OpenSim.OfflineIM.OfflineIMRegionModule.RetrieveInstantMessages (IClientAPI client) [0x00027] in /home/ste/grid/src/coreOS/build/build-r24627/OpenSim/Addons/OfflineIM/OfflineIMRegionModule.cs:172
  at OpenSim.Region.ClientStack.LindenUDP.LLClientView.HandleRetrieveInstantMessages (IClientAPI sender, OpenMetaverse.Packets.Packet Pack) [0x00056] in /home/ste/grid/src/coreOS/build/build-r24627/OpenSim/Region/ClientStack/Linden/UDP/LLClientView.cs:11538
  at OpenSim.Region.ClientStack.LindenUDP.LLClientView.ProcessSpecificPacketAsync (System.Object state) [0x0001b] in /home/ste/grid/src/coreOS/build/build-r24627/OpenSim/Region/ClientStack/Linden/UDP/LLClientView.cs:738
  at OpenSim.Framework.Util+<FireAndForget>c__AnonStorey0.<>m__1 (System.Object o) [0x00091] in /home/ste/grid/src/coreOS/build/build-r24627/OpenSim/Framework/Util.cs:2148

It was stated by someone else, that V2 Offline Messages works for them, so the implication is that there is something wrong in my ROBUST grid's configuration. If there is, I've been unable to find it. Though, I have to say, there is not separate configuration for the ability to send and receive the messages, so that fact that they are correctly stored suggests that my config is correct, or mostly so. It's also not obvious to me how a config error can cause a cast error a such a low level, but then, I'm not a developer ... and perhaps I do have an error of some kind.

I'll review my config again and will post it next for hopeful review by more critical eyes, assuming I still find nothing wrong.
(0025811)
smxy (reporter)
2014-04-20 10:23

Comparing my config against http://opensimulator.org/wiki/Offline_Instant_Messaging_v2 [^]
I find nothing wrong, though my table is slightly different (I think it's changed since the wiki page was written). NOTE: my grid uses 8012/8013, rather than 8002/8003, so that's not a typo.

From my region config:

[Messaging]
    OfflineMessageModule = "Offline Message Module V2"
    OfflineMessageURL = http://sanctuary.homelinux.org:8013 [^]
    StorageProvider = OpenSim.Data.MySQL.dll
    MuteListModule = MuteListModule
    MuteListURL = http://sanctuary.homelinux.org/Mute.php [^]
    ForwardOfflineGroupMessages = true


From my ROBUST HG config:

[ServiceList]
;; Uncomment this if you want offline IM to work
OfflineIMServiceConnector = "8013/OpenSim.Addons.OfflineIM.dll:OfflineIMServiceRobustConnector"

[Messaging]
    ; OfflineIM
    OfflineIMService = "OpenSim.Addons.OfflineIM.dll:OfflineIMService"


From my database:

mysql> describe im_offline;
+-------------+--------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+-------------------+-----------------------------+
| ID | mediumint(9) | NO | PRI | NULL | auto_increment |
| PrincipalID | char(36) | NO | MUL | | |
| FromID | char(36) | NO | MUL | | |
| Message | text | NO | | NULL | |
| TMStamp | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+-------------+--------------+------+-----+-------------------+-----------------------------+
5 rows in set (0.00 sec)

mysql>
(0025812)
smxy (reporter)
2014-04-20 11:00

I just realized I'm seeing a related error on the ROBUST console, too, that suggests maybe it's a mono bug?

13:40:27 - [OFFLINE IM HANDLER]: Exception Text node cannot appear in this state. Line 1, position 1. System.Xml.XmlException: Text node cannot appear in this state. Line 1, position 1.
  at Mono.Xml2.XmlTextReader.ReadText (Boolean notWhitespace) [0x00180] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1643
  at Mono.Xml2.XmlTextReader.ReadContent () [0x0015a] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1295
  at Mono.Xml2.XmlTextReader.Read () [0x00141] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader.cs:573
  at System.Xml.XmlTextReader.Read () [0x0006b] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader2.cs:566
  at System.Xml.XmlReader.MoveToContent () [0x00050] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlReader.cs:602
  at System.Xml.Serialization.XmlSerializationReaderInterpreter.ReadRoot () [0x00000] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml.Serialization/XmlSerializationReaderInterpreter.cs:83
  at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.Serialization.XmlSerializationReader reader) [0x0001c] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml.Serialization/XmlSerializer.cs:359
(0025813)
smxy (reporter)
2014-04-20 11:29

Building with Mono 3.2.3 didn't resolve this. Nor did building *and* running with 3.2.3.
(0025814)
smxy (reporter)
2014-04-20 12:24
edited on: 2014-04-20 12:35

Ditto for 3.1.0 and 2.10.8. Maybe mono doesn't like the stored messages themselves? Here's one:

| 186 | 5adfe612-116c-4146-8f0e-ae839e018615 | 6d989e4b-53ed-4217-98a8-4626a6d5cbda | ?<?xml version="1.0" encoding="utf-8"?><GridInstantMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [^] xmlns:xsd="http://www.w3.org/2001/XMLSchema"><fromAgentID>6d989e4b-53ed-4217-98a8-4626a6d5cbda</fromAgentID><fromAgentName>Shaun [^] Emerald</fromAgentName><toAgentID>5adfe612-116c-4146-8f0e-ae839e018615</toAgentID><dialog>0</dialog><fromGroup>false</fromGroup><message>This is a test to make sure that offline messaging works.</message><imSessionID>37477859-4281-0351-17a6-e8a538d44dcf</imSessionID><offline>0</offline><Position><X>0</X><Y>0</Y><Z>0</Z></Position><binaryBucket>AA==</binaryBucket><ParentEstateID>1</ParentEstateID><RegionID>9d0932b0-ba2b-11df-851a-0800200c9a66</RegionID><timestamp>1397934071</timestamp></GridInstantMessage> | 2014-04-19 15:01:11 |

I doubt this is the issue though, as it seems to fail much lower in the code, before it even gets to reading the messages.

(0025815)
orenh (administrator)
2014-04-20 13:22

The IM message in your database is invalid. It starts with "?<?xml..." -- but the first question mark shouldn't be there, it makes the XML invalid. How on earth did it get there? This is the root cause of your problem.

A secondary problem is that the code that parses the results in the simulator doesn't handle error cases well. I can improve it, but that will just show a nicer error message; it still won't make the offline IM's work. You have to delete the invalid messages from the database in order to do that. And you should probably find out how they got in there.
(0025816)
smxy (reporter)
2014-04-20 13:33
edited on: 2014-04-20 13:34

I emptied the the table just now, then logged in and sent a message to an offline avatar. The entry in the DB looks like all the others did:

| 187 | 5adfe612-116c-4146-8f0e-ae839e018615 | 6d989e4b-53ed-4217-98a8-4626a6d5cbda | ?<?xml version="1.0" encoding="utf-8"?><GridInstantMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [^] xmlns:xsd="http://www.w3.org/2001/XMLSchema"><fromAgentID>6d989e4b-53ed-4217-98a8-4626a6d5cbda</fromAgentID><fromAgentName>Shaun [^] Emerald</fromAgentName><toAgentID>5adfe612-116c-4146-8f0e-ae839e018615</toAgentID><dialog>0</dialog><fromGroup>false</fromGroup><message>This is a test.</message><imSessionID>37477859-4281-0351-17a6-e8a538d44dcf</imSessionID><offline>0</offline><Position><X>0</X><Y>0</Y><Z>0</Z></Position><binaryBucket>AA==</binaryBucket><ParentEstateID>1</ParentEstateID><RegionID>39d655f0-c4f0-11e3-9c1a-0800200c9a66</RegionID><timestamp>1398025896</timestamp></GridInstantMessage> | 2014-04-20 16:31:36 |

(0025817)
smxy (reporter)
2014-04-20 13:38

It's independent of viewer, as messages sent when using Firestorm and Singularity look the same in the DB.
(0025820)
smxy (reporter)
2014-04-20 17:44

I uncommented line 49 of OfflineIMService.cs and then added 2 log messages:

        public bool StoreMessage(GridInstantMessage im, out string reason)
        {
            reason = string.Empty;

            // Check limits
            UUID principalID = new UUID(im.toAgentID);
            long count = m_Database.GetCount("PrincipalID", principalID.ToString());
            if (count >= MAX_IM)
            {
                reason = "Number of offline IMs has maxed out";
                return false;
            }

m_log.DebugFormat("[OfflineIM.V2] THE IM BEFORE: {0}", im.message);
            string imXml;
            using (MemoryStream mstream = new MemoryStream())
            {
                XmlWriterSettings settings = new XmlWriterSettings();
                settings.Encoding = Encoding.UTF8;

                using (XmlWriter writer = XmlWriter.Create(mstream, settings))
                {
                    m_serializer.Serialize(writer, im);
                    writer.Flush();
                }

                imXml = Util.UTF8.GetString(mstream.ToArray());
            }
m_log.DebugFormat("[OfflineIM.V2] THE IM AFTER: {0}", imXml);

            OfflineIMData data = new OfflineIMData();
            data.PrincipalID = principalID;
            data.FromID = new UUID(im.fromAgentID);
            data.Data = new Dictionary<string, string>();
            data.Data["Message"] = imXml;

            return m_Database.Store(data);

        }

I then sent an IM to an offline user, and this was logged at the ROBUST console:

20:39:24 - [OfflineIM.V2]: THE IM BEFORE: This is being sent to an offline user.
20:39:24 - [OfflineIM.V2]: THE IM AFTER: <?xml version="1.0" encoding="utf-8"?><GridInstantMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [^] xmlns:xsd="http://www.w3.org/2001/XMLSchema"><fromAgentID>5adfe612-116c-4146-8f0e-ae839e018615</fromAgentID><fromAgentName>Cabana [^] Boy</fromAgentName><toAgentID>10011659-7dd7-4479-a2f6-59d2d40562bc</toAgentID><dialog>0</dialog><fromGroup>false</fromGroup><message>This is being sent to an offline user.</message><imSessionID>4adef04b-6cbb-053f-2df8-f7514a04e4a9</imSessionID><offline>0</offline><Position><X>0</X><Y>0</Y><Z>0</Z></Position><binaryBucket>AA==</binaryBucket><ParentEstateID>1</ParentEstateID><RegionID>1539ea30-bdee-11e3-b1b6-0800200c9a66</RegionID><timestamp>1398040764</timestamp></GridInstantMessage>

Does this tell us anything? If not, where might I go from here?
(0025822)
smxy (reporter)
2014-04-20 18:17

This time, logging everything before and after serializing:

                using (XmlWriter writer = XmlWriter.Create(mstream, settings))
                {
m_log.DebugFormat("[OfflineIM.V2] im.fromAgentID BEFORE: {0}", im.fromAgentID);
m_log.DebugFormat("[OfflineIM.V2] im.toAgentID BEFORE: {0}", im.fromAgentName);
m_log.DebugFormat("[OfflineIM.V2] im.toAgentID BEFORE: {0}", im.toAgentID);
m_log.DebugFormat("[OfflineIM.V2] im.dialog BEFORE: {0}", im.dialog);
m_log.DebugFormat("[OfflineIM.V2] im.fromGroup BEFORE: {0}", im.fromGroup);
m_log.DebugFormat("[OfflineIM.V2] im.message BEFORE: {0}", im.message);
m_log.DebugFormat("[OfflineIM.V2] im.imSessionID BEFORE: {0}", im.imSessionID);
m_log.DebugFormat("[OfflineIM.V2] im.offline BEFORE: {0}", im.offline);
m_log.DebugFormat("[OfflineIM.V2] im.Position BEFORE: {0}", im.Position);
m_log.DebugFormat("[OfflineIM.V2] im.binaryBucket BEFORE: {0}", im.binaryBucket);
m_log.DebugFormat("[OfflineIM.V2] im.RegionID BEFORE: {0}", im.RegionID);
m_log.DebugFormat("[OfflineIM.V2] im.ParentEstateID BEFORE: {0}", im.ParentEstateID);
m_log.DebugFormat("[OfflineIM.V2] im.timestamp BEFORE: {0}", im.timestamp);
                    m_serializer.Serialize(writer, im);
                    writer.Flush();
                }

                imXml = Util.UTF8.GetString(mstream.ToArray());
m_log.DebugFormat("[OfflineIM.V2] THE XML AFTER: {0}", imXml);

That gets me:

21:14:43 - [OfflineIM.V2]: im.fromAgentID BEFORE: 5adfe612-116c-4146-8f0e-ae839e018615
21:14:43 - [OfflineIM.V2]: im.toAgentID BEFORE: Cabana Boy
21:14:43 - [OfflineIM.V2]: im.toAgentID BEFORE: 10011659-7dd7-4479-a2f6-59d2d40562bc
21:14:43 - [OfflineIM.V2]: im.dialog BEFORE: 0
21:14:43 - [OfflineIM.V2]: im.fromGroup BEFORE: False
21:14:43 - [OfflineIM.V2]: im.message BEFORE: Yet another test IM.
21:14:43 - [OfflineIM.V2]: im.imSessionID BEFORE: 4adef04b-6cbb-053f-2df8-f7514a04e4a9
21:14:43 - [OfflineIM.V2]: im.offline BEFORE: 0
21:14:43 - [OfflineIM.V2]: im.Position BEFORE: <0, 0, 0>
21:14:43 - [OfflineIM.V2]: im.binaryBucket BEFORE: System.Byte[]
21:14:43 - [OfflineIM.V2]: im.RegionID BEFORE: 1539ea30-bdee-11e3-b1b6-0800200c9a66
21:14:43 - [OfflineIM.V2]: im.ParentEstateID BEFORE: 1
21:14:43 - [OfflineIM.V2]: im.timestamp BEFORE: 1398042883
21:14:43 - [OfflineIM.V2]: THE XML AFTER: <?xml version="1.0" encoding="utf-8"?><GridInstantMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [^] xmlns:xsd="http://www.w3.org/2001/XMLSchema"><fromAgentID>5adfe612-116c-4146-8f0e-ae839e018615</fromAgentID><fromAgentName>Cabana [^] Boy</fromAgentName><toAgentID>10011659-7dd7-4479-a2f6-59d2d40562bc</toAgentID><dialog>0</dialog><fromGroup>false</fromGroup><message>Yet another test IM.</message><imSessionID>4adef04b-6cbb-053f-2df8-f7514a04e4a9</imSessionID><offline>0</offline><Position><X>0</X><Y>0</Y><Z>0</Z></Position><binaryBucket>AA==</binaryBucket><ParentEstateID>1</ParentEstateID><RegionID>1539ea30-bdee-11e3-b1b6-0800200c9a66</RegionID><timestamp>1398042883</timestamp></GridInstantMessage>
(0025823)
smxy (reporter)
2014-04-20 19:11

If I put in a bloody awful hack to trim the first character off, offline messaging works:

                imXml = Util.UTF8.GetString(mstream.ToArray());
m_log.DebugFormat("[OfflineIM.V2] THE XML AFTER: {0}", imXml);
imXml = imXml.Remove(0,1);
m_log.DebugFormat("[OfflineIM.V2] THE XML AFTER TRIM HACK: {0}", imXml);

22:08:20 - [OfflineIM.V2]: THE XML AFTER: <?xml version="1.0" encoding="utf-8"?><GridInstantMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [^] xmlns:xsd="http://www.w3.org/2001/XMLSchema"><fromAgentID>6d989e4b-53ed-4217-98a8-4626a6d5cbda</fromAgentID><fromAgentName>Shaun [^] Emerald</fromAgentName><toAgentID>10011659-7dd7-4479-a2f6-59d2d40562bc</toAgentID><dialog>0</dialog><fromGroup>false</fromGroup><message>Another test.</message><imSessionID>7d998812-2e3a-066e-3a5e-1ff472d0a966</imSessionID><offline>0</offline><Position><X>0</X><Y>0</Y><Z>0</Z></Position><binaryBucket>AA==</binaryBucket><ParentEstateID>1</ParentEstateID><RegionID>beabee30-ba21-11df-851a-0800200c9a66</RegionID><timestamp>1398046100</timestamp></GridInstantMessage>
22:08:20 - [OfflineIM.V2]: THE XML AFTER TRIM HACK: <?xml version="1.0" encoding="utf-8"?><GridInstantMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" [^] xmlns:xsd="http://www.w3.org/2001/XMLSchema"><fromAgentID>6d989e4b-53ed-4217-98a8-4626a6d5cbda</fromAgentID><fromAgentName>Shaun [^] Emerald</fromAgentName><toAgentID>10011659-7dd7-4479-a2f6-59d2d40562bc</toAgentID><dialog>0</dialog><fromGroup>false</fromGroup><message>Another test.</message><imSessionID>7d998812-2e3a-066e-3a5e-1ff472d0a966</imSessionID><offline>0</offline><Position><X>0</X><Y>0</Y><Z>0</Z></Position><binaryBucket>AA==</binaryBucket><ParentEstateID>1</ParentEstateID><RegionID>beabee30-ba21-11df-851a-0800200c9a66</RegionID><timestamp>1398046100</timestamp></GridInstantMessage>
(0025824)
kenvc (reporter)
2014-04-20 19:31
edited on: 2014-04-20 19:33

Just confirmed that I am seeing the same thing and offline V2 IMs are no longer working for me too. This is with latest dev master.

(0025826)
orenh (administrator)
2014-04-20 21:46

The characters "" are called a BOM, or Byte Order Mark. They're a flag that indicates that what follows is a Unicode string encoded using UTF8. They shouldn't cause a problem, because they're removed when the data is read. But apparently they ARE a problem for you.

I just confirmed that on my system the BOM also exists, and is correctly removed when reading the string. So why doesn't that work for you?

Move your logging to OfflineIMService.GetMessages(). Let's see what that data looks like.
(0025828)
smxy (reporter)
2014-04-21 04:34

Moving the logging, as suggested, doesn't appear useful, as message retrieval fails while still in Mono's serialization routines:

07:29:01 - [OFFLINE IM HANDLER]: Exception Text node cannot appear in this state. Line 1, position 1. System.Xml.XmlException: Text node cannot appear in this state. Line 1, position 1.
  at Mono.Xml2.XmlTextReader.ReadText (Boolean notWhitespace) [0x00180] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1643
  at Mono.Xml2.XmlTextReader.ReadContent () [0x0015a] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1295
  at Mono.Xml2.XmlTextReader.Read () [0x00141] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader.cs:573
  at System.Xml.XmlTextReader.Read () [0x0006b] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlTextReader2.cs:566
  at System.Xml.XmlReader.MoveToContent () [0x00050] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml/XmlReader.cs:602
  at System.Xml.Serialization.XmlSerializationReaderInterpreter.ReadRoot () [0x00000] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml.Serialization/XmlSerializationReaderInterpreter.cs:83
  at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.Serialization.XmlSerializationReader reader) [0x0001c] in /root/install/mono-3.2.8/mono-3.2.8/mcs/class/System.XML/System.Xml.Serialization/XmlSerializer.cs:359

I wonder ... if the BOM is getting added and removed correctly, on your system, does it also appear as a question mark when viewed in the DB? Perhaps the issue is that it's being mangled on the way in or out of my DB, in a way that Mono can't handle it when read? Maybe there is a DB locale setting I am missing or have wrong?
(0025829)
smxy (reporter)
2014-04-21 04:35

Or an Operating System locale setting, perhaps.
(0025830)
orenh (administrator)
2014-04-21 04:38

That's a good idea. You need to use UTF8.

I started to explain what to do, but on second thought maybe I should change OpenSim not to use Unicode there.
(0025831)
orenh (administrator)
2014-04-21 04:47

What the hell, let's check that this is the issue first.

1) Please run in MySQL (in your Robust database):

show create table im_offline\G

It should show something like this:

Create Table: CREATE TABLE `im_offline` (
...
) ENGINE=InnoDB AUTO_INCREMENT=32 DEFAULT CHARSET=utf8

What is the CHARSET? There might not be any.


2) If the charset isn't UTF8, then run this:

ALTER TABLE im_offline CONVERT TO CHARACTER SET utf8;


3) Try Offline IM's again
(0025832)
smxy (reporter)
2014-04-21 05:07

It was:

) ENGINE=MyISAM AUTO_INCREMENT=199 DEFAULT CHARSET=latin1

So I changed it to:

) ENGINE=MyISAM AUTO_INCREMENT=199 DEFAULT CHARSET=utf8

I then deleted all messages, sent a new one, and the result is still the same.

I've been looking through the Mono code mentioned in the exception, and so far, I'm not finding anything that suggests it knows what to do with this first character ...
(0025833)
smxy (reporter)
2014-04-21 05:14

My CentOS 6.5 system's LANG variable is set to: en_US.UTF-8
I don't see a locale setting in /etc/my.cnf
(0025834)
orenh (administrator)
2014-04-21 05:15

my.cnf should include:

[mysqld]
character-set-server=utf8

[client]
default-character-set=utf8


And then restart mysqld
(0025835)
smxy (reporter)
2014-04-21 05:30

And *THAT* was the final piece needed. Now, when the BOM is stored in the DB, it looks like that odd three-but-really-one character, instead of a question mark, and the message is correctly retrieved by the avatar when they login.

Thank you for your help. :)
(0025836)
orenh (administrator)
2014-04-21 05:32

You're welcome. However, the change to the mysql configuration is too complicated to expect everyone to do. So I think I'll change OpenSim to NOT use Unicode in this table.
(0025837)
smxy (reporter)
2014-04-21 05:37

I'm not sure I agree it's too complicated. It's a couple of lines to add to my.cnf and the ALTER TABLE could be handled with a migration, yes?
(0025838)
orenh (administrator)
2014-04-21 05:44

The ALTER TABLE can be automatic, but the MySQL changes would have to be done manually. There's strong resistance to making any change that might break existing installations. So try and mention this in IRC, and see what Melanie says :)

I suppose it could be argued that it's *already* broken, but there's a code fix that would un-break it (at the cost of giving up Unicode characters).
(0025841)
Dev Random (reporter)
2014-04-21 07:54

Could this be addressed by changing the DB connection string instead of requiring a change to my.cnf? That would keep all changes within OpenSim.
(0025845)
kenvc (reporter)
2014-04-22 10:50
edited on: 2014-04-22 11:06

I changed my.ini to use utf8 as suggested above and everything is configured in the opensim.ini and robust.ini the same way smxy is configured, but seeing this error message in robust.log and offline IMs are not being saved in the DB:

2014-04-22 12:05:00,911 ERROR - OpenSim.Data.MySQL.MySqlFramework Incorrect string value: '\xEF\xBB\xBF<?x...' for column 'Message' at row 1
MySql.Data.MySqlClient.MySqlException (0x80004005): Incorrect string value: '\xEF\xBB\xBF<?x...' for column 'Message' at row 1
   at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
   at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int32& insertedId)
   at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId)
   at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
   at OpenSim.Data.MySQL.MySqlFramework.ExecuteNonQuery(MySqlCommand cmd) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Data\MySQL\MySQLFramework.cs:line 71

(0025847)
kenvc (reporter)
2014-04-22 17:29

In order to make V2 offline messaging work, I had to also alter the MySQL offline message table in the Robust database to use utf8 with the following command:
ALTER TABLE im_offline CONVERT TO CHARACTER SET utf8;

Thanks smxy for supplying the remaining step to make V2 messaging work. Unfortunately, I don't think the average user will be willing to do all these manual steps to make it work. Hopefully something can be done in code to do most if not all of this conversion automatically.
(0025849)
orenh (administrator)
2014-04-23 06:15

I have changed the table im_offline to use the UTF8 character set. As usual, the table will be migrated automatically the first time Robust is started after the update.

Users have to change their MySQL configuration on their own, because this can't be done by OpenSim. In the config file /etc/my.cnf (Linux) or my.ini (Windows), add these settings:

   [mysqld]
   character-set-server=utf8

   [client]
   default-character-set=utf8

And then restart MySQL (on Linux: "sudo service mysqld restart").


In IRC, there was talk of changing some other tables to UTF8 as well. This was driven by the assumption that currently all of the tables use the Latin1 character set. But after checking I have found that many of the tables in OpenSim/Robust *already* use UTF8: assets, inventoryitems, etc. So there's no need to change any more tables just for testing purposes. At some point we might want to go through and change the remaining tables that don't use UTF8 to use UTF8, but it's not a lot of tables, and this is a low priority.

BTW, OSGrid was checked and its 'inventoryitems' table uses the ASCII character set, not UTF8. That was the only table that was checked, but doubtless this is the case in other tables as well. This must mean that at some time in the past a database migration was skipped. OSGrid, and any other grids that skipped updates on purpose, should update their tables to UTF8 at their own convenience. If they don't then there probably isn't much harm; they're no worse off than they are now. Probably the only downside of not migrating is that any attempt to insert non-Latin1 characters into the database will result in garbled characters.
(0025852)
orenh (administrator)
2014-04-23 07:47

Changed the table to UTF8 in Git master 2c98593
(0025882)
smxy (reporter)
2014-04-24 06:05

Offline IMs are working fine for me now. Thanks!

- Issue History
Date Modified Username Field Change
2014-04-19 13:00 smxy New Issue
2014-04-20 09:53 smxy Note Added: 0025810
2014-04-20 10:23 smxy Note Added: 0025811
2014-04-20 11:00 smxy Note Added: 0025812
2014-04-20 11:29 smxy Note Added: 0025813
2014-04-20 12:24 smxy Note Added: 0025814
2014-04-20 12:35 smxy Note Edited: 0025814 View Revisions
2014-04-20 13:22 orenh Note Added: 0025815
2014-04-20 13:33 smxy Note Added: 0025816
2014-04-20 13:34 smxy Note Edited: 0025816 View Revisions
2014-04-20 13:38 smxy Note Added: 0025817
2014-04-20 17:44 smxy Note Added: 0025820
2014-04-20 18:17 smxy Note Added: 0025822
2014-04-20 19:11 smxy Note Added: 0025823
2014-04-20 19:31 kenvc Note Added: 0025824
2014-04-20 19:33 kenvc Note Edited: 0025824 View Revisions
2014-04-20 21:46 orenh Note Added: 0025826
2014-04-21 04:34 smxy Note Added: 0025828
2014-04-21 04:35 smxy Note Added: 0025829
2014-04-21 04:38 orenh Note Added: 0025830
2014-04-21 04:47 orenh Note Added: 0025831
2014-04-21 04:47 orenh Assigned To => orenh
2014-04-21 04:47 orenh Status new => assigned
2014-04-21 05:07 smxy Note Added: 0025832
2014-04-21 05:14 smxy Note Added: 0025833
2014-04-21 05:15 orenh Note Added: 0025834
2014-04-21 05:30 smxy Note Added: 0025835
2014-04-21 05:32 orenh Note Added: 0025836
2014-04-21 05:37 smxy Note Added: 0025837
2014-04-21 05:44 orenh Note Added: 0025838
2014-04-21 07:54 Dev Random Note Added: 0025841
2014-04-22 10:50 kenvc Note Added: 0025845
2014-04-22 11:06 kenvc Note Edited: 0025845 View Revisions
2014-04-22 17:29 kenvc Note Added: 0025847
2014-04-23 06:15 orenh Note Added: 0025849
2014-04-23 07:47 orenh Note Added: 0025852
2014-04-23 07:47 orenh Status assigned => resolved
2014-04-23 07:47 orenh Resolution open => fixed
2014-04-24 06:05 smxy Note Added: 0025882
2014-04-24 06:05 smxy Status resolved => closed
2014-04-24 06:05 smxy Fixed in Version => master (dev code)
2014-05-21 19:11 AliciaRaven Relationship added related to 0006973


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker