[Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23 shutdown problems

Thomas Ringate tringate at gmail.com
Sat Nov 28 00:15:15 UTC 2015 

I finished the commit search and discovered that because of the naming issue with the commits the code appears to work, then not work, then work but actually the commits are intermixed but not from the same code branch.  I had to go back into the August time frame when the merge started and prior to the merge everything is fine.  Then, until they made it release 0.9.0,  there is this mixture of commits that work and don’t.  When the merge began, the commits would not compile error free for me so I didn’t bother trying to test those.   On the first commit with errors I encountered I could not start the compiled instance at all.  It was not until early September in the commits I found the first error free compile.  It had the shutdown problem.  All of those I tested after that date failed unless they were from the other code branch that was intermixed with the faulty commit names.  That held true up to the last of those kind of commits on 2015-10-23.

These two commits were made of the same fix, one to one code branch, and the other to the opensim-0.6.9.rc1 code branch.

2015-10-21 23:47 Diva Canto Fix an issue introduced in 70a46fe0907c822a5244e36 SHUTDOWN FAILS - opensim-0.6.9.rc1-15867-g25552dd.zip
2015-10-21 23:47 Diva Canto Fix an issue introduced in 70a46fe0907c822a5244e36 SHUTS DOWN NORMALLY – opensim-41b2855.zip

All commits since then fail with the shutdown issue.  That is because this is the last commit made to the V0.8.3 branch and everything shifted to the opensim-0.6.9.rc1 branch.

All of this has been added as a note to the mantis.
     


From: Shaun T. Erickson 
Sent: Thursday, November 26, 2015 11:37 PM
To: opensim-users at opensimulator.org 
Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23 shutdown problems

If you use bit bisect, which does what you plan, but even better, you'll likely find the bad commit sooner.

On Thursday, November 26, 2015, Thomas Ringate <tringate at gmail.com> wrote:

  Tomorrow I will attempt to see if I can find when the problem first appeared.  I will start trying each git commit by dividing them in half from when it was not failing and when it was failing.  If I can locate exactly when this first appeared for me, I will report it in  the mantis.

  From: AJLDuarte 
  Sent: Thursday, November 26, 2015 7:14 PM
  To: javascript:_e(%7B%7D,'cvml','opensim-users at opensimulator.org'); 
  Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev)5e4b166: 2015-11-23shutdown problems

  Hi

                  This is a open issue. The cause of problem is still not identified.

  It seems to happen on some regions or configurations, not others.

  Some persons noted that changing mono version apparently fixed it. Some also notice it in windows.

                  I tested several configurations in windows and all worked as expected,

  so still looking for it…

                  All information like you are providing is fundamental help.

                  We can only report issues to mono team when we do know it’s a mono issue.                   

  Regards





  From: javascript:_e(%7B%7D,'cvml','opensim-users-bounces at opensimulator.org'); [mailto:javascript:_e(%7B%7D,'cvml','opensim-users-bounces at opensimulator.org');] On Behalf Of Thomas Ringate
  Sent: Thursday, November 26, 2015 21:10
  To: javascript:_e(%7B%7D,'cvml','opensim-users at opensimulator.org');
  Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23shutdown problems



  Hi,



  I see there is no way to actually reopen a bug in mantis, but I did post some additional information to thes bug in mantis.



  Here is the latest thing I posted.



  More information on this bug.



  To say windows with .net is working correctly is also false.  It is indeed throwing an ERROR  at shutdown and could well be what is causing mono to hang.



  Here is a shutdown in windows using .net for the second to the last version of opensim for OSgrid, and the last version of opensim released for OSgrid.



  You can clearly see there are no ERRORS in shutdown previously, and there is after the current release.



  There is something wrong that was not wrong before in mono and in .net.  The difference is that .NET still closes, and mono does not.



  OSgrid 0.8.3.0 (Dev) 41b2855: 2015-10-21



  2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads

  2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread

  2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients

  2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating

  2015-11-26 15:46:32,310 INFO  (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Development

  2015-11-26 15:46:32,357 DEBUG (Heartbeat-(Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Development), ID 67

  2015-11-26 15:46:32,420 DEBUG (Maintenance (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Development), ID 72

  2015-11-26 15:46:32,826 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects

  2015-11-26 15:46:37,029 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Development (570c1600-3516-11e4-8c21-0800200c9a66) at 10054-9986

  2015-11-26 15:46:37,295 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Development

  2015-11-26 15:46:37,310 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_search.php?cond_channame=570c1600-3516-11e4-8c21-0800200c9a66D&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>

  2015-11-26 15:46:37,466 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Development: uuid 570c1600-3516-11e4-8c21-0800200c9a66: located directory id 4443169

  2015-11-26 15:46:37,482 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_search.php?&cond_chanparent=4443169&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>

  2015-11-26 15:46:37,576 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_mod.php?mode=delete&chan_id=4443169&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>

  2015-11-26 15:46:37,717 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Development is being removed, removing from indexing

  2015-11-26 15:46:37,748 INFO  (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 492 scripts in Development

  2015-11-26 15:46:37,842 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 58

  2015-11-26 15:46:37,857 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 59

  2015-11-26 15:46:37,857 INFO  (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Development

  2015-11-26 15:46:37,857 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop

  2015-11-26 15:46:37,857 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop

  2015-11-26 15:46:37,904 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_signout.php?auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>

  2015-11-26 15:46:37,967 DEBUG (Outgoing Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Development), ID 62

  2015-11-26 15:46:37,967 DEBUG (Incoming Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Development), ID 61

  2015-11-26 15:46:37,982 DEBUG (bulletunmanaged (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Development), ID 57

  2015-11-26 15:46:37,998 INFO  (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector

  2015-11-26 15:46:37,998 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector

  2015-11-26 15:46:38,092 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php notified. Secret: 24383411-9692-40df-b8f1-e22e2bb9aa45

  2015-11-26 15:46:38,107 INFO  (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete.  Exiting...





  OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23



  2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads

  2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread

  2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients

  2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating

  2015-11-26 15:49:02,654 INFO  (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Development

  2015-11-26 15:49:03,170 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: TriggerSceneShuttingDown

  2015-11-26 15:49:03,170 INFO  (1) - OpenSim.Region.Framework.Scenes.EventManager [EVENT MANAGER]: TriggerSceneShuttingDown done

  2015-11-26 15:49:03,170 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects

  2015-11-26 15:49:03,186 DEBUG (Maintenance (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Development), ID 77

  2015-11-26 15:49:03,217 DEBUG (36) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Development), ID 72

  2015-11-26 15:49:03,217 ERROR (36) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Development)". ThreadState=Stopped. Last tick was 625ms ago.  

  2015-11-26 15:49:03,295 DEBUG (TouchAllSceneAssets) - OpenSim.Region.CoreModules.Asset.FlotsamAssetCache [FLOTSAM ASSET CACHE]: Could not find asset ce5301d9-0943-4ffa-891e-f8305b748897, type 0 referenced by object @-Heather Genie Dressing Booth at <48.14737, 75.16174, 22.52853> in scene Development when pre-caching all scene assets

  2015-11-26 15:49:23,795 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Development (570c1600-3516-11e4-8c21-0800200c9a66) at 10054-9986

  2015-11-26 15:49:24,061 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Development

  2015-11-26 15:49:24,280 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Development: uuid 570c1600-3516-11e4-8c21-0800200c9a66: located directory id 4443177

  2015-11-26 15:49:24,686 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Development is being removed, removing from indexing

  2015-11-26 15:49:24,733 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 64

  2015-11-26 15:49:24,749 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 65

  2015-11-26 15:49:24,749 INFO  (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 492 scripts in Development

  2015-11-26 15:49:24,827 INFO  (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Development

  2015-11-26 15:49:24,827 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop

  2015-11-26 15:49:24,827 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop

  2015-11-26 15:49:24,842 DEBUG (MapItemRequestThread (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread MapItemRequestThread (Development), ID 58

  2015-11-26 15:49:24,858 DEBUG (Outgoing Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Development), ID 68

  2015-11-26 15:49:24,858 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Dispose Physics

  2015-11-26 15:49:24,889 DEBUG (Incoming Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Development), ID 67

  2015-11-26 15:49:24,952 DEBUG (bulletunmanaged (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Development), ID 57

  2015-11-26 15:49:25,014 INFO  (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector

  2015-11-26 15:49:25,014 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector

  2015-11-26 15:49:25,093 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php notified. Secret: d73bfc15-b3db-46b6-800c-d5d320185478

  2015-11-26 15:49:25,108 INFO  (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete.  Exiting...



  As  I said, I have no proof this is the mono problem, but there is something wrong and it sure is suspicious that it might well be what is causing mono to hang and not close.



  From: AJLDuarte 

  Sent: Wednesday, November 25, 2015 11:11 PM

  To: javascript:_e(%7B%7D,'cvml','opensim-users at opensimulator.org'); 

  Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23shutdown problems



  Hi

                  Similar issue was already reported, and found to be related to mono

                  Check http://opensimulator.org/mantis/view.php?id=7743

  regards



  From: javascript:_e(%7B%7D,'cvml','opensim-users-bounces at opensimulator.org'); [javascript:_e(%7B%7D,'cvml','opensim-users-bounces at opensimulator.org');] On Behalf Of Han Held
  Sent: Thursday, November 26, 2015 03:59
  To: javascript:_e(%7B%7D,'cvml','opensim-users at opensimulator.org');
  Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23 shutdown problems



  More likely than not, it's a mono problem. I've encountered that sporadically over the years. What is your os (vendor, version, 32 bit or 64 bit) and what's the output of mono --version ?

  On 11/25/15 1:39 PM, Thomas Ringate wrote:

    Anyone else having a problem with shutting down a simulator since upgrading to release OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23?



    I can only shut down the simulator by using a Linux “kill” command.  The opensim “quit” command no longer works.  It goes to the point you get the message “Exiting...” and then hangs forever.



    The simulators appear to be running fine, but kind of a nuisance not to be able to use normal commands to shut them down.



    They are running in a “screen” terminal window.



    Here is the log of a shutdown immediately after launching a region.



    2015-11-25 17:18:29,952 INFO  (1) - OpenSim.Framework.Servers.ServerBase [SERVER BASE]: Running shutdown_commands.txt

    2015-11-25 17:18:29,978 INFO  (1) - OpenSim.Framework.Servers.ServerBase [SERVER BASE]: Running 'backup'

    2015-11-25 17:18:33,925 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads

    2015-11-25 17:18:33,926 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread

    2015-11-25 17:18:33,926 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients

    2015-11-25 17:18:33,926 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating

    2015-11-25 17:18:33,927 INFO  (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Sanctuary

    2015-11-25 17:18:34,148 DEBUG (Maintenance (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Sanctuary), ID 79

    2015-11-25 17:18:34,428 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: TriggerSceneShuttingDown

    2015-11-25 17:18:34,428 INFO  (1) - OpenSim.Region.Framework.Scenes.EventManager [EVENT MANAGER]: TriggerSceneShuttingDown done

    2015-11-25 17:18:34,429 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects

    2015-11-25 17:18:34,588 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Sanctuary (3b15c050-1fe4-11df-8a39-0800200c9a66) at 10054-9986

    2015-11-25 17:18:34,909 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Sanctuary

    2015-11-25 17:18:35,678 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Sanctuary: uuid 3b15c050-1fe4-11df-8a39-0800200c9a66: located directory id 4439471

    2015-11-25 17:18:35,773 DEBUG (Threadpool worker) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Sanctuary), ID 75

    2015-11-25 17:18:35,774 ERROR (Threadpool worker) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Sanctuary)". ThreadState=Stopped. Last tick was 1916ms ago.  

    2015-11-25 17:18:36,025 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Sanctuary is being removed, removing from indexing

    2015-11-25 17:18:36,031 INFO  (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 487 scripts in Sanctuary

    2015-11-25 17:18:36,208 INFO  (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Sanctuary

    2015-11-25 17:18:36,208 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop

    2015-11-25 17:18:36,209 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop

    2015-11-25 17:18:36,210 DEBUG (Outgoing Packets (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Sanctuary), ID 59

    2015-11-25 17:18:36,223 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 66

    2015-11-25 17:18:36,223 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 67

    2015-11-25 17:18:36,231 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Dispose Physics

    2015-11-25 17:18:36,234 DEBUG (bulletunmanaged (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Sanctuary), ID 57

    2015-11-25 17:18:36,246 DEBUG (Incoming Packets (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Sanctuary), ID 58

    2015-11-25 17:18:36,391 INFO  (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector

    2015-11-25 17:18:36,392 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector

    2015-11-25 17:18:36,798 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php notified. Secret: 38384e09-f875-4b31-a5a0-dccc4566ac1d

    2015-11-25 17:18:36,801 INFO  (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete.  Exiting...




----------------------------------------------------------------------------


         This email has been checked for viruses by Avast antivirus software. 
          www.avast.com 
         






_______________________________________________Opensim-users mailing listjavascript:_e(%7B%7D,'cvml','Opensim-users at opensimulator.org');http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


------------------------------------------------------------------------------

  _______________________________________________
  Opensim-users mailing list
  javascript:_e(%7B%7D,'cvml','Opensim-users at opensimulator.org');
  http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users




------------------------------------------------------------------------------


       This email has been checked for viruses by Avast antivirus software. 
        www.avast.com 
       




------------------------------------------------------------------------------
  _______________________________________________
  Opensim-users mailing list
  javascript:_e(%7B%7D,'cvml','Opensim-users at opensimulator.org');
  http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users




------------------------------------------------------------------------------
          This email has been checked for viruses by Avast antivirus software. 
        www.avast.com 
       




-- 
        -ste



--------------------------------------------------------------------------------
_______________________________________________
Opensim-users mailing list
Opensim-users at opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-users/attachments/20151127/5812ee04/attachment-0001.html>

More information about the Opensim-users mailing list