[Opensim-dev] Any Ideas??

Sat Nov 29 02:14:46 UTC 2014

Hi Butch.  You actually linked to the 'ok' pic twice below.  It's good to see the actual output as that can sometimes 
give extra clues to someone looking at the code as to what's going wrong.

However, on reviewing the code, I believe it is possible for an exception in the script compilation process to 
permanently stop all future compilation until the simulator is restarted.  I also believe any such exception would be 
silently dropped and not logged.

So to help identify the issue, I have both logged any exception and changed the code to increase the chance that future 
script compilation will still happen even if something does go wrong.  This is in commit 690fe0c in master.

If you see an exception logged with this code then please report.  If the problem still occurs with no extra log clues, 
then my next hypothesis would be that compilation is hanging indefinitely, possibly due to some Mono bug.  In this case, 
the ideal thing to do would be to take a thread dump so we can see if a thread is stuck.  On Mono I have just documented 
the process at [1].  However, I suspect you're on Windows and I'm not sure how one would do it there (or even if that 
kind of thing is possible).

If we find the bug in master a fix can be backported to 0.8-post-fixes branch.

[1] http://opensimulator.org/wiki/Debugging#Thread_Dumps

On 28/11/14 16:46, Butch Arnold wrote:
> Hello Justin,
>
> Thank you.
>
> A little more info...
> It seems not "All" scripts stop working... those which were running will remain running, but any that should start
> "After" scripts break will not start - instead they are shown in the "xengine status" report from the console as
> "Scripts Waiting For Load" with the number of scripts waiting to load... see screen captures here:
>
> Working fine:
> This is after a fresh restart of the instance....
> http://3rdrockgrid.com/new/testpics/WelcomeWorking.jpg
>
> This is immediately after we were able to break the same "Fresh Restart" from above....
> Broken:
> http://3rdrockgrid.com/new/testpics/WelcomeWorking.jpg
>
> This seems to happen on any of our instances as it should since they are all configured the same.
>
> When I upgraded, I did not use anything from the previous grid setup... ie.. I did not "Merge files" - I built a new
> grid beside the old one in case I needed to roll back to the old.
>
> Everything is fresh in this setup.
> I've installed a new database, but reloaded with a dump from the old one and new folders with no previously compiled
> scripts.
> Our setup recompiles scripts on every instance start.
>
> I thought maybe one of our custom .dll's were to blame so I removed them, running only "Bone Stock" OpenSim and still
> have the same issue.
>
>   - I have made the changes you've suggested with the same results.
>
>   - I have upgraded to the recent Ghost/Trunk merge 2 days ago with the same results.
>
>   - I have tried "Stopping and Restarting" scripts from the console when the issue starts... same results.
>
>   - I have tried "Suspending and Resuming" scripts from the console when the issue starts... same results.
>
>   - The only way to get scripts running correctly again is to restart the OpenSim instance.
>
> It seems a 32bit OpenSim instance will break instantly... ie we have found a combination which reliably crashes the
> instance each time in 32bit mode.
> If running a 32bit instance, any time a HyperGrid visitor arrives it causes the issue.
>
> In 64bit mode, we have not yet identified an event, a chain of events which cause the issue.
> Last evening, I made some changes and restarted the region in 64bit mode, we tried several times to break scripts and
> was unable to, so we were hopeful things were fixed.
> We left the region run overnight and this morning I checked and scripts were broken again.
> Logs do not indicate any issues.
>
> We are still trying to find the combination which will break scripts every time in 64bit mode.
>
> You can see it in action for yourself at:  (providing it isn't already broken)
> grid.3rdrockgrid.com:8002:welcome
>
> We have a "Rez Zone" there where you can rez objects...
>
>
> Our machine is running Windows Server 2008 R2 64bit.
>
> I have tried numerous configurations, but currently I am running:
> [BulletSim]
>      UseSeparatePhysicsThread = true
>      TerrainImplementation=0
>      AvatarToAvatarCollisionsByDefault = true
>
> [XEngine]
>      ;# {Enabled} {} {Enable the XEngine scripting engine?} {true false} true
>      ;; Enable this engine in this OpenSim instance
>      Enabled = true
>
>      ;; How many threads to keep alive even if nothing is happening
>      MinThreads = 2
>
>      ;; How many threads to start at maximum load
>      MaxThreads = 250
>
>     IdleTimeout = 60
>
>      Priority = "AboveNormal"
>
>      MaxScriptEventQueue = 300
>
>      ThreadStackSize = 462144
>
>     AppDomainLoading = true
>
>      ScriptStopStrategy = co-op
>
>      DeleteScriptsOnStartup = true
>
>      DefaultCompileLanguage = "lsl"
>
>      AllowedCompilers = "lsl"
>
>      CompileWithDebugInformation = true
>
>      AllowMODFunctions = false
>
>     AllowOSFunctions = true
>
>      Allow_osTeleportAgent = true
>      Allow_osAvatarName2Key = true
>      Allow_osNpcSay = true
>
>      AllowLightShareFunctions = false
>
>     OSFunctionThreatLevel = VeryHigh
>
>     EventLimit = 1800
>
>      KillTimedOutScripts = true
>
>      ScriptDelayFactor = 1.0
>
>      ScriptDistanceLimitFactor = 1.0
>
>      NotecardLineReadCharsMax = 255
>
>     SensorMaxRange = 96.0
>      SensorMaxResults = 16
>
>     DisableUndergroundMovement = true
>
>      ScriptEnginesPath = "ScriptEngines"
>
> Thank you.
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>

-- 
Justin Clark-Casey (justincc)
OSVW Consulting
http://justincc.org
http://twitter.com/justincc