Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007278opensim[REGION] Scripting Enginepublic2014-07-22 04:322019-02-05 12:25
Reportermewtwo0641 
Assigned Toadministrator 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionopen 
PlatformOSOS Version
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0007278: Attachment scripts always reset themselves after TP/Region Cross
DescriptionSince commit d7b92604 I have noticed that on each region cross or teleport scripted attachments always seem to reset themselves once the transfer to the region has completed.
Steps To ReproduceFor this test I have set ScriptStopStrategy to abort; I can't seem to get scripted attachments to work at all when it is set to co-op.

1. Create a new prim and put a script in it that gives you some information upon entering state_entry() in default state; Create a new script with the default "Script running" is fine for this test.

2. Take it and attach it somewhere.

3. Cross over to another region; Alternately directly TP there (as long as it is a different region than the one you're currently in)
    a. Upon entering the new region the script is highly likely to reset itself
Additional InformationI have noticed that region crossing seems to very consistently trigger the issue every single time; more so than directly teleporting (either via Map, Double Click TP, or TP Lure) but it does happen very often with direct TP. It will show up after just a few TP attempts.

-----------------
Git Bisect Info:
-----------------

$ git bisect good
d7b92604963c7ecbddf76db37eabb84ed36fcfde is the first bad commit
commit d7b92604963c7ecbddf76db37eabb84ed36fcfde
Author: Justin Clark-Casey (justincc) <jjustincc@googlemail.com>
Date: Fri Jul 11 00:03:02 2014 +0100

    If [XEngine] ScriptStopStrategy is changed between abort and co-op, for the
existing session use the previous strategy for that script rather than not start
ing the script at all.

    We have to do this since we can't unload existing DLLs if they're all in the
 same AppDomain.
    But we can still update the underlying DLL which will be used in the next si
mulator session.

:040000 040000 506e44b0969bf8fb687bdf5f24e841eec78b2c97 66997f0c4302c86bc810f0b9
473a2ecbb851f635 M OpenSim
TagsNo tags attached.
Git Revision or version numbermaster
Run Mode Standalone (Multiple Regions)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
ViewerAll
Attached Files? file icon statetest.iar [^] (104,829 bytes) 2014-12-13 22:55
? file icon failing doorscript.lsl [^] (4,921 bytes) 2015-01-25 02:20
txt file icon mono-sgen open files.txt [^] (61,046 bytes) 2015-01-27 01:46 [Show Content]
? file icon AO with OSSL.lsl [^] (16,649 bytes) 2015-01-27 14:02
? file icon * OS AO Config [^] (1,548 bytes) 2015-01-27 14:02 [Show Content]

- Relationships
related to 0007319closedadministrator XEngine Errors, Can't access an unloaded application domain 

-  Notes
(0026557)
justincc (administrator)
2014-07-22 11:03

Are these regions in the same simulator? If not, do you control both simulators (i.e. do you have access to the console/logs)?
(0026559)
mewtwo0641 (reporter)
2014-07-22 11:46

They are within the same simulator/same OpenSim process. I do have access to the console and logs for it if any further information is needed?
(0026729)
mewtwo0641 (reporter)
2014-08-11 03:44
edited on: 2014-08-11 03:59

Still an issue with commit 8738445

I have done a bit more testing with it and uncovered some more strangeness; I normally can't seem to get any of my attachment scripts working at all with the default ScriptStopStrategy of co-op. However if I recompile the scripts they start working again but the compiler gives me an error saying "Cannot find script assembly and no script text present." for each script compiled. The recompiled attachments appear to work until I cross over into another region and then, instead of resetting, they quit working entirely until I recompile again. Scripted objects rezzed in world seem to quit working completely if they are recompiled and don't know of a way to get them working again except set ScriptStopStrategy to abort.

With ScriptStopStrategy set to abort attachments simply reset on every region cross/teleport to another region. Scripted objects rezzed in world seem to work fine even if recompiled.

I can't seem to find anything out of the ordinary in the console log except for the fact that in co-op mode it never logs any attachment scripts running and does in abort mode (I have set <logger name="OpenSim.Region.ScriptEngine.XEngine"> to DEBUG in both OpenSim.exe.config and OpenSim.32BitLaunch.exe.config to check for logging of loaded scripts.)

In between switching ScriptStopStrategy back and forth I am deleting the compiled scripts in the ScriptEngines directory before starting OS again.

(0027039)
mewtwo0641 (reporter)
2014-11-28 22:40

Still an issue in commit 690fe0c although now I get a flurry of red text saying this: "[XEngine]: Failure in DoOnRezScriptQueue(). Exception"; but it doesn't give me a stack trace or any other information after the line. Is there a way to view this information (if there is any?) so I can post it here?
(0027043)
justincc (administrator)
2014-11-29 07:09

I made a mistake and didn't actually log the exception! This is now being done in git master 432f0e8 so please try that and tell me what you see.

I am actually pleased that you're now seeing this error since it will hopefully give us a clue as to what's going wrong here with the script resets.
(0027044)
mewtwo0641 (reporter)
2014-11-29 16:05
edited on: 2014-11-29 16:14

Aha! Got it :)

There seem to be two slightly different exceptions, one for attachments, and one for objects rezzed in world. The exceptions appear to be the same whether script strategy is set to co-op or abort.

Looking at the exception it seems to be a file access error on compile? I tried one thing to see if it would be any help. I checked the file permissions in the ScriptEngines directory and noticed some were read-only; I set the entire ScriptsEngines directory and its contents to be not read-only and tried to recompile some of the objects. But it didn't work and the exceptions continued to appear.


Attachments:

17:58:54 - [XEngine]: Failure in DoOnRezScriptQueue() in Test Region 2. Excepti
on System.Exception: Unable to delete old existing script-file before writing n
ew. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\
Users\admin\Desktop\ostest\opensim-432f0e8\bin\ScriptEngines\bf4a1f26-f6f2-4871-
bbf5-3620e8a39872\CommonCompiler_compiled_75226327-a618-4d32-b7f4-6b9c094840ce.d
ll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetTex
t(String Script, enumCompileType lang, String asset, String assembly) in c:\User
s\admin\Desktop\ostest\opensim-432f0e8\OpenSim\Region\ScriptEngine\Shared\CodeTo
ols\Compiler.cs:line 472
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetTex
t(String Script, enumCompileType lang, String asset, String assembly) in c:\User
s\admin\Desktop\ostest\opensim-432f0e8\OpenSim\Region\ScriptEngine\Shared\CodeTo
ols\Compiler.cs:line 476
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile
(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& a
ssembly, Dictionary`2& linemap) in c:\Users\admin\Desktop\ostest\opensim-432f0e8
\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 389
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms)
in c:\Users\admin\Desktop\ostest\opensim-432f0e8\OpenSim\Region\ScriptEngine\XEn
gine\XEngine.cs:line 1351
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dumm
y) in c:\Users\admin\Desktop\ostest\opensim-432f0e8\OpenSim\Region\ScriptEngine\
XEngine\XEngine.cs:line 1066


In World Objects:

17:36:24 - [XEngine]: Failure in DoOnRezScriptQueue() in Test Region 2. Excepti
on System.IO.IOException: The process cannot access the file 'C:\Users\admin\De
sktop\ostest\opensim-432f0e8\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39
872\CommonCompiler_compiled_65703ddf-427b-4fc9-864f-14ecf07cac2f.dll.map' becaus
e it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, I
nt32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions o
ptions, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolea
n useLongPath)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access,
FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.File.Create(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.WriteMapFile(String
filename, Dictionary`2 linemap) in c:\Users\admin\Desktop\ostest\opensim-432f0e8
\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 790
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile
(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& a
ssembly, Dictionary`2& linemap) in c:\Users\admin\Desktop\ostest\opensim-432f0e8
\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 370
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms)
in c:\Users\admin\Desktop\ostest\opensim-432f0e8\OpenSim\Region\ScriptEngine\XEn
gine\XEngine.cs:line 1351
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dumm
y) in c:\Users\admin\Desktop\ostest\opensim-432f0e8\OpenSim\Region\ScriptEngine\
XEngine\XEngine.cs:line 1066

(0027045)
AliciaRaven (manager)
2014-11-30 00:36

I have a related error that i have been seeing for months now. The following appears on the console some times although i havn't noticed any effect in world so have been ignoring it. Seems sharing violations are common with xengine.

ERROR [XEngine]: Linemap file ScriptEngines/cef549bc-f382-491b-b757-e6bbabbbd2ec/CommonCompiler_compiled_ee21f20b-b504-42d3-8089-5e52d58b93c4.dll.map already exists! Sharing violation on path /OsBuild/ScriptEngines/cef549bc-f382-491b-b757-e6bbabbbd2ec/CommonCompiler_compiled_ee21f20b-b504-42d3-8089-5e52d58b93c4.dll.map

This is under linux and mono (multiple version numbers)
(0027055)
justincc (administrator)
2014-12-03 13:26

@mewtwo0641 - On reviewing the code, for the second error I did find that OpenSimulator never closed the map file after reading it, a problem that has been fixed by git master 72d1d96. However, I can't directly see how this would cause your problem - once the line map is loaded it stays in memory and so should never need to be loaded again. No other code should have that file open. Nonetheless, please try the latest master and tell me what you see.

Regarding the first error, this is also somewhat mysterious. If you look at the freshly created DLL and associated files I presume there are no permission problems (e.g. read-only)?

I have also changed the logging so that script recompiles in response to a change in ScriptStopStrategy are explicitly logged. However, these are at debug level. I have also made a change so that the explicit logging about script loading now only occurs if such loggings has been enabled via the "xengine debug log" command, so you should be able to remove the XEngine section in OpenSim.exe.config without seeing massive log spam. So I would like to see more of the log on these errors to check that somehow recompilation isn't constantly retriggering and causing problems.

@AliciaRaven - I improved the log message slightly at 805b7cc to remove the "already exists!" text from the exception since this is not actually the problem (as you can see the actual issue is the sharing violation). The change to properly close the map file after reading may help you here, though I'm relatively unfamiliar with that part of the code since it's only executed if [XEngine] TrustBinaries = true which is not a commonly used setting.
(0027057)
justincc (administrator)
2014-12-03 13:53

Also, I expect the problem with attachment scripts stopping is related to these exceptions since previously they would completely stop any further compilation in the region until restart. Since 690fe0c other scripts should continue to compile and work even if exceptions are thrown when processing others.
(0027061)
mewtwo0641 (reporter)
2014-12-03 21:29

Commit 9dbe99a seems to be a bit better but it depends on which ScriptStopStrategy is being used. When set to abort scripts seem to compile and run fine except that attachments still reset themselves upon entering a different region. I did notice the occasional warning when entering another region to the effect that it could not delete the attachment script state for a random script. This doesn't always happen (I tested maybe 1 out of 8 region crossings that it happens).

When set to co-op the issue appears again with compile exceptions (Regardless of prior removal of ScriptEngines or not). I believe the exception is the same as previous but I am providing it below just in case it isn't. (Previously I would get the exceptions on either setting)

It doesn't appear to properly switch over when changing the setting from co-op to abort or vice versa (I get the exceptions again until I do so). I have to completely remove the ScriptEngines directory to get the scripts to load properly under the abort setting; co-op gives me the exceptions no matter what. I also tried switching back and forth without removing the ScriptEngines directory and it resulted in a different exception (also listed below); This was the result of starting at co-op switching to abort and then back to co-op. I believe it was associated with an attachment.

The strange thing seems to be is that some scripts will compile and function on the initial load up... at least until a recompile is attempted on said scripts. But most scripts don't seem to want to cooperate right off the bat.

I have checked the file permissions on the contents of ScriptEngines and the only attribute they have is archive; nothing read-only. I also double checked the file permissions and ownership via Windows' Security tab in the directory properties window and that seems to check out good as well. For good measure and to rule out any likelihood that it might be a path access issue I have also tried moving the OS install to another disk, as close to root as possible, i.e. G:\opensim-9dbe99a\bin; but this did not seem to change anything.

I am a bit confused on the usage of the xengine debug log command. I keep receiving 'Invalid command' from it.

On a side note; I am also noticing some slow handling LOGHTTP on script saves although I am unsure if this has any relation.

-----------------------------------------------------------------------------------

Slow handling:
--------------

22:56:39 - [LOGHTTP]: Slow handling of 38 POST /CAPS/088ce0bd-9293-4c90-aa97-13b
f40631a655705 TaskInventoryScriptUpdater from 192.168.1.100:52966 took 4103ms


Warning on region cross (abort):
---------------------------------

22:31:08 - [SCRIPT INSTANCE]: Could not delete script state G:\opensim-9dbe99a\b
in\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\552e1bc0-4be1-4675-a096-a5
34fec1ded1.state for script - SimpleAO Texture - (id 552e1bc0-4be1-4675-a096-a53
4fec1ded1) in part Simple AO v1.0.0 (id 3556bba7-57ec-416e-8fef-961753d7f25f) in
 object Simple AO v1.0.0 in Test Region 2. Exception System.IO.IOException: Th
e process cannot access the file 'G:\opensim-9dbe99a\bin\ScriptEngines\bf4a1f26-
f6f2-4871-bbf5-3620e8a39872\552e1bc0-4be1-4675-a096-a534fec1ded1.state' because
it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.RemoveState() i
n c:\Users\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\Shar
ed\Instance\ScriptInstance.cs:line 506


Exception on co-op:
--------------------

21:44:56 - [XEngine]: Failure in DoOnRezScriptQueue() in Test Region. Exception
  System.Exception: Unable to delete old existing script-file before writing new
. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\Us
ers\admin\Desktop\ostest\opensim-9dbe99a\bin\ScriptEngines\bf4a1f26-f6f2-4871-bb
f5-3620e8a39872\CommonCompiler_compiled_75d39dc4-0f0f-49d9-aa54-e855a2ecfa5a.dll
' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetTex
t(String Script, enumCompileType lang, String asset, String assembly) in c:\User
s\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\Shared\CodeTo
ols\Compiler.cs:line 472
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetTex
t(String Script, enumCompileType lang, String asset, String assembly) in c:\User
s\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\Shared\CodeTo
ols\Compiler.cs:line 476
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile
(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& a
ssembly, Dictionary`2& linemap) in c:\Users\admin\Desktop\ostest\opensim-9dbe99a
\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 389
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms)
in c:\Users\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\XEn
gine\XEngine.cs:line 1359
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dumm
y) in c:\Users\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\
XEngine\XEngine.cs:line 1066


Exception when switch to abort and back to co-op with out removal of ScriptEngines:
------------------------------------------------------------------------------------

23:05:55 - [XEngine]: Failure in DoOnRezScriptQueue() in Test Region. Exception
  System.BadImageFormatException: Could not load file or assembly 'CommonCompile
r_compiled_75d39dc4-0f0f-49d9-aa54-e855a2ecfa5a' or one of its dependencies. The
 module was expected to contain an assembly manifest.
File name: 'CommonCompiler_compiled_75d39dc4-0f0f-49d9-aa54-e855a2ecfa5a' ---> S
ystem.BadImageFormatException: Could not load file or assembly 'file:///G:\opens [^]
im-9dbe99a\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler
_compiled_75d39dc4-0f0f-49d9-aa54-e855a2ecfa5a.dll' or one of its dependencies.
The module was expected to contain an assembly manifest.
File name: 'file:///G:\opensim-9dbe99a\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5 [^]
-3620e8a39872\CommonCompiler_compiled_75d39dc4-0f0f-49d9-aa54-e855a2ecfa5a.dll'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String cod
eBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark&
stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppre
ssSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String code
Base, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& s
tackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppres
sSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName as
semblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntr
ospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Ev
idence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm,
Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackM
ark)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.OnAssemblyResolve(Object sende
r, ResolveEventArgs args) in c:\Users\admin\Desktop\ostest\opensim-9dbe99a\OpenS
im\Region\ScriptEngine\XEngine\XEngine.cs:line 1730
   at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String a
ssemblyFullName)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\M
icrosoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure lo
gging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fus
ion!EnableLog].

   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String cod
eBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark&
stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppre
ssSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String code
Base, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& s
tackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppres
sSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName as
semblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntr
ospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evid
ence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.AppDomain.Load(String assemblyString)
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms)
in c:\Users\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\XEn
gine\XEngine.cs:line 1330
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dumm
y) in c:\Users\admin\Desktop\ostest\opensim-9dbe99a\OpenSim\Region\ScriptEngine\
XEngine\XEngine.cs:line 1066
(0027064)
justincc (administrator)
2014-12-04 13:08

@mewtwo0641 - Are you calling llResetScript() or llResetOtherScript() somewhere in these attachments? If so, can you attach a copy of this script so I can do direct testing and think about how these calls may be interacting with other thread activity on region cross?
(0027065)
justincc (administrator)
2014-12-04 13:09

Also, the debug log command is "debug xengine log 1" to turn extra logging on and "debug xengine log 0" to turn it off. However, right now it won't show anything apart from notifications on loading each script (though that may still be useful).
(0027067)
mewtwo0641 (reporter)
2014-12-04 22:59

A couple of the attachments do but they aren't set up to be triggered on a region cross; only when invoked from a menu. However I have been able to consistently replicate the issue with a simple default new script.

I tried 'debug xengine log 1' and it says setting the debug level but there does not seem to be a difference in log output on the console; no notifications of scripts being loaded.
(0027079)
Gavin Hird (reporter)
2014-12-09 03:45

Adding two notes from the opensim-dev which observes the same issue with scripts on teleports and region crossing:

I took commit r/25599 this morning to test if the changes to scripts improved the situation as described before and the first impression was that "nothing works" when it came to avatar attached scripts. – Which was not entirely true as HUD attachments started fine. Detaching and attaching attachments started them up again.

When teleporting to a region on the same instance, the same happened, HUDs started, the attached ones did not, and reattaching did not even start them.

So I reverted.

Turned out the exact same thing happened so it was not r/25599.

Checking back what I did a few days earlier was I set ScriptStopStrategy = co-op as that is the default in the dev code. AppDomainLoading = true has been set like that for quite a few months.

Setting ScriptStopStrategy = abort and recompiling all scripts restored the attachment loading; also on teleports between regions in the same simulator instance. Teleporting to a new instance produced the same result as described before; HUD and avatar attached scripts do not start.

So the question is; are AppDomainLoading = true and ScriptStopStrategy = co-op mutually exclusive in reality?
(0027080)
Gavin Hird (reporter)
2014-12-09 03:45

I took the master @ r/25604, compiled and installed on my test sims instance.

I think I can conclude that with AppDomainLoading = true script loading for attachments on teleports between sims does not work at all.

With AppDomainLoading = false, and ScriptStopStrategy = abort or co-op it both works, but is very inconsistent. Everything loads on login, then it may or may not work on teleports and there is no pattern. The only thing I can say is there seem to be a slightly higher chance the scripts don't load when returning to the login sim than to other sims. The scripts I tested were both HUD and direct avatar attached.

With r/25593 I could turn on script logging, but there was nothing special logged as I could see. It logged that they loaded, if they did not there was not a pip.

The HUD I tested was an AO I have seen in use or distributed many places and it has a Zhao II script modified for opensim (not modified by me.)
(0027081)
justincc (administrator)
2014-12-09 16:43

Hi @mewtwo0641. So I finally went back and properly read what you had first reported again. I was promptly able to recreate the same problem of scripts in attachments restarting and not preserving state on cross.

With AddDomainLoading = false and multiple regions, the same compiled script DLL is used on both regions. This in itself is fine, but I believe that a change in d7b92604 meant that the state and other files for both regions were being posted to the same location, hence overwriting each other and quite possibly causing the exceptions you have been seeing.

I have fixed this in git master 2b9f064 and on my own testing script state is now properly preserved on region cross and events such as state_entry are not fired. Please test and let me know what you see. Sorry for taking so long to get to this - an awful lot of my free time was absorbed by the conference and post-conference work.

@Gavin - Please could you re-test as well. Although the fixed issue immediate relates to ScriptDomainLoading = false with two regions in the same simulator, it's not impossible that there were knock-on effects.

Regarding AppDomainLoading = true and ScriptStopStrategy = co-op, these should be able to co-exist.
(0027082)
Gavin Hird (reporter)
2014-12-10 00:51
edited on: 2014-12-10 00:53

I tested with both AppDomainLoading = false and ScriptStopStrategy = co-op or abort taking care to delete compiled scripts and script states between startups of the simulator to check if even initial state behavior was consistent.

On most runs all scripts would load when logging on to a freshly initiated simulator, but there was one case where none loaded at all.

On teleporting between regions in the simulator it is still completely random when the scripts starts or not. The same for walking across region borders. The logging shows that sometimes the scripts starts at other times there is nothing logged.

To me it does not look like the changes have not made any difference at all. Still the HUD attached scripts are more likely to survive a region crossing (TP or walking).

– I even diffed the files to make sure I had pulled and compiled the changes as the behavior was more or less identical to before.


To me it looks like xengine at times never gets the message an avatar has changed region and therefore never does anything to start the scripts.

Mono 3.10.0.20

(0027083)
mewtwo0641 (reporter)
2014-12-10 08:42

I am seeing similar results as Gavin.

ScriptStopStrategy = abort and AppDomainLoading = false seemed to yield the best results for me but it was fairly inconsistent. At first I thought that set-up was working consistently; scripts loading fine and states being maintained; so I proceeded to try the other possible combinations of the two settings. They all yielded very inconsistent results regarding whether a script would run or not. On a whim, I switched back to the original setting of ScriptStopStrategy = abort and AppDomainLoading = false and tried one more time but this time it resulted in the same exceptions and script state issues as described previously in this mantis. So that basically foiled my attempts to do a systematic test with each setting combination :) As I was attempting to test the settings I was making sure to remove the script dll's and states before each OS load.

One thing I did notice though was on the times I was able to get the scripts working; script states would consistently reset themselves on every login; but would be fine most region crosses. I wasn't able to trigger a state reset on a region cross (Attempted 0000006:0000010 crosses in quick succession) this time around so as far as I can tell when scripts are working properly region crosses maintain the script state. The exception seemed to be TP lures across regions; states would randomly but not consistently reset upon arrival. There was no exception logged in the console in this case.

On a side note I am having to test using OpenSim.32BitLaunch.exe because OpenSim.exe is resulting in an exception for me upon login. I believe this happened previously and had something to do with libomv which was fixed by compiling it under Windows. It has worked for a little while with no issues but I am now getting login exceptions again. (Perhaps I should file this under a new Mantis?)

No worries about the time Justin :) Life can be hectic at times understandably.

Exception on Login with OpenSim.exe:

2014-12-10 09:32:14,067 WARN - OpenSim.Services.LLLoginService.LLLoginService [LLOGIN SERVICE]: Exception processing login for Test User: System.TypeInitializationException: The type initializer for 'OpenMetaverse.AppearanceManager' threw an exception. ---> System.ArgumentException: Value does not fall within the expected range.
   at System.Runtime.CompilerServices.RuntimeHelpers.InitializeArray(Array array, RuntimeFieldHandle fldHandle)
   at OpenMetaverse.AppearanceManager..cctor()
   --- End of inner exception stack trace ---
   at OpenSim.Framework.AvatarAppearance.SetDefaultTexture() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Framework\AvatarAppearance.cs:line 307
   at OpenSim.Framework.AvatarAppearance..ctor() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Framework\AvatarAppearance.cs:line 139
   at OpenSim.Services.Interfaces.AvatarData.ToAvatarAppearance() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Services\Interfaces\IAvatarService.cs:line 193
   at OpenSim.Services.AvatarService.AvatarService.GetAppearance(UUID principalID) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Services\AvatarService\AvatarService.cs:line 57
   at OpenSim.Services.LLLoginService.LLLoginService.Login(String firstName, String lastName, String passwd, String startLocation, UUID scopeID, String clientVersion, String channel, String mac, String id0, IPEndPoint clientIP) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Services\LLLoginService\LLLoginService.cs:line 464 at OpenSim.Framework.AvatarAppearance.SetDefaultTexture() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Framework\AvatarAppearance.cs:line 307
   at OpenSim.Framework.AvatarAppearance..ctor() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Framework\AvatarAppearance.cs:line 139
   at OpenSim.Services.Interfaces.AvatarData.ToAvatarAppearance() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Services\Interfaces\IAvatarService.cs:line 193
   at OpenSim.Services.AvatarService.AvatarService.GetAppearance(UUID principalID) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Services\AvatarService\AvatarService.cs:line 57
   at OpenSim.Services.LLLoginService.LLLoginService.Login(String firstName, String lastName, String passwd, String startLocation, UUID scopeID, String clientVersion, String channel, String mac, String id0, IPEndPoint clientIP) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Services\LLLoginService\LLLoginService.cs:line 464
(0027084)
justincc (administrator)
2014-12-10 16:43

@mewtwo0641 - During the ghosts merge I managed to insert the Linux built DLL again. I've corrected this in 92c9dd4 so please try that. Please notify me if it doesn't work and if it does, please use OpenSim.exe since that is what I am using. Tell me if the same problems are occurring.

Otherwise things get more difficult. I just did a bunch of teleporting around a LAN mini grid with some regions on Linux (with ScriptStopStrategy = co-op, AppDomainLoading = false) and others on Windows (ScriptStopStrategy = co-op, AppDomainLoading = true) with my scripted attachment active and state preserving all the time. Here's the test script I use.

integer n = 0;

default
{
    state_entry()
    {
        llSay(0, "Script running");
    }
    
    touch_start(integer i)
    {
        n++;
        llOwnerSay("n: " + (string)n);
    }
}

I put this in a prim attached to my head and click it to make sure the counter increments and doesn't reset and that one never sees "Script Running" after the initial script save.

Could you try with exactly this script? Also, could you please run "debug xengine log 1" on all the applicable simulators during the teleport and post a log where things go wrong. There won't be much extra data yet but this will let me think what more I need to add to get progress on this problem. Also, please tell me the steps that led up to the failure (if you had successful teleports before, region cross, etc.) whether the regions are on the same simulator, on the same machine, etc.

@Gavin - I would ask you to do the same. I would also ask you, if at all possible, to use Mono 3.2.8 instead of 3.10 as 3.2.8 is the one commonly shipped with distros (e.g. Ubuntu) and the version I am using. At this stage, I am not convinced that Mono 3.10 is without bugs or behaviour changes compared with something like 3.2.8.

As for the scripts not always being active during startup, I believe this is a different bug that I have observed elsewhere but didn't have time to resolve.
(0027085)
Gavin Hird (reporter)
2014-12-11 01:25
edited on: 2014-12-11 06:06

Observation:

1. Replacing the DLL did not make any difference.

2. Crossing between regions (in the same simulator instance - I am only testing that for now) in rapid succession makes script not load, and nothing is logged. However, the Singularity viewer flags a script error on crossing, but there is no message text. Cool VL viewer does not flag anything.

3. Waiting about a minute or so between region crossing significantly increase the likelyhood of the script loading – is there a timeout issue?

4. Regions that has been "tainted" by a script not loading is less likely to ever load that script again unless you log out. Question: What happens to script state of attached scripts on region crossings? Should they not immediately be invalidated on the avatar leaving the region, and script state be passed from the region the avatar is coming in from? (This might already happen for what I know, but is seems to me it is picking up the old state...)

Reverting back to mono 3.2.7 did not make much of a difference (there was never a 3.2.8 OS X version as this was a Linux only bug cleanup.)

UPDATE: It was tested both with your script and the before mentioned AO

(0027088)
Gavin Hird (reporter)
2014-12-11 08:06
edited on: 2014-12-11 09:03

I think maybe this might be significant, and you can try and verify it.

This file: openjpeg-dotnet-x86_64.dll is included in the bin directory with a 32-bit equivalent

I am running 32-bit mono and on restarting Robust I happen to spot an error message to the extent this 64-bit file could not be loaded as it was in an invalid format. I could not find it in the log opensim log file, so it might have come directly from mono. Unfortunately I was running in screen without scroll back, so the message is lost, but nevertheless, I removed the file from bin in both instances and restarted everything.

Guess what? Scripts are loading on both teleports and walking across the sim border on both instances fairly consistently. After testing multiple runs I have only seen one instance it did not.

There may still be an issue when teleporting between simulator instances running on different machines.

So maybe this attempt to load a 64bit dll tripped mono to behave erratically. One of the instances I tested on is running 3.2.7 the other 3.10.0.20


UPDATE: The problem still exist, but it is significantly less pronounced after removing the 64-bit DLL, and even less so on 3.2.7 than on 3.10.0.20. There might still be a timeout issue where the Singularity viewer flags a script error in some cases even on initial login. There are no log messages to this extent.

(0027089)
Gavin Hird (reporter)
2014-12-11 13:54

After I deleted openjpeg-dotnet-x86_64.dll is the first time I have had visitors with 64-bit viewers on the grid without any issues in the sims. Before they have gradually been locking themselves and others out of sims with denied access messages.
(0027090)
zadark (reporter)
2014-12-11 16:08

Robust uses its own log file Robust.log, also located in the bin directory.

I run Opensim on 32 bit linux both robust and simulator and have not or cannot see any references to openjpeg libraries with any log file.

Would be of interest what the message robust actually reported.
(0027091)
Gavin Hird (reporter)
2014-12-11 23:46
edited on: 2014-12-11 23:46

The structure of the message was identical to this one:

Failed to load assembly D:\Games\Kerbal Space Program\GameData\Kethane\Plugins\MMI_Kethane.dll:
System.BadImageFormatException: Format of the executable (.exe) or library (.dll) is invalid

except the path and file was to openjpeg-dotnet-x86_64.dll

I did not copy it off the console immediately because my knee-jerk reaction was it is going to be in the log file, but it wasn't – which puzzles me. The simulators logged the same on startup, but there is no log entry for those either. Which puzzles me even more.

(0027093)
Gavin Hird (reporter)
2014-12-12 09:29

I am not sure if this entirely the same as discussed with scripts and sim crossings, but this is a snippet from the log when the simulators starts to deny agents access to the regions.

It starts when they want to teleport to a region that is not the immediate regions, but one hop away, and the simulators wants to reuse existing child scene presence. I have no idea if these agents had script attachments but I suppose they have the FIRESTORM BRIDGE attached.

It is always Firestorm causing this, and in this case Firestorm-Releasex64 4.6.7.42398 which is the previous version and not the one released yesterday/today 4-6-9-42974

I am tempted to ban the 4.6.7.xxx versions from the grid, haha!

-------------------




907d0331-2b11-416c-8f3d-75eca111970c to Windfall
2014-12-12 15:34:47,379 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLClientView [CLIENT]: Close has been called for April.Sis @grid.kitely.com attached to scene Windfall
2014-12-12 15:34:47,380 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Removing child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c from Windfall
2014-12-12 15:34:47,380 DEBUG - OpenSim.Region.CoreModules.Framework.CapabilitiesModule [CAPS]: Remove caps for agent 907d0331-2b11-416c-8f3d-75eca111970c in region Windfall
2014-12-12 15:34:47,381 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Request: https://www.virwox.com:419/OSMoneyGateway/4.0.3/API/Simulator.php?method=leaveUser&avatarUUID=907d0331-2b11-416c-8f3d-75eca111970c&regionUUID=facb89eb-2f52-4914-94ac-652ab4510276&gridShortName=xmir&gridURL=http://grid.xmir.org:8003 [^]
2014-12-12 15:34:47,381 DEBUG - OpenSim.Region.Framework.Scenes.SceneCommunicationService [SCENE COMMUNICATION SERVICE]: Sending close agent ID 907d0331-2b11-416c-8f3d-75eca111970c to Herad
2014-12-12 15:34:47,384 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLClientView [CLIENT]: Close has been called for April.Sis @grid.kitely.com attached to scene Herad
2014-12-12 15:34:47,384 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Removing child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c from Herad
2014-12-12 15:34:47,384 DEBUG - OpenSim.Region.CoreModules.Framework.CapabilitiesModule [CAPS]: Remove caps for agent 907d0331-2b11-416c-8f3d-75eca111970c in region Herad
2014-12-12 15:34:47,385 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Request: https://www.virwox.com:419/OSMoneyGateway/4.0.3/API/Simulator.php?method=leaveUser&avatarUUID=907d0331-2b11-416c-8f3d-75eca111970c&regionUUID=facb89eb-2f52-4914-94ac-652ab4510276&gridShortName=xmir&gridURL=http://grid.xmir.org:8003 [^]
2014-12-12 15:34:47,572 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Response: {'status':'OK'}
2014-12-12 15:34:47,604 DEBUG - OpenSim.Region.Framework.Scenes.ScenePresence [SCENE PRESENCE]: Making April.Sis @grid.kitely.com a child agent in andwest
2014-12-12 15:34:47,605 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferStateMachine [ENTITY TRANSFER STATE MACHINE]: Agent 907d0331-2b11-416c-8f3d-75eca111970c cleared from transit in andwest
2014-12-12 15:34:47,782 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Response: {"success":"true"}
2014-12-12 15:34:47,785 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Response: {"success":"true"}
2014-12-12 15:34:47,797 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Response: {"success":"true"}
2014-12-12 15:34:47,884 DEBUG - OMEconomy.OMBase.CommunicationHelpers [OMECONOMY] Response: {"success":"true"}
2014-12-12 15:34:48,115 DEBUG - OpenSim.Region.CoreModules.Avatar.BakedTextures.XBakesModule [XBakes]: read 5 textures for user 907d0331-2b11-416c-8f3d-75eca111970c
2014-12-12 15:34:48,177 DEBUG - OpenSim.Region.CoreModules.Avatar.BakedTextures.XBakesModule [XBakes]: read 5 textures for user 907d0331-2b11-416c-8f3d-75eca111970c
2014-12-12 15:34:48,244 DEBUG - OpenSim.Region.Framework.Scenes.ScenePresence [SCENE PRESENCE]: Baked textures are in the cache for April.Sis @grid.kitely.com in andwest Bay
2014-12-12 15:34:48,314 DEBUG - OpenSim.Region.CoreModules.Avatar.AvatarFactory.AvatarFactoryModule [AVFACTORY]: Received texture update for April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c
2014-12-12 15:34:48,474 DEBUG - OpenSim.Region.CoreModules.Avatar.BakedTextures.XBakesModule [XBakes]: stored 5 textures for user 907d0331-2b11-416c-8f3d-75eca111970c
2014-12-12 15:34:48,926 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour Barcola 81.166.26.186:9010 at (1000,999)
2014-12-12 15:34:48,927 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region Barcola told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <281.4176, 502.2954, 25.87437>.
2014-12-12 15:34:48,927 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Reusing existing child scene presence for April.Sis @grid.kitely.com, state Running in Barcola
2014-12-12 15:34:48,928 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Adjusting known seeds for existing agent 907d0331-2b11-416c-8f3d-75eca111970c in Barcola
2014-12-12 15:34:49,057 DEBUG - OpenSim.Region.CoreModules.Avatar.BakedTextures.XBakesModule [XBakes]: read 5 textures for user 907d0331-2b11-416c-8f3d-75eca111970c
2014-12-12 15:34:49,436 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour andwest Sea 81.166.26.186:9011 at (1001,999)
2014-12-12 15:34:49,440 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region andwest Sea told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <25.41759, 502.2954, 25.87437>.
2014-12-12 15:34:49,441 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Reusing existing child scene presence for April.Sis @grid.kitely.com, state Running in andwest Sea
2014-12-12 15:34:49,441 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Adjusting known seeds for existing agent 907d0331-2b11-416c-8f3d-75eca111970c in andwest Sea
2014-12-12 15:34:49,963 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour Landmark 81.166.26.186:9012 at (1002,999)
2014-12-12 15:34:49,963 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region Landmark told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <-230.5824, 502.2954, 25.87437>.
2014-12-12 15:34:49,984 WARN - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Region Landmark did not accept April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c: Failed to verify user presence in the grid for April.Sis @grid.kitely.com, access denied to region Landmark.
2014-12-12 15:34:50,486 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour andwest 81.166.26.186:9000 at (1000,1000)
2014-12-12 15:34:50,487 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region andwest told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <281.4176, 246.2954, 25.87437>.
2014-12-12 15:34:50,488 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Reusing existing child scene presence for April.Sis @grid.kitely.com, state Running in andwest
2014-12-12 15:34:50,488 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Adjusting known seeds for existing agent 907d0331-2b11-416c-8f3d-75eca111970c in andwest
2014-12-12 15:34:51,003 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour East andwest 81.166.26.186:9002 at (1002,1000)
2014-12-12 15:34:51,003 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region East andwest told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <-230.5824, 246.2954, 25.87437>.
2014-12-12 15:34:51,026 WARN - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Region East andwest did not accept April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c: Failed to verify user presence in the grid for April.Sis @grid.kitely.com, access denied to region East andwest.
2014-12-12 15:34:51,449 DEBUG - OpenSim.Region.CoreModules.World.Land.LandManagementModule [LAND MANAGEMENT MODULE]: Got parcelID 00000000-0000-0000-0000-000000000000
2014-12-12 15:34:51,534 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour Hengill 81.166.26.186:9003 at (1000,1001)
2014-12-12 15:34:51,535 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region Hengill told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <281.4176, -9.704605, 25.87437>.
2014-12-12 15:34:51,535 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Reusing existing child scene presence for April.Sis @grid.kitely.com, state Running in Hengill
2014-12-12 15:34:51,535 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Adjusting known seeds for existing agent 907d0331-2b11-416c-8f3d-75eca111970c in Hengill
2014-12-12 15:34:52,038 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour Lake Atlas 81.166.26.186:9004 at (1001,1001)
2014-12-12 15:34:52,039 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region Lake Atlas told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <25.41759, -9.704605, 25.87437>.
2014-12-12 15:34:52,039 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Reusing existing child scene presence for April.Sis @grid.kitely.com, state Running in Lake Atlas
2014-12-12 15:34:52,040 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Adjusting known seeds for existing agent 907d0331-2b11-416c-8f3d-75eca111970c in Lake Atlas
2014-12-12 15:34:52,584 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Informing April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c about neighbour Norderhov 81.166.26.186:9005 at (1002,1001)
2014-12-12 15:34:52,584 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region Norderhov told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (Default), position <-230.5824, -9.704605, 25.87437>.
2014-12-12 15:34:52,604 WARN - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Region Norderhov did not accept April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c: Failed to verify user presence in the grid for April.Sis @grid.kitely.com, access denied to region Norderhov.
2014-12-12 15:35:07,861 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferStateMachine [ENTITY TRANSFER STATE MACHINE] SetInTransit. agent=907d0331-2b11-416c-8f3d-75eca111970c, newState=Preparing
2014-12-12 15:35:07,862 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE] GetRegionContainingWorldLocation: call, XY=<256639.769554138,256385.239776611>
2014-12-12 15:35:07,862 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE] GetRegionContainingWorldLocation: Found region using legacy size. rloc=<256512,256256>. Rname=Norderhov
2014-12-12 15:35:07,863 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.HGEntityTransferModule [HG ENTITY TRANSFER MODULE]: region Norderhov flags: 4
2014-12-12 15:35:07,863 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Teleporting April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c from andwest Bay to http://grid.xmir.org:9000/ [^] (http://grid.xmir.org:9000/ [^]) Norderhov/<127.7696, 129.2398, 36.88613>
2014-12-12 15:35:07,865 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: andwest Bay max transfer version is SIMULATION/0.3, Norderhov max version is SIMULATION/0.3
2014-12-12 15:35:07,866 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.HGEntityTransferModule [HG ENTITY TRANSFER MODULE]: CreateAgent http://grid.xmir.org:9000/ [^] http://grid.xmir.org:9000/ [^]
2014-12-12 15:35:07,867 DEBUG - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Region Norderhov told of incoming child agent April.Sis @grid.kitely.com 907d0331-2b11-416c-8f3d-75eca111970c (circuit code 414768131, IP 62.235.160.117, viewer Firestorm-Releasex64 4.6.7.42398, teleportflags (ViaLure), position <127.7696, 129.2398, 36.88613>. From region andwest Bay (40196348-7ba9-43d5-bc5c-c6c421e2e57d) @ http://grid.xmir.org:8002/ [^]
2014-12-12 15:35:07,881 DEBUG - OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule [ENTITY TRANSFER MODULE]: Teleport of April.Sis @grid.kitely.com from andwest Bay to Norderhov was refused because Failed to verify user presence in the grid for April.Sis @grid.kitely.com, access denied to region Norderhov.
(0027094)
Gavin Hird (reporter)
2014-12-12 13:25

I reverted back to the previous master on the prod server because suddenly a lot of meshes were like butter without physics models. Could be anything that caused it.

Test instance is still running the latest master. Not seen any problems like that there.
(0027095)
mewtwo0641 (reporter)
2014-12-13 22:55

For good measure I made multiple copies of your test script and attached one to each avatar point (including HUDs) each with the attachment point name in the object. Probably a bit excessive but I wanted to see if it was random attachment points that reset or if there was a consistent pattern of points showing up. My thinking here is that there seems to be a higher chance of seeing the issue the more scripts there are attached.

The tests below were done and results were achieved assuming the user waits for all scripts to finish loading before attempting another region cross or accepting TP lure.

In my case the HUDs did not seem to be exempt from loading issues / script state resets.

Tests 3 and 4 seemed to yield the best results for me.

In Tests 3 and 4 I was able to region cross back and forth in rapid succession without triggering exceptions or state resets. I am wondering if this has to do with AppDomainLoading = false in contrast to Tests 1 and 2 it being set to true? This was tested by flying parallel to the region borders and zig zagging in a very tight very fast fashion between the two regions.

As a final test (After completing with the settings in Test 4) I took off the test attachments and put on my normal every day attachments and there was a flood of exceptions again (The "Failure in DoOnRezScriptQueue() in Test Region. Exception
System.Exception: Unable to delete old existing script-file before writing new. Compile aborted..." exceptions) and those scripts wouldn't function.

I have attached an IAR of the objects I tested with; it is your test script copied into one object per attach point.

-------
Test 1
-------

ScriptStopStrategy = abort
AppDomainLoading = true
DeleteScriptsOnStartup = false

Initial load up there seemed to be no issues loading scripts.

In this setup individual script load appears to be very slow on login, attach object, teleports, region crosses, etc. (about 1 script per 0.5 second).

10 Region Crosses (On system separate from the OpenSim system)
    * All Good; All scripts load; Script states preserved
    
10 Region Crosses (On same system as OpenSim is running)
    * All Good; All scripts load; Script states preserved

10 TP Lures Cross Region (Split in half, 5 per user, 2 users, one user on OpenSim system, one user on separate system):
    * 3 Bad; Scripts did not load at all or only some loaded; nothing was logged in the console in the case of the non loading scripts.
    * 6 Good; All scripts loaded; Script states preserved.
    * 1 Exception; Exceptions were logged on some scripts (Counted about 5 out of the 40 scripts all the same except for the different attachment points)

In the case of non loading scripts, they do load upon the next TP or region cross, but their states are reset when they load. It did not seem to matter whether the crossing was done on a separate system or on the system that OpenSim was running on.

In this set-up I noticed during testing that if I attempted a region cross while the scripts were still loading the script states would be consistently be reset. This is easier to achieve when there are a fair amount of scripts involved in the attachment(s). In this case there were 40 scripts due to having one test script in each object per attach point.

Exception:

2014-12-11 00:55:26,690 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failed to save state of script Script State Test (Right Hip).Script State Test, item UUID 6bafe989-4c13-4199-ba16-afbf5ec278ff, prim UUID b2d87a39-2a25-427f-b376-8179de033ee2 in Test Region. Exception
System.AppDomainUnloadedException: The target application domain has been unloaded.

Server stack trace:
   at System.Threading.Thread.InternalCrossContextCallback(Context ctx, IntPtr ctxID, Int32 appDomainID, InternalCrossContextDelegate ftnToCall, Object[] args)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(IMessage reqMsg)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at OpenSim.Region.ScriptEngine.Shared.ScriptBase.IScript.GetVars()
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.GetVars() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\Instance\ScriptInstance.cs:line 1022
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptSerializer.Serialize(ScriptInstance instance) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\Instance\ScriptSerializer.cs:line 80
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.SaveState() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\Instance\ScriptInstance.cs:line 1064
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoBackup(Object o) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 840
2014-12-11 00:55:26,743 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failed to save state of script Script State Test (L Forearm).Script State Test, item UUID d7f2039d-c4d9-4083-8785-440a1bc40058, prim UUID 9d0f57dd-b776-4d7d-a99a-32915a4c9a10 in Test Region. Exception
System.AppDomainUnloadedException: The target application domain has been unloaded.

Server stack trace:
   at System.Threading.Thread.InternalCrossContextCallback(Context ctx, IntPtr ctxID, Int32 appDomainID, InternalCrossContextDelegate ftnToCall, Object[] args)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)
   at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(IMessage reqMsg)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at OpenSim.Region.ScriptEngine.Shared.ScriptBase.IScript.GetVars()
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.GetVars() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\Instance\ScriptInstance.cs:line 1022
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptSerializer.Serialize(ScriptInstance instance) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\Instance\ScriptSerializer.cs:line 80
   at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.SaveState() in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\Instance\ScriptInstance.cs:line 1064
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoBackup(Object o) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 840
   
   
-------
Test 2
-------

ScriptStopStrategy = co-op
AppDomainLoading = true
DeleteScriptsOnStartup = false

Initial load up there seemed to be no issues loading in world scripts.

In this setup individual script load appears to be very slow on login, attach object, teleports, region crosses, etc. (about 1 script per 0.5 second).

Attachment scripts fail to load (Including HUDs) with a bunch of red text per script

10 Region Crosses (On system separate from the OpenSim system)
    * All failed/Was not able to complete this test; scripts refuse to work at all; nothing is logged except for either the exception or the normal 'Loading script...' log messages, although the ones that get logged normally don't seem to work either.
    * Strangely I am unable to detach the attachments with the non functioning scripts or region cross again (Avatar floats off infinitely if I try forcing a relog)
    
10 Region Crosses (On same system as OpenSim is running)
    * All failed; scripts refuse to work at all; nothing is logged except for either the exception or the normal 'Loading script...' log messages, although the ones that get logged normally don't seem to work either.
    * Did not have issues attaching/detaching the objects
    * Did not have issues region crossing

10 TP Lures Cross Region (Split in half, 5 per user, 2 users, one user on OpenSim system, one user on separate system):
    * All failed for both users; scripts refuse to work at all; nothing is logged except for either the exception or the normal 'Loading script...' log messages, although the ones that get logged normally don't seem to work either.
    * Did not have issues attaching/detaching the objects
    * Did not have issues region crossing


In the case of not being able to region cross, upon relog I found my self stuck in between the borders of the two regions never able to get myself out, and the avatar was bouncing all over the place between the two regions causing all sorts of log frenzy on the console. This did reveal to me a slightly different exception on the scripts (Posted below)

Script loading log:

2014-12-13 23:28:06,778 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script Script State Test (Avatar Center).Script State Test, item UUID f576a747-c983-40df-957a-71ccd2be910a, prim UUID f65f2bc3-cfa0-4248-8fc3-ffe8d027ebba @ <255.2374, 108.2633, 28.18715>.Test Region

Exception:

2014-12-13 23:27:09,493 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() in Test Region 2. Exception
System.Exception: Unable to delete old existing script-file before writing new. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\Users\admin\Desktop\ostest\opensim-289cf2e\bin\ScriptEngines\a6885bb3-049c-11de-ae71-0050c2490048\CommonCompiler_compiled_658b31c9-cf2e-4077-be45-0bc6cada00b8.dll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 472
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 686
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& assembly, Dictionary`2& linemap) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 389
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1358
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1053
   
Exception when stuck between the regions:

2014-12-13 23:39:37,742 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script Script State Test (Right Foot).Script State Test, item UUID 2d05191d-3ff1-46d8-9a46-d72d541c0a03, prim UUID ba9f7a63-5d48-4653-9cd7-15658aeb0bfc @ <255.8321, 98.53467, 28.1>.Test Region
2014-12-13 23:39:37,747 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() in Test Region. Exception
System.BadImageFormatException: Could not load file or assembly 'CommonCompiler_compiled_658b31c9-cf2e-4077-be45-0bc6cada00b8' or one of its dependencies. The module was expected to contain an assembly manifest.
File name: 'CommonCompiler_compiled_658b31c9-cf2e-4077-be45-0bc6cada00b8'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.AppDomain.Load(String assemblyString)
   at System.AppDomain.Load(String assemblyString)
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1329
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\admin\Desktop\ostest\opensim-289cf2e\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1053

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].


-------
Test 3
-------

ScriptStopStrategy = abort
AppDomainLoading = false
DeleteScriptsOnStartup = false

Initial load up there seemed to be no issues loading in world scripts.

Attachment scripts (including HUDs) load and function on initial login

10 Region Crosses (On system separate from the OpenSim system)
    * All good; scripts load and function, attachment states are preserved between region crosses
    
10 Region Crosses (On same system as OpenSim is running)
    * All good; scripts load and function, attachment states are preserved between region crosses
    
10 TP Lures Cross Region (Split in half, 5 per user, 2 users, one user on OpenSim system, one user on separate system):
    * All good for both users; scripts load and function, attachment states are preserved between region crosses


-------
Test 4
-------

ScriptStopStrategy = co-op
AppDomainLoading = false
DeleteScriptsOnStartup = false

Initial load up there seemed to be no issues loading in world scripts.

Attachment scripts (including HUDs) load and function on initial login

10 Region Crosses (On system separate from the OpenSim system)
    * All good; scripts load and function, attachment states are preserved between region crosses
    
10 Region Crosses (On same system as OpenSim is running)
    * All good; scripts load and function, attachment states are preserved between region crosses
    
10 TP Lures Cross Region (Split in half, 5 per user, 2 users, one user on OpenSim system, one user on separate system):
    * All good for both users; scripts load and function, attachment states are preserved between region crosses
(0027096)
Gavin Hird (reporter)
2014-12-14 00:52

I have also observed the failure rate is higher with increasing number of scripts.

Did you run your tests on mono or Windows?
(0027097)
mewtwo0641 (reporter)
2014-12-14 00:59

I'm testing on Windows with .NET, .NET 4.0 specifically
(0027098)
Gavin Hird (reporter)
2014-12-14 01:02

I started suspecting that as I don't seen any of the issues you have with AppDomainLoading = true on mono; it simply does not load any attached scripts at all. Thanks. :-)
(0027099)
mewtwo0641 (reporter)
2014-12-14 01:05

AppDomainLoading = true does seem to cause a higher failure rate in my case as well. Although with it set to false it is a lot better, but I do still see the issue with scripts not loading in some cases; but the scripts that fail to load are seemingly random and don't fail on all attachments.
(0027111)
Gavin Hird (reporter)
2014-12-16 00:55

I took the change r/25610 and compiled it with mono 3.2.7. It is now deployed to both the test and the prod regions. The HUD attachment seems to survive most crossings and teleports between regions on the same hosts. Avatar attached scripts are pretty much random still.

I'll report stability and issues as they come up.
(0027114)
justincc (administrator)
2014-12-16 16:44
edited on: 2014-12-17 10:02

Please try git master e901253. On Windows with

ScriptStopStrategy = abort
AppDomainLoading = true
DeleteScriptsOnStartup = false

I was able to reproduce a race condition which was a regression from recent ghosts branch changes. This would occasionally mean attachment scripts would not start on teleport. Curiously, I was not able to reproduce this on Linux with the same settings so it could simply be down to subtle timing differences (e.g. the window regions are in a different box on the same LAN from the viewer).

tbh, it would surprise me if AppDomainLoading interfered with that so quite possibly there are still other issues.

Region cross is done differently so appears not to suffer from the same issue.

To answer some other points

@mewtwo0641

1. The exception you list in test 1 is a timing issue where a script state backup occurs simultaneously with a removal. This needs to be addressed but it should be rare and I would not expect it to impact attachment scripts.

2. I have not got to looking at

ScriptStopStrategy = co-op
AppDomainLoading = true
DeleteScriptsOnStartup = false

yet.

@Gavin

1. I would be surprised if openjpeg-dotnet-x86_64.dll loading issues had anything to do with what you have been seeing. This message does pop-up occasionally and I believe is something to do with the module 'loading' mechanism (which is very dumb) but has nothing to do with anything else (physics DLLs are loaded by a separate mechanism).

2. The "Failed to verify user presence in the grid for April.Sis @grid.kitely.com, access denied to region East andwest." type messages appear if the destination simulator could not contact the home grid's user agent service to verify the agent movement [1]. So it's either a network issue or a problem on the Kitely end. This would not be an issue connected with attachment crossing.

[1] http://opensimulator.org/wiki/Hypergrid_Protocol#Protocol_Flow [^]

(0027115)
Gavin Hird (reporter)
2014-12-17 03:49
edited on: 2014-12-17 03:51

Definitely some progress my side:

With ScriptStopStrategy = abort and using Justin's test script I could rapidly teleport to or walk region borders in the same instance, teleport to the other instance and walk or teleport to the regions there, teleport back to the first instance, then back to the second, back to first, then do a HG teleport to Kitely, back to the first instance, then on to the second, walk a region border, teleport back to first instance, and the script loaded and counted up for every crossing or teleport. - Whew!

Detaching the attachment and re-attaching loaded the script to the state it was when detached, and it continued counting up on subsequent teleports. So most definitely progress.

For testing multiple attachments with 3 scripts communicating with each other, the result was a bit more mixed, but it might be one of the scripts needs to be rewritten. Logging showed they all loaded on each crossing or teleport, but one of them had to be reset to execute properly.

With ScriptStopStrategy = co-op the results are more mixed and I would have to test more. Preliminary it looks like the scripts don't reload in many cases on region crossings.

I'm gonna leave it like this with ScriptStopStrategy = abort on both test and prod and it will be interesting to see how / if this has any impact on how 64-bit Firestorm viewers behave on visiting. I believe the Firestorm LSL bridge that is attached to the avatar by default for that viewer might be the root cause of the issues; Justin's point (2) in the previous message. It has nothing to do with problems on Kitely because this has consistently happened with every FS 64-bit viewer visiting regardless of origin.

It's up and running on grid.xmir.org:8002 it anyone wants to test the 64-bit FS behavior. I don't have the ability to run any Windows version of a viewer at the moment after vmware goofed up. The Mac 64-bit version of FS is so unstable it is not worth testing anything with.

(0027116)
mewtwo0641 (reporter)
2014-12-17 13:38

Testing commit e901253 with the settings:

ScriptStopStrategy = abort
AppDomainLoading = true
DeleteScriptsOnStartup = false

I am still seeing compile failures but I did notice something new... It appears that if XEngine determines that it needs a recompile due to switch in ScriptStopStrategy that is when I see the failures. What I am wondering though is why XEngine is trying to recompile the script(s) if I haven't changed the ScriptStopStrategy between sessions and I am also starting with no script DLLs in my ScriptEngines directory (completely removed the directory to be sure)?

I have a good feeling about this one; Hopefully I'm looking in the right direction :)

Log excerpt:

2014-12-17 15:26:49,127 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script AO v5.8.23.* Radar v1.0.8, item UUID 5ce57769-a9a2-48e9-a329-e8f8dfea86e9, prim UUID 8c23a87d-10ef-447f-89c4-7eaf881f82f6 @ <251.0043, 123.2486, 28.02308>.Test Region

2014-12-17 15:26:49,650 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script AO v5.8.23.* Typing Override v1.0, item UUID a05a3077-568d-4ebe-8926-612db4e13f47, prim UUID 8c23a87d-10ef-447f-89c4-7eaf881f82f6 @ <251.0043, 123.2486, 28.02308>.Test Region

2014-12-17 15:26:49,668 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Recompiling script AO v5.8.23.* Typing Override v1.0, item UUID a05a3077-568d-4ebe-8926-612db4e13f47, prim UUID 8c23a87d-10ef-447f-89c4-7eaf881f82f6 @ <251.0043, 123.2486, 28.02308>.Test Region to switch it to abort termination. Will be active on next restart.

2014-12-17 15:26:49,877 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() in Test Region. Exception
System.Exception: Unable to delete old existing script-file before writing new. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\Users\admin\Desktop\ostest\opensim-e901253\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_c5bcb4af-3dff-4079-92c1-3f083a95afec.dll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\admin\Desktop\ostest\opensim-e901253\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 472
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\admin\Desktop\ostest\opensim-e901253\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 686
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& assembly, Dictionary`2& linemap) in c:\Users\admin\Desktop\ostest\opensim-e901253\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 389
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\admin\Desktop\ostest\opensim-e901253\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1350
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\admin\Desktop\ostest\opensim-e901253\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1045
(0027117)
Gavin Hird (reporter)
2014-12-17 13:57

I had one case of the same type messages when switching ScriptStopStrategy to co-op

2014-12-17 11:20:04,215 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Recompiling script AO.ZHAO II INTERFACE, item UUID 4f8d77f4-8dfb-4f0b-99b1-99da191b655e, prim UUID 6ddc4213-25f6-4be8-91db-021c89e4bf33 @ <246.909, 35.18608, 16.71862>.Murat to switch it to co-op termination. Will be active on next restart.

I was a bit puzzled over the "Will be active on next restart" as these massages came on simulator restart, so it indicated there had to be another restart before the newly set ScriptStopStrategy was active. Perhaps Justin could clarity?

It should be related to this change http://opensimulator.org/viewgit/?a=commit&p=opensim&h=d34ad345d5f6a8e57b4262e72b38e60e8623de30 [^]
(0027280)
justincc (administrator)
2015-01-19 15:36

Is this still happening with current master?
(0027298)
Gavin Hird (reporter)
2015-01-20 00:18

I wanted to test it because of the changes you made to Xengine over the last few days, but the simulators would not start up.

mono-seen went into almost idle state around the point it started to generate the first map tile (not sure exactly where this happened as there are not console message) and it never proceeded from there, so I had to revert. This was up to and including commit 1f04e1.
(0027301)
mewtwo0641 (reporter)
2015-01-20 07:11

Testing commit ac93ba9 yielded some strange and very odd results for me.

Script states seem to save properly on detach... But upon reattach some variables have lost their previous values and reverted back to the values initially declared in global scope. It doesn't seem to be consistent in which scripts and which variables do this and it seems to "switch around" when you add or remove scripts. There isn't (always) a full script reset when this happens but sometimes there is; verified by adding some debug text in state_entry().

I proceeded to test with variations on the ScriptStopStrategy and AppDomainLoading settings and the results are:

Test 1:
----------
ScriptStopStrategy = co-op
AppDomainLoading = false
DeleteScriptsOnStartup = false

Variables issue as reported in the beginning of this note occurs

Test 2:
----------
ScriptStopStrategy = abort
AppDomainLoading = false
DeleteScriptsOnStartup = false

Could not test; scripts give "Failure in DoOnRezScriptQueue()" errors and will not run

Test 3:
----------
ScriptStopStrategy = co-op
AppDomainLoading = true
DeleteScriptsOnStartup = false

Variables issue as reported in the beginning of this note occurs

Test 4:
----------
ScriptStopStrategy = abort
AppDomainLoading = true
DeleteScriptsOnStartup = false

Could not test; scripts give "Failure in DoOnRezScriptQueue()" errors and will not run

I'm unable to reliably reproduce this with a simple script. I can send you the attachment that is exhibiting this behaviour although I would rather not attach it to this Mantis since it has some code I use on SL for commercial products. May I send it to you via email Justin, and if so, what is a good address to send it to? Thanks :)
(0027310)
justincc (administrator)
2015-01-20 16:21

@Gavin - Could you take a thread dump with instructions at [1]? This will show if there is thread deadlock.

@mewtwo - What are the stack traces of the DoOnRezScriptQueue() failures? Even if they are the same as previously reported please could you repaste them.

Please send any script to justincc at justincc.org.

[1] http://opensimulator.org/wiki/Debugging#Thread_Dumps [^]
(0027315)
Gavin Hird (reporter)
2015-01-21 06:57

Not sure what you did overnight, but now it started up fine with the latest master, so no dump required... :-)

Running with async_call_method = Thread

Previous attempt failed both for Thread and SmartThreadPool

Finding that Thread is again stable on 3.12.0 after having to give it up for a few months. It is faster too, particularly on sim crossings.

I'll let you know if I see different / improved script behavior after the update.
(0027316)
Gavin Hird (reporter)
2015-01-21 07:11

hmmm, actually not gonna run this version in test or prod because it breaks both the OMC and the opensimsearch modules. The changes that Diva provided was quite intrusive in that respect I am afraid.
(0027319)
mewtwo0641 (reporter)
2015-01-21 14:28

Hi Justin, I sent an IAR to your email and here is the stack trace:

2015-01-21 16:24:30,674 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script AO v5.8.23.* Typing Override v1.0, item UUID da6cef01-0504-488a-9e25-ad797f38a15f, prim UUID 0501fd8b-1d3c-40d0-9c41-dd3e217fb874 @ <151.3793, 113.8925, 21.97288>.Test Region
2015-01-21 16:24:30,684 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Recompiling script AO v5.8.23.* Typing Override v1.0, item UUID da6cef01-0504-488a-9e25-ad797f38a15f, prim UUID 0501fd8b-1d3c-40d0-9c41-dd3e217fb874 @ <151.3793, 113.8925, 21.97288>.Test Region to switch it to co-op termination. Will be active on next restart.
2015-01-21 16:24:30,714 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() in Test Region. Exception
System.Exception: Unable to delete old existing script-file before writing new. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\Users\Admin\Desktop\ostest\opensim-abf1836\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_c5bcb4af-3dff-4079-92c1-3f083a95afec.dll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\Admin\Desktop\ostest\opensim-abf1836\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 472
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\Admin\Desktop\ostest\opensim-abf1836\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 686
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& assembly, Dictionary`2& linemap) in c:\Users\Admin\Desktop\ostest\opensim-abf1836\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 389
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\Admin\Desktop\ostest\opensim-abf1836\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1359
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\Admin\Desktop\ostest\opensim-abf1836\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1054
(0027326)
justincc (administrator)
2015-01-22 16:55
edited on: 2015-01-22 16:55

Thanks for the IAR, mewtwo. With the help of that, I identified a regression since commit f132f642 (2014-08-28) where the state of every second script in an object was not reloaded. I fixed this in d0a2ea0. Please test and tell me if that resolves the problem for you.

Also, in commit 840e440 OpenSimulator now sets the assembly file attributes to Normal before trying to delete it> I suspect that your system is somehow writing these with the read-only flag set or similar. One response is to force the user to sort it out but I think it's better to avoid that grief by setting it in code (OpenSimulator should always be able to access the file anyway as it created it). Please let me know if this changes what you are seeing when switching between co-op and abort.

(0027327)
mewtwo0641 (reporter)
2015-01-22 21:10

Testing on commit 840e440 issues seem to be much better when using ScriptStopStrategy = abort (I couldn't discern a difference between AppDomainLoading = true/false). Script states seem to function properly now, persists between reattach, region cross, and relog. I might venture to say abort is working reliably now but I'll need to test this for a bit before I can say for sure.

With co-op though I am still receiving the exceptions and am unable to do anything with scripts. I'm still curious though why XEngine is attempting to recompile scripts if I've previously deleted the ScriptEngines directory prior to switching to co-op and reloading OpenSim? From what I can see in the log, it first attempts to load a script, then tries to recompile it due to switch to co-op, then exception pops up. Should it be doing this if starting from a blank ScriptEngines directory?

It seems to be the same exception as previous but I've provided it just in case:

2015-01-22 22:34:19,919 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script AO v5.8.23.* Typing Override v1.0, item UUID 9dabd09a-9033-48c9-9713-10735bd670ee, prim UUID 92e950f8-1489-411e-a676-645a2d452593 @ <224.1419, 96.22991, 21.97332>.Test Region

2015-01-22 22:34:19,929 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Recompiling script AO v5.8.23.* Typing Override v1.0, item UUID 9dabd09a-9033-48c9-9713-10735bd670ee, prim UUID 92e950f8-1489-411e-a676-645a2d452593 @ <224.1419, 96.22991, 21.97332>.Test Region to switch it to co-op termination. Will be active on next restart.

2015-01-22 22:34:19,959 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() in Test Region. Exception
System.Exception: Unable to delete old existing script-file before writing new. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\Users\Admin\Desktop\ostest\opensim-840e440\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_c5bcb4af-3dff-4079-92c1-3f083a95afec.dll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\Admin\Desktop\ostest\opensim-840e440\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 475
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\Admin\Desktop\ostest\opensim-840e440\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 690
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& assembly, Dictionary`2& linemap) in c:\Users\Admin\Desktop\ostest\opensim-840e440\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 389
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\Admin\Desktop\ostest\opensim-840e440\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1359
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\Admin\Desktop\ostest\opensim-840e440\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1054
(0027328)
Gavin Hird (reporter)
2015-01-23 01:17

It probably works better with the latest changes, but I will have to test some more to be sure. I'll let it run in test couple days and see how it behaves.

I run with ScriptStopStrategy = abort AppDomainLoading = false and async_call_method = Thread on mono 3.12

I don't see any of the exceptions mewtwo sees when switching to co-op and back to abort, but for co-op the attached scripts really don't survive a teleport or region border walk, so sticking to abort.
(0027332)
justincc (administrator)
2015-01-23 17:02

If you are deleting the appropriate ScriptEngines directory then you should also see compile messages with the abort setting. Are you absolutely sure you are deleting the correct directory? All these messages are consistent with the script DLL still being present. The compilation is necessary because the 'abort' and 'co-op' script types have different class structures.

If you go to

C:\Users\Admin\Desktop\ostest\opensim-840e440\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_c5bcb4af-3dff-4079-92c1-3f083a95afec.dll

is there a file there? What timestamp does it have?
(0027334)
mewtwo0641 (reporter)
2015-01-23 19:09
edited on: 2015-01-23 19:11

I am deleting the entirety of C:\Users\Admin\Desktop\ostest\opensim-840e440\bin\ScriptEngines (including the ScriptEngines) directory itself. On abort I do see the initial compile messages (i.e the "Loading script..." messages) but not the recompile messages as seen on co-op.

I ended my testing yesterday on the abort setting so I repeated the deletion of the ScriptEngines directory, changed the setting to co-op, and loaded OpenSim up and logged in. After every thing loaded up I checked for the presence of C:\Users\Admin\Desktop\ostest\opensim-840e440\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_c5bcb4af-3dff-4079-92c1-3f083a95afec.dll; It is there and the timestamp is 1/23/2015 8:31 PM (Just moments ago as of this writing) however it seems non functional.

Starting in co-op; If I manually recompile my scripted attachments they start working and will persist working across OS restarts. If I shut OS down, delete the ScriptEngines directory, restart OS, log back in, the attachments that I previously recompiled still compile and work with out the exceptions pop up.

If I switch back to abort, delete the ScriptEngines directory, restart OS; then the attachment scripts that were working in co-op cease to work, exceptions are thrown, and I have to manually recompile them again to get them to work.

Switching back to co-op exhibits the same behavior as switching to abort; manual recompile, start working again, etc.

Checking the console I noticed that objects that are rezzed in world seemed to initially compile, run, and function just fine. Recompile also works on them as well. This is true on either setting. They seem to handle the switching back and forth just fine; no manual recompiling necessary.

(0027335)
Gavin Hird (reporter)
2015-01-24 01:51

It seems the scripts will consistently load now but I am getting sharing violation on script states. The scripts will execute despite the error condition.


10:47:04 - [SCRIPT INSTANCE]: Not starting script AO with OSSL (id 57d4a4f5-8e98-46a6-9b46-9f0e03e50d3f) in part AO for OpenSim (id 76f0ba64-ed78-4b50-b17d-06aeb54f617d) in object AO for OpenSim in Telegraph Bay. Unable to load script state file ScriptEngines/03369b6e-a9b1-4cd8-a728-ff3048c95b71/57d4a4f5-8e98-46a6-9b46-9f0e03e50d3f.state. XML is . Exception Sharing violation on path /Users/opensim/build/xmirhg-next/bin/ScriptEngines/03369b6e-a9b1-4cd8-a728-ff3048c95b71/57d4a4f5-8e98-46a6-9b46-9f0e03e50d3f.state at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.Open (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.Load (System.AppDomain dom, System.Reflection.Assembly scriptAssembly, System.String dataPath, StateSource stateSource) [0x00000] in <filename unknown>:0
(0027336)
Gavin Hird (reporter)
2015-01-24 01:57

In commit 8d724e you save the initial .state file, but it seems like it is trying to load it later too causing the sharing violation?
(0027338)
Gavin Hird (reporter)
2015-01-24 02:04

Just to clarify the events leading to a sharing violation on the .state file.

It happens when you cross back or teleport back to a region where the attachment script has executed before, so on first region crossing everything looks fine.
(0027343)
Gavin Hird (reporter)
2015-01-25 02:19

I wonder if this code broke some other script behavior. I have a door script I have used a lot which is "automatic", meaning it supposedly relies on script states to operate correctly in any orientation.

It now opens and closes the doors in not completely erratic positions, but most of the time it overruns the programmed opening of PI/4 with typically 30 deg in either direction. Randomly it gets the closed or open position right. Script reset or re rez does not make it behave any better.

I have attached the script so you can check if you see the same behavior.
(0027350)
justincc (administrator)
2015-01-26 16:09

@mewtwo0641 please try git master 1bed3af or latest which fixes a problem I encountered on my own Linux system with AppDomain = true. This adjusts the way DLLs are loaded so that regions load their own DLLs which fixed my problem. It may have an impact on yours. Also, could you remind me how many regions you running, on what OS and with what [XEngine] config (apart from ScriptStopStrategy). Thanks.

@gavin - The state is not saved in that commit. On object rez, the scene code reinserts script variables and these are serialized as a temporary .state file for the engine to load later. However, this file does not need to remain and a bunch of later .state saves for attachments are unnecessary. I made this change to try and reduce unnecessary .state files to reduce any side effects from files that shouldn't be there and to make debugging easier.

The subsequent sharing violation is puzzling. I've looked through the code many times and nothing should be holding on to a .state file. Please try git master 1bed3af though I suspect this won't help. You can also try a new temporary git branch called "sedebug" which only currently runs up to Jan 15th in order to identify where/if problems have been recently introduced.

Rather, it prevents attachments from unnecessarily saving .state files (state is saved in the asset) and deletes
(0027357)
Gavin Hird (reporter)
2015-01-27 01:46

After taking 1bed3af the sharing violation still logs the following on re-entry to a region where the attached script has been running before:


0:21:31 - [SCRIPT INSTANCE]: Not starting script AO with OSSL (id 4f350eab-f6c8-42dc-aa4c-d6c0e10e0456) in part AO for OpenSim (id 5fb036c3-121a-476d-9e3c-8658e012dac9) in object AO for OpenSim in Ghul. Unable to load script state file ScriptEngines/d6ab91df-345c-46ef-881a-db85efa5cd2c/4f350eab-f6c8-42dc-aa4c-d6c0e10e0456.state. XML is . Exception Sharing violation on path /Users/opensim/build/xmirhg-next/bin-not quite there/ScriptEngines/d6ab91df-345c-46ef-881a-db85efa5cd2c/4f350eab-f6c8-42dc-aa4c-d6c0e10e0456.state at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.Open (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.Load (IScript script, System.Threading.EventWaitHandle coopSleepHandle, System.String assemblyPath, System.String dataPath, StateSource stateSource, Boolean coopTermination) [0x00000] in <filename unknown>:0


If I use the OS X lsof command to list all the open files by mono-sgen it does not list any .state files but a large number of script dlls so I suppose the sharing violation does not happen at file system level. I uploaded a file with the list - it might be of some interest.

From the log above it seems the problem is at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.Load?

I will try and rerun with Appdomain = true and see what the yields.
(0027358)
Gavin Hird (reporter)
2015-01-27 02:04

Oh, with AppDomain = true it was a complete mess on system startup with loads of messages like below being logged even with DeleteScriptsOnStartup = true


0:51:14 - [SCRIPT INSTANCE]: Not starting script PoseBall .2 (id 97bfdfda-48b9-41b4-9ce4-c89375ec44e8) in part PILLOW & POSE BALL SIT Unisex BASIC (id 1f44736d-811d-40c0-aa9b-082751059d42) in object PILLOW & POSE BALL SIT Unisex BASIC in Ghul. Unable to load script state file ScriptEngines/244cbf66-d024-4d4a-8b69-ff3048c95b72/97bfdfda-48b9-41b4-9ce4-c89375ec44e8.state. XML is . Exception Sharing violation on path /Users/noklebye/build/xmirhg-next/bin-not quite there/ScriptEngines/244cbf66-d024-4d4a-8b69-ff3048c95b72/97bfdfda-48b9-41b4-9ce4-c89375ec44e8.state at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.Open (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.Load (IScript script, System.Threading.EventWaitHandle coopSleepHandle, System.String assemblyPath, System.String dataPath, StateSource stateSource, Boolean coopTermination) [0x00000] in <filename unknown>:0



I deleted the content of ScriptEngines manually, and no sharing violations on startup, but a few messages like this:

10:56:51 - [XEngine]: Failure in DoOnRezScriptQueue() for item f1747551-cd4d-46e4-a4d3-0775c6b75f4e in Ghul. Continuing. Exception System.IO.FileNotFoundException: Could not load file or assembly 'CommonCompiler_compiled_f0d28aa6-c74b-49aa-a314-f7cf58cc1668' or one of its dependencies. The system cannot find the file specified.
File name: 'CommonCompiler_compiled_f0d28aa6-c74b-49aa-a314-f7cf58cc1668'
  at (wrapper xdomain-invoke) System.AppDomain:CreateInstanceAndUnwrap (string,string,bool,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo,object[])
  at (wrapper remoting-invoke-with-check) System.AppDomain:CreateInstanceAndUnwrap (string,string,bool,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo,object[])
  at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript (System.Object[] parms) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue (System.Object dummy) [0x00000] in <filename unknown>:0




On first avatar login it logs, so is the sharing violation on script state saved in the attachment?:

10:59:36 - [SCRIPT INSTANCE]: Not starting script AO with OSSL (id 11a2ddf3-6b4a-4307-9a54-1b6255a44a36) in part AO for OpenSim (id bed69f34-29ff-432a-81c0-36cc14747128) in object AO for OpenSim in Ghul. Unable to load script state file ScriptEngines/d6ab91df-345c-46ef-881a-db85efa5cd2c/11a2ddf3-6b4a-4307-9a54-1b6255a44a36.state. XML is . Exception Sharing violation on path /Users/noklebye/build/xmirhg-next/bin-not quite there/ScriptEngines/d6ab91df-345c-46ef-881a-db85efa5cd2c/11a2ddf3-6b4a-4307-9a54-1b6255a44a36.state at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.Open (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance.Load (IScript script, System.Threading.EventWaitHandle coopSleepHandle, System.String assemblyPath, System.String dataPath, StateSource stateSource, Boolean coopTermination) [0x00000] in <filename unknown>:0


It produced the same type of sharing violations on region crossings after the initial attachment script loads + the regions crossings were horrendously slow, so not running AppDomain = true :-))
(0027359)
Gavin Hird (reporter)
2015-01-27 02:18
edited on: 2015-01-27 02:21

In commit 8d724e9 you say "For scripts in attachments, don't save .state files apart from the initial one as these are ignored since .state is saved in the attachment's asset.

So the question is; does the sharing violation on attached scripts occur on the .state in the attachment and not on a file system .state file as it is no longer written, unless there is a stub of code still trying to access the initial .state file you say you save?

Also, should not the algorithm be that the initial .state file is deleted as soon as the .state is established and safely stored in the attachment?

If the initial .state file is needed for region consistency or whatever, should it not at least be deleted when the avatar has left the region, because it will in any case be invalid on return as anything could have happened to the script state in the meantime?

(0027360)
Gavin Hird (reporter)
2015-01-27 04:46
edited on: 2015-01-27 04:48

As for taking the sedebug fork, I downloaded it and diffed the XEngine.cs file with the "good" I am using for a working build. The one I use has datestamp 21 jan 2015 10.12, meaning I have taken your changes up to and including commit d9bfc7 which was submitted at 21 jan 2015 00.54.

When diffing the files my version of XEngine.cs has a number of changes related to state.

The working build does not produce any sharing violations and the door script works as expected, but not all avatar attached scripts load. This was fixed in d0a2ea after which they consistently loaded on all teleports and region crossings.

(0027361)
mewtwo0641 (reporter)
2015-01-27 08:15

Testing commit 13ba2f2 with AppDomainLoading = true I am not seeing any difference in the behavior of the script loading; They behave in the same manner as my previous note described. I also tested with AppDomainLoading = false and no difference there as well.

The DoOnRezScriptQueue() exceptions are slightly different now (different line numbers as far as I can tell):

2015-01-27 10:00:12,007 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Loading script AO v5.8.23.* Typing Override v1.0, item UUID 4376f38a-933c-4e0a-9f83-e36a9e347dfc, prim UUID ae815bd5-962d-4637-aa66-32822da7d690 @ <221.9765, 87.74454, 21.98984>.Test Region

2015-01-27 10:00:12,017 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Recompiling script AO v5.8.23.* Typing Override v1.0, item UUID 4376f38a-933c-4e0a-9f83-e36a9e347dfc, prim UUID ae815bd5-962d-4637-aa66-32822da7d690 @ <221.9765, 87.74454, 21.98984>.Test Region to switch it to abort termination. New termination will be active on next restart.

2015-01-27 10:00:12,047 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() for item 4376f38a-933c-4e0a-9f83-e36a9e347dfc in Test Region. Continuing. Exception
System.Exception: Unable to delete old existing script-file before writing new. Compile aborted: System.UnauthorizedAccessException: Access to the path 'C:\Users\Admin\Desktop\ostest\opensim-13ba2f2\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_874614d7-ab52-40ac-a4a1-a0442bc38450.dll' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.Delete(String path)
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\Admin\Desktop\ostest\opensim-13ba2f2\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 481
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.CompileFromDotNetText(String Script, enumCompileType lang, String asset, String assembly) in c:\Users\Admin\Desktop\ostest\opensim-13ba2f2\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 696
   at OpenSim.Region.ScriptEngine.Shared.CodeTools.Compiler.PerformScriptCompile(String source, String asset, UUID ownerUUID, Boolean alwaysRecompile, String& assembly, Dictionary`2& linemap) in c:\Users\Admin\Desktop\ostest\opensim-13ba2f2\OpenSim\Region\ScriptEngine\Shared\CodeTools\Compiler.cs:line 395
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\Admin\Desktop\ostest\opensim-13ba2f2\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1464
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\Admin\Desktop\ostest\opensim-13ba2f2\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1056
(0027364)
justincc (administrator)
2015-01-27 11:54

@mewtwo - So, I finally reproduced this failure on my Windows box. Basically, the act of loading the script DLL on Windows, even if the script instantiation was unsuccessful, locks the file against deletion, hence the error.

This makes automatic recompilation increasingly tricky, especially if AppDomainLoading = false. So in git master e0a3440 I have switched to a manual approach. If the script engine detects that the [XEngine] ScriptStopStrategy does not match that in the compiled DLL it will warn in the log but continue with the old strategy for that script as before.

However, it will now not attempt recompilation itself. Instead, the log message will ask the user to set DeleteScriptsOnStartup = true for one session (or one could manually delete the DLLs as you have been doing). I think this is a reasonable as an operator will only have to do this once and then only if they have DeleteScriptsOnStartup = false.
(0027365)
Gavin Hird (reporter)
2015-01-27 12:09

Justin, maybe this a Duh-moment, but would not the sharing violation messages simply indicate that the script was never unloaded from the region in a multi-region simulator with AppDomainLoading=false ?

In that case it tries to take back the script, fails and kicks it off again as a new instance, as the scripts are restarted on the crossing (very evident in the messages from the AO I use for testing.)
(0027366)
justincc (administrator)
2015-01-27 12:14

Do you have any anti-virus enabled, Gavin? If so, please test with it temporarily disabled. This is to elminate any possibility that such a program is delaying file writes [1]. Please also remind me of your operating system details.

[1] http://stackoverflow.com/questions/6350224/does-filestream-dispose-close-the-file-immediately [^]
(0027367)
Gavin Hird (reporter)
2015-01-27 12:16

You gotta be kidding!

There are no know Mac viruses in the wild, and has not been since the mid 1990-ties :-)))))
(0027368)
Gavin Hird (reporter)
2015-01-27 12:23

Test server is running OS X 10.9.5 with OS X Server 3.2 and mono 3.12
Prod server is running OS X 10.10.1 with OS X Server 4.2 and mono 3.12

They both behave identical, and there is no delay in writing files. I can assure you about that.
(0027374)
justincc (administrator)
2015-01-27 13:51

@Gavin - Please could you attach an attachment script that produces these state errors.

Also, let us not get confused by LSL states (default, etc.) and the entire script state (current script variables, event queue, etc.). These are separate concepts.
(0027375)
Gavin Hird (reporter)
2015-01-27 14:06

Uploaded 2 files:

AO with OSSL.lsl - which you will see from the logs above causing the sharing violation
It needs the config file * OS AO Config which must refer to animations in the content of the attachment

Actually any script will cause the violation, but this is a good example of a more complicated one.
(0027377)
justincc (administrator)
2015-01-27 16:59

I don't encounter any problems with that attachment when moving between regions under Mono 3.2.8 on Ubuntu 14.10, though it looks like I would have to populate it with animations too for it to do much?
(0027378)
mewtwo0641 (reporter)
2015-01-28 00:19

Testing commit cf0087e I see the occasional warning:

2015-01-28 01:41:38,151 WARN - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: At least one existing compiled script DLL in Test Region has abort as ScriptStopStrategy whereas config setting is True.
Continuing with script compiled strategy but to remove this message please set [XEngine] DeleteScriptsOnStartup = true for one simulator session to remove old script DLLs (script state will not be lost).

From what I can tell, setting DeleteScriptsOnStartup = true on next start up or deleting the script DLLs from the ScriptEngines directory does not seem to make the warning go away on next start-up; still have to manually recompile the scripts that are giving the warning(s) to make it go away. However, attachment scripts now otherwise work fine even with the warnings, and no more exceptions are being logged.

Overall the issue has vastly improved for me. I am unsure of this as far as a Linux/Mono environment is concerned though.

Also... I forgot to answer your system setup questions in your previous note :) I'm running these tests on Windows 7 x64 on a 2 region standalone config (non hypergrid).

Results of config show XEngine:

[XEngine]
  Enabled = true
  MinThreads = 2
  MaxThreads = 100
  IdleTimeout = 60
  Priority = BelowNormal
  MaxScriptEventQueue = 300
  ThreadStackSize = 600000
  AppDomainLoading = false
  ScriptStopStrategy = co-op
  WriteScriptSourceToDebugFile = false
  DefaultCompileLanguage = lsl
  AllowedCompilers = lsl
  CompileWithDebugInformation = true
  AllowMODFunctions = false
  AllowOSFunctions = true
  AllowLightShareFunctions = false
  OSFunctionThreatLevel = Severe
  SaveInterval = 120
  MaintenanceInterval = 10
  EventLimit = 30
  KillTimedOutScripts = false
  WaitForEventCompletionOnScriptStop = 1000
  ScriptDelayFactor = 0.0
  ScriptDistanceLimitFactor = 1.0
  SensorMaxRange = 96.0
  SensorMaxResults = 16
  DeleteScriptsOnStartup = false
(0027379)
Gavin Hird (reporter)
2015-01-28 02:01
edited on: 2015-01-28 02:02

I took the latest changes without it making much of a difference.

The door script I supplied earlier is also failing still, so something has been introduced after jan 21...

I found this article in the xamarin bugzilla, and as you can see it is a know problem. https://bugzilla.xamarin.com/show_bug.cgi?id=21981 [^]

They point to thread safe code to resolve these types of issues it seems. (comment 5)

....

Animations for the script?

If you HG to grid.xmir.org:8002:Murat and go into the main store there is an Zhao based AO with all the animations you need to populate an attachment and make the script work.

(0027414)
justincc (administrator)
2015-01-29 11:25

@Gavin - The door script code works perfectly fine for me on current master. If you want to help, please do a git bisect and tell me the exact revision where it started going wrong.

I am well aware of multi-threaded issues in OpenSimulator believe me. However, from my analysis I can't see any instance where a thread may be reading state at the same time a delete is attempted. However, clearly something is going wrong. Please try git master b4e955d which does resolve a recently introduced race condition.

I have also updated a separate sedebug branch in Git which leaves out the major scripting related changes in master, apart from the one that fixes the issue with only loading every second state in an attachment. Please test that if the current master does not help.

Also, I want to be clear that we are testing OpenSimulator branch code here. I cannot address bugs where commits have been combined or cherry picked privately.
(0027419)
Gavin Hird (reporter)
2015-01-29 11:50
edited on: 2015-01-29 11:53

it is not possible for me to run exactly the master because there is a bug that does not load the PGSQL driver in the initial startup phase. I guess Diva's changes did not even think about that.

I am going to file a Mantis about it, but I discovered it yesterday, and there has been other things to attend to. There were other issues with her changes that I sorted out before that having to do with the OMC module.

My test code has taken all of your changes up till the last commit. It has only take a tiny amount of the code Diva submitted sufficiently to make it compile and sort out issues where some interface code was moved to other locations in the tree. I have also taken all the bulletsim patches.

The code that runs without errors up is up to and including commit d9bfc7 which was submitted at 21 jan 2015 00.54.

The race condition fix that is in b4e955 is included in the code I have tested, and that also fails.

After commit d9bfc7 it starts going wrong, and I might take commit by commit to try and pin down exactly where it goes wrong. My hunch is that commit d0a2ea is where it fails.

I have also tested it without llvm for mono, but it behaves the same. I have not tried reverting mono back to an earlier version, but I doubt there is an issue there.

If testing commit by commit does not yield anything I might just freeze it at the code level that runs, and see if I rather can contribute something to the remaining PGSQL adapter issues I am experiencing.

(0027425)
Gavin Hird (reporter)
2015-01-30 01:04

Justin, commit b4e955 produces the same result as before.

Since the sharing violation always occur when an avatar walks/tp back to a region where the script has previously executed, is not the problem that it tries to access something that has a lock on it from the previous time the avatar was in that region?

Setting AppDomainLoading = false causes per Opensim.ini

"However, setting this to false will also prevent script DLLs from being unloaded from memory if the script is deleted."

So is there something that tells the runtime to try and access the previous still in-memory script even if it has been deleted?
(0027426)
Gavin Hird (reporter)
2015-01-30 01:36
edited on: 2015-01-30 04:19

Using this as a record for the testing I will do today:

---------
As of opensim-d9bfc71 everything is OK.

Door scripts behave as expected. There are no sharing violations, all scripts survive region crossings, meaning they continue to execute and don't restart.

---------
As of opensim-2d574c3

Number of compiler warnings are up from 71 to 113
Door scripts behave as expected. There are no sharing violations, most scripts survive region crossings, meaning they continue to execute and don't restart. The most scripts survive is possibly present in the previous build and has been reported before. It happens randomly.

--------
Adding commits 185e70 + d0a2ea

Number of compiler warnings are still 113
I had to reset scripts that did not execute at the end of the previous session and build. Isn't that a bit odd given it started up clean-slate with empty assets cache and scriptengines folders? Is script state stored in the database?

Door scripts behave as expected. There are no sharing violations, most scripts survive region crossings, meaning they continue to execute and don't restart. The most scripts survive is possibly present in the previous build and has been reported before. It happens randomly.

Not all attached scripts execute after login even if they were running on the last login.


-------
Adding commit 840e44

Number of compiler warnings are still 113
Status is same as for above

-------
Adding commits 3289aa + 1bed3a + 13ba2f

Number of compiler warnings are up to 115
There is an improvement in that all attached scripts runs on login, but I have had to reset one (random) on region crossings, and not always.
Door scripts behave as expected and still no sharing violation.

-------
Adding commits e0a344 + cf0087 + b4e955 meaning we are at master sans some of Diva's changes

Number of compiler warnings are up to 113
All attached scripts runs on login, but I have had to reset one (random) on region crossings, and not always.
Door scripts behave as expected and still no sharing violation.


Meaning it is building and running without any of the sharing violation issues I have seen before. Which is GOOD!

... but I am whipping myself to figure out why it failed before because the code changes have been added exactly the same way as this time around, the starting point was the same and before it has been built I have often cleaned to make sure there was a fresh build. The only difference is that this time around I have always killed the assetcache directory at the destination for each build, while before I let it survive to speed up startup/login. So there might be something there...

(0027427)
Gavin Hird (reporter)
2015-01-30 05:31

I think I have found why it failed.

I did not take commit 185e70 because the description says "On the GridService, the central simulator features: ensure that the map tile url ends with '/' because the viewer is dumb and just appends to it." so it seemed insignificant in the context of testing Justin's changes to the script engine.

However, diffing the GridService.cs from this commit with the previous version there are some substantial changes.
(0027612)
mewtwo0641 (reporter)
2015-02-25 20:10

Testing commit 412dd7 I'm starting to see the DoOnRezScriptQueue() exceptions again but they look to be a couple of slightly different ones.

It was actually slightly more complicated to trigger this though.

Steps taken:
1. Region cross then logout
2. Log back in then region cross again, then logout
3. Log back in and exception occurs on some but not all the scripts.

(NOTE: I was also at the same time testing rapid succession of region crosses back and forth to see if scripts would consistently load prior to the steps taken)


Exceptions:
------------

2015-02-25 21:57:43,862 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() for item 8c975491-862f-44e5-ab80-f0033ee65cfe in Test Region 2. Continuing. Exception
System.IO.FileLoadException: Could not load file or assembly 'CommonCompiler_compiled_5d7c6999-d616-409c-a6aa-6470847ffa3a' or one of its dependencies. Access is denied.
File name: 'CommonCompiler_compiled_5d7c6999-d616-409c-a6aa-6470847ffa3a' ---> System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Users\Admin\Desktop\ostest\opensim\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_5d7c6999-d616-409c-a6aa-6470847ffa3a.dll' [^] or one of its dependencies. Access is denied.
File name: 'file:///C:\Users\Admin\Desktop\ostest\opensim\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_5d7c6999-d616-409c-a6aa-6470847ffa3a.dll' [^]
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.OnAssemblyResolve(Object sender, ResolveEventArgs args) in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1834
   at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1361
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1062



2015-02-25 22:01:04,229 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Failure in DoOnRezScriptQueue() for item d96e0cba-798c-4b86-b29d-bebbb478678c in Test Region 2. Continuing. Exception
System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Users\Admin\Desktop\ostest\opensim\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_5d7c6999-d616-409c-a6aa-6470847ffa3a.dll' [^] or one of its dependencies. Access is denied.
File name: 'file:///C:\Users\Admin\Desktop\ostest\opensim\bin\ScriptEngines\bf4a1f26-f6f2-4871-bbf5-3620e8a39872\CommonCompiler_compiled_5d7c6999-d616-409c-a6aa-6470847ffa3a.dll' [^]
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms) in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1361
   at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScriptQueue(Object dummy) in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\Region\ScriptEngine\XEngine\XEngine.cs:line 1062

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
(0027614)
Gavin Hird (reporter)
2015-02-26 00:28

Tested your sequence, but not able to repro. Scripts are behaving as expected. I am on mono 3.12.
(0027615)
mewtwo0641 (reporter)
2015-02-26 00:38

@Gavin: It may be that it's a random occurrence and may not necessarily happen in 3 steps although I am on Windows using .NET so there may be some differences there?
(0027657)
mewtwo0641 (reporter)
2015-02-28 23:05
edited on: 2015-02-28 23:14

I just realized that there is a possible common ground with state resets. If something triggers a script to changes an attachment prim property (such as color, texture, alpha, scale etc) then the reset rate seems to be high.

I believe I have come up with steps to trigger attachment state reset at a fairly consistent rate:

//Test Script

integer count = 0;
integer toggle = FALSE;

default
{
    state_entry()
    {
        llSay(0, "Script started or has been reset.");
        llSetText("Count: " + (string)count, <1,1,1>, 1.0);
        llSetColor(<1,1,1>, ALL_SIDES);
        llSetTexture(TEXTURE_DEFAULT, ALL_SIDES);
    }
    
    touch_start(integer num)
    {
        count++;
        llSetText("Count: " + (string)count, <1,1,1>, 1.0);
        
        llSetColor(<llFrand(1), llFrand(1), llFrand(1)>, ALL_SIDES);
        
        if(toggle)
            llSetTexture(TEXTURE_BLANK, ALL_SIDES);
            
        else if(!toggle)
            llSetTexture(TEXTURE_DEFAULT, ALL_SIDES);
            
        toggle = !toggle;
    }
}

1. Put on an attachment with this script in it; a simple cube will do.
2. Click the cube a few times to trigger changes in the prim
3. Region cross between two regions a few times and check to see if script reset; if not, continue:
4. Log off OS and shut OS down
5. Restart OS and log back in
6. Without clicking the box; region cross again a few times. By this point the script has most likely reset.
    * If it resets: The box will say it reset, Count will read 0, color will be changed back to white, and the texture back to the default wood texture.

The steps were performed with ScriptStopStrategy = co-op and without ever detaching the object during testing.

This may also be triggered with simply logging out and back in again and then region crossing a few times (skipping the OS shutdown and restart part); but I have had better luck triggering it after OS has been shut down and restarted for some reason.

Some observations about ScriptStopStrategy:

With ScriptStopStrategy = co-op the reset seems to be a bit harder to trigger. I find I need to go through all 6 steps to get it to reset.

With ScriptStopStrategy = abort the reset seems to be a lot easier to trigger. I managed to get it to reset at Step 3; after a region cross and then a region cross back to the original region where I first attached it.

I'm not sure if it matters or not but I started with co-op in my tests then switched to abort (recompiled the script manually) and tested again.

(0027658)
Gavin Hird (reporter)
2015-02-28 23:41

I have not tested your script yet, but I managed to trigger the same type of exceptions yesterday when making outfits and rapidly attaching and detaching scripted attachments. No sim crossings involved.
(0028020)
mewtwo0641 (reporter)
2015-04-16 12:42

This is still an issue for me at commit 67d4e447

Using the test script from note 0007278:0027657 I was able to trigger it at step 3 this time using co-op; but it also was triggered consistently by continuing to follow all the steps to reproduce.
(0028145)
Gavin Hird (reporter)
2015-04-30 07:44

Starting up my test regions on the latest OSG build with mono 4.0.0 – which has a lot of Microsoft .NET code in it, reintroduced the sim crossing issues that was fixed for mono earlier.

I reverted back to 3.12.1 and it works again. So much for Microsoft fixed code, hehe!
(0029132)
mewtwo0641 (reporter)
2015-08-09 18:55

Still seeing this issue as of Release 0.8.1.1 and latest master. I decided to sit down and poke at the settings some more. This time I tested with DeleteScriptsOnStartup = false (I normally set this to true for the script load speed gains) and ScriptStopStrategy = abort. I noticed that in this configuration it the issue seems to be a lot better (Have not been able to trigger a script reset... Yet.)

Kind of a bummer though since with DeleteScriptsOnStartup = false it takes significantly longer for regions with a lot of scripts to start up. It also takes a while longer for attachment scripts to start up if entering a region previously unvisited since the last OpenSim start up.

I am not completely 100% sure that DeleteScriptsOnStartup = false is the setting that is helping with the issue (It makes sense that it might be though if not a lot of people are seeing the issue since this is the default un-configured setting for it); This is just my observations though... Maybe there is something to it :) If it actually is the cause (or a contributing factor) of the issue then it essentially makes DeleteScriptsOnStartup = true useless for grid setups where attachment script states are critical.
(0029133)
melanie (administrator)
2015-08-09 19:09

DeleteScriptsOnStartup was put in with debugging in mind. Using it carries serious functionality penalties and therefore is not recommended. The files the region deletes are the files containing the state of scripts. That messes up state handling. Again, this option is for debugging and the paranoid only, it carries severe penalties in functionality and features.
Do not use it unless you need to. Region start up time is not a sufficient reason to throw away the script engine's state handling ability.
(0029134)
mewtwo0641 (reporter)
2015-08-09 19:20

Ahh OK. I wasn't aware that this was strictly a debug setting. I actually hadn't had any issues (that I could tell) with it up until the change with how OpenSim handles the script stop strategy. I'll leave it false now and test further to see if the script state issue remains subsided.
(0029135)
Gavin Hird (reporter)
2015-08-10 01:37

Thanks @melanie for that information. I think a certain amount of handwringing on this mantis could have been avoided of that was known.

I think we got the impression from Justin that setting it to true would be better. It is also the default in OpenSim.ini.example

Perhaps the text in Opensim.ini needs to reflect this is a debug setting and change the default?

;# {DeleteScriptsOnStartup} {} {Delete previously compiled script DLLs on startup?} (true false) true
    ;; Controls whether previously compiled scripts DLLs are deleted on sim restart. If you set this to false
    ;; then startup will be considerably faster since scripts won't need to be recompiled. However, then it becomes your responsibility to delete the
    ;; compiled scripts if you're recompiling OpenSim from source code and internal interfaces used
    ;; by scripts have changed.
(0029136)
mewtwo0641 (reporter)
2015-08-10 03:08
edited on: 2015-08-10 03:15

@melanie: I initially had the impression had the impression in general that it would be better to have it set false just because of how the comment for the setting was worded in OpenSim.ini since it seemed to imply a good upside and the only downside it mentioned was maybe having to delete your script dll's and let OpenSim re-compile them if there was significant change to how scripts are handled in OpenSim (which for this Mantis I've done till my face turned blue admittedly lol)

As per your previous note; testing further with this set to its default of true however still maintains that the script states do not tend to reset now except under one condition that I've seen so far: There is about a 10 - 15 second window when you cross over to a new region (one you haven't been in yet since starting OpenSim) before attachment scripts are started again (This script load delay isn't seen when the setting is set to false; they load almost immediately upon region cross). If you attempt to cross to a new region (or back to the previous region) before the attachment scripts load, all the scripts will end up reset, and their states also reset to initial settings. If you wait until scripts are loaded before attempting another cross there is no reset of the scripts.

(0034236)
BillBlight (developer)
2019-02-05 12:25

Old Issues, closed, can be reopened if they still exist

- Issue History
Date Modified Username Field Change
2014-07-22 04:32 mewtwo0641 New Issue
2014-07-22 11:03 justincc Note Added: 0026557
2014-07-22 11:03 justincc Assigned To => justincc
2014-07-22 11:03 justincc Status new => assigned
2014-07-22 11:46 mewtwo0641 Note Added: 0026559
2014-08-11 03:44 mewtwo0641 Note Added: 0026729
2014-08-11 03:59 mewtwo0641 Note Edited: 0026729 View Revisions
2014-11-28 22:40 mewtwo0641 Note Added: 0027039
2014-11-29 07:09 justincc Note Added: 0027043
2014-11-29 16:05 mewtwo0641 Note Added: 0027044
2014-11-29 16:05 mewtwo0641 Note Edited: 0027044 View Revisions
2014-11-29 16:06 mewtwo0641 Note Edited: 0027044 View Revisions
2014-11-29 16:14 mewtwo0641 Note Edited: 0027044 View Revisions
2014-11-30 00:36 AliciaRaven Note Added: 0027045
2014-12-03 13:26 justincc Note Added: 0027055
2014-12-03 13:26 justincc Status assigned => feedback
2014-12-03 13:48 justincc Relationship added related to 0007319
2014-12-03 13:53 justincc Note Added: 0027057
2014-12-03 21:29 mewtwo0641 Note Added: 0027061
2014-12-03 21:29 mewtwo0641 Status feedback => assigned
2014-12-04 13:08 justincc Note Added: 0027064
2014-12-04 13:09 justincc Note Added: 0027065
2014-12-04 22:59 mewtwo0641 Note Added: 0027067
2014-12-09 03:45 Gavin Hird Note Added: 0027079
2014-12-09 03:45 Gavin Hird Note Added: 0027080
2014-12-09 16:43 justincc Note Added: 0027081
2014-12-09 16:43 justincc Status assigned => feedback
2014-12-10 00:51 Gavin Hird Note Added: 0027082
2014-12-10 00:52 Gavin Hird Note Edited: 0027082 View Revisions
2014-12-10 00:53 Gavin Hird Note Edited: 0027082 View Revisions
2014-12-10 08:42 mewtwo0641 Note Added: 0027083
2014-12-10 08:42 mewtwo0641 Status feedback => assigned
2014-12-10 16:43 justincc Note Added: 0027084
2014-12-11 01:25 Gavin Hird Note Added: 0027085
2014-12-11 06:06 Gavin Hird Note Edited: 0027085 View Revisions
2014-12-11 08:06 Gavin Hird Note Added: 0027088
2014-12-11 09:03 Gavin Hird Note Edited: 0027088 View Revisions
2014-12-11 13:54 Gavin Hird Note Added: 0027089
2014-12-11 16:08 zadark Note Added: 0027090
2014-12-11 23:46 Gavin Hird Note Added: 0027091
2014-12-11 23:46 Gavin Hird Note Edited: 0027091 View Revisions
2014-12-12 09:29 Gavin Hird Note Added: 0027093
2014-12-12 13:25 Gavin Hird Note Added: 0027094
2014-12-13 22:55 mewtwo0641 Note Added: 0027095
2014-12-13 22:55 mewtwo0641 File Added: statetest.iar
2014-12-14 00:52 Gavin Hird Note Added: 0027096
2014-12-14 00:59 mewtwo0641 Note Added: 0027097
2014-12-14 01:02 Gavin Hird Note Added: 0027098
2014-12-14 01:05 mewtwo0641 Note Added: 0027099
2014-12-16 00:55 Gavin Hird Note Added: 0027111
2014-12-16 16:44 justincc Note Added: 0027114
2014-12-17 03:49 Gavin Hird Note Added: 0027115
2014-12-17 03:50 Gavin Hird Note Edited: 0027115 View Revisions
2014-12-17 03:51 Gavin Hird Note Edited: 0027115 View Revisions
2014-12-17 10:02 justincc Note Edited: 0027114 View Revisions
2014-12-17 13:38 mewtwo0641 Note Added: 0027116
2014-12-17 13:57 Gavin Hird Note Added: 0027117
2015-01-19 15:36 justincc Note Added: 0027280
2015-01-20 00:18 Gavin Hird Note Added: 0027298
2015-01-20 07:11 mewtwo0641 Note Added: 0027301
2015-01-20 16:21 justincc Note Added: 0027310
2015-01-21 06:57 Gavin Hird Note Added: 0027315
2015-01-21 07:11 Gavin Hird Note Added: 0027316
2015-01-21 14:28 mewtwo0641 Note Added: 0027319
2015-01-22 16:55 justincc Note Added: 0027326
2015-01-22 16:55 justincc Note Edited: 0027326 View Revisions
2015-01-22 21:10 mewtwo0641 Note Added: 0027327
2015-01-23 01:17 Gavin Hird Note Added: 0027328
2015-01-23 17:02 justincc Note Added: 0027332
2015-01-23 19:09 mewtwo0641 Note Added: 0027334
2015-01-23 19:11 mewtwo0641 Note Edited: 0027334 View Revisions
2015-01-24 01:51 Gavin Hird Note Added: 0027335
2015-01-24 01:57 Gavin Hird Note Added: 0027336
2015-01-24 02:04 Gavin Hird Note Added: 0027338
2015-01-25 02:19 Gavin Hird Note Added: 0027343
2015-01-25 02:20 Gavin Hird File Added: failing doorscript.lsl
2015-01-26 16:09 justincc Note Added: 0027350
2015-01-27 01:46 Gavin Hird Note Added: 0027357
2015-01-27 01:46 Gavin Hird File Added: mono-sgen open files.txt
2015-01-27 02:04 Gavin Hird Note Added: 0027358
2015-01-27 02:18 Gavin Hird Note Added: 0027359
2015-01-27 02:20 Gavin Hird Note Edited: 0027359 View Revisions
2015-01-27 02:21 Gavin Hird Note Edited: 0027359 View Revisions
2015-01-27 04:46 Gavin Hird Note Added: 0027360
2015-01-27 04:48 Gavin Hird Note Edited: 0027360 View Revisions
2015-01-27 08:15 mewtwo0641 Note Added: 0027361
2015-01-27 11:54 justincc Note Added: 0027364
2015-01-27 12:09 Gavin Hird Note Added: 0027365
2015-01-27 12:14 justincc Note Added: 0027366
2015-01-27 12:16 Gavin Hird Note Added: 0027367
2015-01-27 12:23 Gavin Hird Note Added: 0027368
2015-01-27 13:51 justincc Note Added: 0027374
2015-01-27 14:02 Gavin Hird File Added: AO with OSSL.lsl
2015-01-27 14:02 Gavin Hird File Added: * OS AO Config
2015-01-27 14:06 Gavin Hird Note Added: 0027375
2015-01-27 16:59 justincc Note Added: 0027377
2015-01-28 00:19 mewtwo0641 Note Added: 0027378
2015-01-28 02:01 Gavin Hird Note Added: 0027379
2015-01-28 02:02 Gavin Hird Note Edited: 0027379 View Revisions
2015-01-29 11:25 justincc Note Added: 0027414
2015-01-29 11:50 Gavin Hird Note Added: 0027419
2015-01-29 11:51 Gavin Hird Note Edited: 0027419 View Revisions
2015-01-29 11:53 Gavin Hird Note Edited: 0027419 View Revisions
2015-01-30 01:04 Gavin Hird Note Added: 0027425
2015-01-30 01:36 Gavin Hird Note Added: 0027426
2015-01-30 01:49 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 02:07 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 02:08 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 02:35 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 02:39 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 02:50 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 02:59 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 03:04 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 03:30 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 03:41 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 04:19 Gavin Hird Note Edited: 0027426 View Revisions
2015-01-30 05:31 Gavin Hird Note Added: 0027427
2015-02-25 20:10 mewtwo0641 Note Added: 0027612
2015-02-26 00:28 Gavin Hird Note Added: 0027614
2015-02-26 00:38 mewtwo0641 Note Added: 0027615
2015-02-28 23:05 mewtwo0641 Note Added: 0027657
2015-02-28 23:14 mewtwo0641 Note Edited: 0027657 View Revisions
2015-02-28 23:41 Gavin Hird Note Added: 0027658
2015-04-16 12:42 mewtwo0641 Note Added: 0028020
2015-04-30 07:44 Gavin Hird Note Added: 0028145
2015-08-09 18:55 mewtwo0641 Note Added: 0029132
2015-08-09 19:09 melanie Note Added: 0029133
2015-08-09 19:20 mewtwo0641 Note Added: 0029134
2015-08-10 01:37 Gavin Hird Note Added: 0029135
2015-08-10 03:08 mewtwo0641 Note Added: 0029136
2015-08-10 03:14 mewtwo0641 Note Edited: 0029136 View Revisions
2015-08-10 03:15 mewtwo0641 Note Edited: 0029136 View Revisions
2019-02-05 12:11 BillBlight Assigned To justincc => administrator
2019-02-05 12:25 BillBlight Note Added: 0034236
2019-02-05 12:25 BillBlight Status assigned => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker