Chat log from the meeting on 2022-07-12

 [11:04] Ubit Umarov: ahh andrew's cat did remember the meeting [11:04] Andrew Hellershanks: I'm on a phone call, [11:04] Misterblue Waves: we got up to 48c last summer -- the plants didn't do well [11:05] Ubit Umarov: Ohh ok we wil type softly [11:05] Ubit Umarov: yeah but 48 here is rare.. like 45years rare [11:06] Ubit Umarov: well about opensim, again not much [11:06] Vincent.Sylvester @hg.zetaworlds.com:8002: Two changes last week, another cleanup on libomv and a fix for fsassets not properly using its folders. I rolled back mono to 122 on Friday and am now awaiting what that does to regions. So far haven't heard of further crashes, but will have to leave it for a bit to make sure. FS has a new beta out with support for the grid status rss now supported in OpenSim. [11:06] Misterblue Waves: same for us -- getting above 100F is rare [11:06] Ubit Umarov: i did a little commit related to a fsassets flag [11:07] Ubit Umarov: or option, actually not that good... [11:07] Ubit Umarov: to run several instances of fsassets will just not work [11:07] Ubit Umarov: unless you do have a network disk as storage [11:07] Andrew Hellershanks: .0 [11:08] Ubit Umarov: fs just writes things on a disk.. has no idea abotu multi instances [11:08] Ubit Umarov: the metadata does depend on mysql doing the multi server thing, that seems not that good either [11:09] Ubit Umarov: so, not that usefaull that option.. [11:09] Ubit Umarov: or useful [11:09] Ubit Umarov: hmmwhat was the other? libomv? [11:10] Ubit Umarov: hmm guess minor thing also [11:10] Vincent.Sylvester @hg.zetaworlds.com:8002: Is the fsassets thing mostly file locking or what's the problem there? [11:11] Ubit Umarov: ah yes.. was using a generic c# exceptionto report bad string on parse [11:12] Kayaker Magic: One problem with multiple fsassets is there is small chance of two of them generating the same UUID at the same time. [11:12] Ubit Umarov: changed to FormatException to match ehat .net guid does [11:12] Ubit Umarov: fs does not gen uuids [11:12] Kayaker Magic: OK, then multiple Robusts. [11:13] Ubit Umarov: the thing with multiple robusts is that you do need a shared storage [11:13] Ubit Umarov: fs as no provision for any on that [11:14] Ubit Umarov: each instance will just read and write from own disk [11:14] Ubit Umarov: something that will not work that all, unless it is a special shared network disk [11:14] Ubit Umarov: or you are running all that on same machine, not that useful either [11:15] Ubit Umarov: on that fsassets is worse than just normal assets, that at least depend on mysql/maria only [11:16] Ubit Umarov: fsassets multiinstances that did work, where done by spliting as per a few bits of the asset UUID [11:16] Ubit Umarov: thing we still have code for that, not sure [11:17] Ubit Umarov: ofc does not work that well with the "random" guid type we are using [11:17] Ubit Umarov: not that random [11:18] Ubit Umarov: this is why osgrid is running on own proprietary assets.. [11:18] Ubit Umarov: was using a variant of fsassets, that actually also did smoke if using more than one instance [11:18] Ubit Umarov: now using somthign else [11:19] Ubit Umarov: ofc no issue if you have a nove NAS machine and a 100Gbit network ;) [11:19] Ubit Umarov: a nice NAS.. [11:19] Vincent.Sylvester @hg.zetaworlds.com:8002: Given the simplicity of what that has to do in the end it's easier to write in something else that supports proper file locking out of the box I guess. Ideally directly in C to get the most speed [11:20] Ubit Umarov: file lock actually seems to fail on linux [11:21] Ubit Umarov: that fsassets code also runs several instances on a single robust... [11:21] Vincent.Sylvester @hg.zetaworlds.com:8002: It's not built-in so whatever language or code needs to handle it on its own [11:21] Ubit Umarov: one as assets main instance, others as slaves that connect to it [11:21] Misterblue Waves: the GUIDs are not random enough or is the content hashing not being unique? [11:22] Ubit Umarov: there was still a lock issue, bc several maintnance threads could be fired in that case.. [11:22] Ubit Umarov: that should be ok now... [11:23] Ubit Umarov: my comments where only when ppl did tried to run several assets service instances on dif machines.. [11:23] Ubit Umarov: as i said, that is a fail, unless they are usiign a network shared disk [11:24] Vincent.Sylvester @hg.zetaworlds.com:8002: Or something that emulates the behavior of one I suppose [11:24] Ubit Umarov: yeah [11:24] Ubit Umarov: then ofc one needs to compare the added latency to the latency of just one... [11:25] Ubit Umarov: on this, region and viewers caches are fundamental [11:26] Ubit Umarov: in nomral cases, only region caches should be used [11:26] Ubit Umarov: in cases on users that spend most time on a few regions, even viewer caches [11:27] Vincent.Sylvester @hg.zetaworlds.com:8002: From what I log on disk io I rarely see more than a few mbit when assets are moving, so it's not a big load until you get to hundreds of gigabytes [11:27] Vincent.Sylvester @hg.zetaworlds.com:8002: Bigger problem is backing up millions of files in thousands of directories [11:28] Ubit Umarov: well many ppl even totally forgot that fsassets uses mysql and disk files [11:28] Ubit Umarov: and lost the disk files :( [11:28] Kayaker Magic: oops! I hate it when tht happens! [11:28] Misterblue Waves: would something like S3 be better -- not AWS but one of the compatible interfaces of modern hash buckets [11:28] Ubit Umarov: sadly happened a few times [11:29] Vincent.Sylvester @hg.zetaworlds.com:8002: It should probably use even more sql on that, things like notecards saving their description to the assets means even if they are empty the hash always differs so it creates tons of empty notecard assets [11:30] Vincent.Sylvester @hg.zetaworlds.com:8002: AWS is overpriced junk you'll be paying more running that than just some decent bare metal and when they lose your data you are sol [11:30] Ubit Umarov: several large grids are running fine [11:30] Ubit Umarov: some mb with own solutions [11:31] Ubit Umarov: well i nev4er did like fsassets [11:31] Misterblue Waves: agree on AWS, but a lot of other places are providing S3 compatible storage (DigitalOcean, ...) [11:31] Ubit Umarov: sure mysql is possible a bad choice [11:31] Vincent.Sylvester @hg.zetaworlds.com:8002: Toying with the idea of crawling all notecards for empty lines and just nuking them or setting their hash to a single one instead [11:31] Ubit Umarov: inworldz did used other db engine, if i remember [11:32] Ubit Umarov: a more direct key,value thing [11:32] Ubit Umarov: guess non sql even [11:32] Vincent.Sylvester @hg.zetaworlds.com:8002: For a few things a few data structures in OpenSim things like mongodb would work, potentially even better than sql, but then you have another db system to maintain and deal with also, so a trade off [11:33] Vincent.Sylvester @hg.zetaworlds.com:8002: For avatar data and groups it might be faster, but means whole new db connector etc [11:33] Vincent.Sylvester @hg.zetaworlds.com:8002: Already a pain to maintain the ones we have [11:33] Ubit Umarov: well our dbs are a disaster [11:33] Vincent.Sylvester @hg.zetaworlds.com:8002: Some, not all [11:33] Ubit Umarov: guess a good example on how to make things as slow as possible [11:33] Ubit Umarov: region db ... duhh what a waste [11:34] Vincent.Sylvester @hg.zetaworlds.com:8002: Inventory might also benefit from document style db rather than large table [11:34] Vincent.Sylvester @hg.zetaworlds.com:8002: Anything bound to a single user or uuid I suppose [11:35] Vincent.Sylvester @hg.zetaworlds.com:8002: Useless for assets, want a proper table with keys and indexes there [11:35] Vincent.Sylvester @hg.zetaworlds.com:8002: Come to think of it I meant to check if different charset and collation had a performance impact on groups as it did with griduser data [11:35] Vincent.Sylvester @hg.zetaworlds.com:8002: Seems mixing those things is a fail on mariadb [11:36] Vincent.Sylvester @hg.zetaworlds.com:8002: Needs all uniform ones to speed up inner join [11:36] Ubit Umarov: seems mariadb is a fail on anything large [11:36] Ubit Umarov: even mysql [11:36] Kayaker Magic: I have some stuff set up at OpenSim Fest, and one of the regions stopped working. Scripts froze, I was not allowed to compile my own scripts or rez more objects. Shelenn told me it was caused by an OpenSim bug that Ubit already knew about. Which bug was that? [11:36] Ubit Umarov: only a few forks of mysql do hold, i was told [11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: Development has picked up pace lately so maybe that will be change positively in future [11:37] Misterblue Waves: I've used MariaDB successfully for small deployments (that's what I use for my Docker based regions) [11:37] Ubit Umarov: no idea [11:37] Ubit Umarov: they are running a thing called opensim-ngc [11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: If you overload a region with too many scene updates the queue for that fills up and it just stop processing completely [11:37] Ubit Umarov: that ofc does claim to be better than core [11:38] Vincent.Sylvester @hg.zetaworlds.com:8002: Not much can be done on that though [11:38] Ubit Umarov: so you do need to ask them :) [11:38] Vincent.Sylvester @hg.zetaworlds.com:8002: scene update queue has a fixed size, once full it's full and just dies [11:38] Ubit Umarov: i do not remember such issue.. [11:39] Vincent.Sylvester @hg.zetaworlds.com:8002: Seen that a few times, always told to remove some heavy scripts, timers and things, no more issues [11:39] Ubit Umarov: what scene updates?? [11:39] Vincent.Sylvester @hg.zetaworlds.com:8002: The ones generated by scripts, avatars etc. Forgot the actual name, but it's not scripts that die it's scene updates [11:39] Ubit Umarov: you are imagining thngs again vicent :p [11:39] Ubit Umarov: events queue? [11:40] Ubit Umarov: those self limit.. do not stop things [11:40] Ubit Umarov: ofc many ways a script can still freze a region.. [11:40] Vincent.Sylvester @hg.zetaworlds.com:8002: You can test if its that by typing commands into the console, if it fails with some error no matter what you type you know its borked [11:41] Vincent.Sylvester @hg.zetaworlds.com:8002: Reducing script load usually resolves this [11:41] Kayaker Magic: After they "fixed it" I was able to add more scripted items. [11:41] Ubit Umarov: well i don't remember that issue on core.. possible.. but don't remember [11:42] Ubit Umarov: i mean a specific one [11:42] Vincent.Sylvester @hg.zetaworlds.com:8002: You have to load the regions pretty heavily for that to happen most stay well under that or write scripts that don't cause so many updates to be fired [11:43] Ubit Umarov: well i do remember regions totally down [11:43] Vincent.Sylvester @hg.zetaworlds.com:8002: Doesn't happen often enough to really warrant looking into it that much. Easier to just write better scripts [11:43] Ubit Umarov: jst because a simple LED scritp [11:43] Ubit Umarov: we jsut could not tp in the region or nearby ones :) [11:44] Ubit Umarov: similar script will do the same still [11:44] Vincent.Sylvester @hg.zetaworlds.com:8002: I have only seen this on a region with like 90k prims and over 8000 scripts [11:44] Ubit Umarov: change glow or color etc on a prim is a full update [11:44] Ubit Umarov: that region had like 1000 LEDs :) [11:45] Ubit Umarov: we all see that issue at xmas.. with some bad xmas lights [11:45] Vincent.Sylvester @hg.zetaworlds.com:8002: Could probably find the queue and figure out what the max size is and add some debug to alert when it gets close to full [11:45] Vincent.Sylvester @hg.zetaworlds.com:8002: Just never looked where that is [11:46] Ubit Umarov: there you are again with a queue... waht queue?? [11:46] Ubit Umarov: there are several [11:46] Kayaker Magic: After seeing some bad xmas light scripts, I build some ZERO LAG ones that use texture animation to blink. [11:46] Ubit Umarov: and none a issue [11:47] Ubit Umarov: yes blink needs to be done with clever tex anim [11:48] Vincent.Sylvester @hg.zetaworlds.com:8002: When a script moves a prim, that gets added to a queue as a scene update, usually gets processed almost immediately unless you have a lot of those updates happening [11:48] Ubit Umarov: wel it is a old issue, also at SL [11:48] Vincent.Sylvester @hg.zetaworlds.com:8002: Not sure where in code that is I never looked much into it [11:48] Ubit Umarov: lludp updates queue per user?? [11:49] Ubit Umarov: well those are heavy.. but a major issue currently [11:49] Vincent.Sylvester @hg.zetaworlds.com:8002: Possible, all I know is it's something that fills up and then overloads stopping update processing because you can still walk around and chat, but scripts appear as if they are stopped [11:49] Ubit Umarov: they do not fill up [11:49] Ubit Umarov: ma fill your machine ram :p [11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: As it doesn't happen immediately and only when more scripts and people are on the region tells me it's some processing for the updates done on scene [11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: Usually just writing better scripts and reducing the load fixes this [11:50] Ubit Umarov: scripts need to be good [11:51] Vincent.Sylvester @hg.zetaworlds.com:8002: You can have a region with hundreds of thousands of prims and one script that kills it [11:51] Ubit Umarov: a single bad script can stop a region, as i said [11:51] Ubit Umarov: .. still... [11:51] Ubit Umarov: specially using some OSSL there should had never need added [11:51] Ubit Umarov: details.. [11:51] Kayaker Magic: A bad programmer can write FORTRAN in any language;. [11:52] Vincent.Sylvester @hg.zetaworlds.com:8002: Writing stuff for light changing and animations can be insanely heavy [11:52] Andrew Hellershanks: Hello, everyone. Finally off the phone call. :P Last part of it was giving the person directions to the place they were driving to as it turned out they were headed north when they should have been going south. [11:52] Vincent.Sylvester @hg.zetaworlds.com:8002: Still writing on danceballs and other lighting stuff I check what cpu does, sometimes a poorly written line gives me 100% cpu heh [11:53] Andrew Hellershanks: Vincent, doesn't take much to do that. [11:53] Vincent.Sylvester @hg.zetaworlds.com:8002: In the same tune if you can figure out what it does internally you can change a color 60 times a second without much load [11:54] Andrew Hellershanks: I have been helping people do some X and Y engine compatibility tests this past week. Their scripts trigger an exception in YEngine during compilation. Not sure what is causing that. Runtime I was getting an out of heap exception. [11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: lol ouch [11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: That takes some serious effort [11:55] Andrew Hellershanks: Looking at the OS code I found some ini settings to change heap sizes and that solved the heap issue. [11:55] Andrew Hellershanks: Still no idea about the compile time error. [11:55] Ubit Umarov: ohh really? [11:55] Ubit Umarov: mab next time RTFM ? [11:55] Ubit Umarov: maybe.. :p [11:56] Andrew Hellershanks: Ubit, yes. There are ini settings for setting stack and heap sizes in YEngine but they aren't listed in the ini file. [11:56] Vincent.Sylvester @hg.zetaworlds.com:8002: The compiler errors are usually very descriptive unless you broke something internally [11:56] Vincent.Sylvester @hg.zetaworlds.com:8002: You ideally shouldn't need to adjust heap at all [11:56] Kayaker Magic: If people use old OpenSim.ini settings for YEngine heap sizes, that can cause the problem you describe [11:56] Andrew Hellershanks: The error is the generic one -> NullReferenceException: Object reference not set to an instance of an object [11:57] Vincent.Sylvester @hg.zetaworlds.com:8002: Also I think latest release should have proper heap size calc in there, was a bug on that before [11:57] Ubit Umarov: http://opensimulator.org/wiki/YEngine [11:57] Ubit Umarov: "Memory heap and stack use control" very clear there [11:57] Vincent.Sylvester @hg.zetaworlds.com:8002: Always test with release or master code [11:58] Vincent.Sylvester @hg.zetaworlds.com:8002: I'd be curious to see that script and figure out where it generates all that heap [11:58] Andrew Hellershanks: YEngine will accept ini settings to set the stack and heap sizes but those settings are not in the existing ini files. [11:58] Vincent.Sylvester @hg.zetaworlds.com:8002: Haven't had a single script hit those limits yet [11:58] Vincent.Sylvester @hg.zetaworlds.com:8002: Well outside of testing for it specifically [11:58] Andrew Hellershanks: I'm not as concerned about the heap issue as much as I am about the exception when trying to compile the code. [11:59] Vincent.Sylvester @hg.zetaworlds.com:8002: Yeah that you kinda need to have a debugger connected to it I guess [11:59] Andrew Hellershanks: I found the YEngine has some settings for debug modes but the debug ones that are supposed to save the original script and save the IL file don't seem to work (or the file was saved somewhere other than I expected). [12:00] Ubit Umarov: they are at opensimdefaults.ini [12:00] Vincent.Sylvester @hg.zetaworlds.com:8002: Mean like visual studio to see directly in code where it barfs [12:00] Andrew Hellershanks: Vincent, Yes, that is what I was planning on doing. I might add some extra debug code as the backtrace tells me where it dies. [12:01] Vincent.Sylvester @hg.zetaworlds.com:8002: If you get anywhere with it feel free to mantis it so others can test as well, more brains the better to figure illusive things like that out [12:01] Ubit Umarov: yeah to read the IL will help you a lot :p [12:01] Andrew Hellershanks: I didn't write the two scripts in question. One is almost 4600 lines line. [12:02] Andrew Hellershanks: Ubit, I know. I was just hoping to find something that might help narrow down what is triggering the exception. [12:02] Andrew Hellershanks: A debugger or adding in extra debug messages may be my best option. [12:02] Ubit Umarov: x had no control of mem usage [12:03] Vincent.Sylvester @hg.zetaworlds.com:8002: If they are that long maybe what you are seeing is the fail on the heap allocation itself [12:03] Ubit Umarov: so it is normal that some scripts will try to eat all memory [12:03] Andrew Hellershanks: It had a 256k stack size setting in the ini but that was about it. [12:03] Vincent.Sylvester @hg.zetaworlds.com:8002: You almost have as bad a luck with bugs as I do lol [12:03] Ubit Umarov: Y controls stack and heap [12:04] Andrew Hellershanks: Vincent, that's what I'm thinking. it would have been nicer if it had said that in the first place and not just no ref to object. [12:04] Andrew Hellershanks: Vincent, only when it isn't my code. [12:04] Ubit Umarov: and yes in same cases ppl need to make those very large to run old X scripts [12:04] Selby.Evans @grid.kitely.com:8002: byr all [12:04] Andrew Hellershanks: This is part of a compatibility test to see what changes (if any) need to be made to the script so it works in YEngine. [12:05] Ubit Umarov: well assuming it did fail by controled Yengined stach overflow [12:05] Kayaker Magic: Bye Selby! [12:05] Ubit Umarov: cya Selby.Evans [12:05] Andrew Hellershanks: Selby, ok. Thanks for being here. [12:05] Ubit Umarov: it may had crash because native stack overflow :) [12:05] Vincent.Sylvester @hg.zetaworlds.com:8002: Well my best guess is that with scripts that long there is probably something in there not up to proper LSL standard and it's barfing on that. I have seen a lot of old scripts written under X perform poorly because they used bad list appends and such things. Fixing those and writing in proper LSL standard you get performance and less issues with memory consumption [12:05] Ubit Umarov: well when ubode does that, it is very visible :) [12:06] Vincent.Sylvester @hg.zetaworlds.com:8002: Rewriting script that large not so easy of course, might be better to just lint it a bit [12:07] Andrew Hellershanks: Vincent, That is part of my question. Is it something bad in the script or something in the YEngine code. I wasn't expecting an exception during what I think is just the compilation phase. [12:08] Ubit Umarov: well it depends on what you call exception [12:08] Ubit Umarov: "exception" can be a lot of things [12:08] Vincent.Sylvester @hg.zetaworlds.com:8002: Line 2547 probably and maybe the list append on line 574, need to see the script to determine that really [12:08] Andrew Hellershanks: What I call an exception is that thing that is reported by YEngine as ...Exception in the error message. [12:09] Ubit Umarov: it usually tells more [12:10] Vincent.Sylvester @hg.zetaworlds.com:8002: Well null reference errors are something you see on parts of code that try to get properties or iterate for example. It tells you a reference is missing which means whatever code is trying to interact with an object the object has changed format for example. That's just one way that can happen though. Given it runs out of heap perhaps internal structures like list sizes are exceeded when it allocates space for them so the reference to each item falls apart [12:10] Andrew Hellershanks: Sure. There is a backtrace. [12:11] Ubit Umarov: it is on on compile that it has nothing to do with those script memory settings [12:11] Ubit Umarov: if it is.. [12:11] Kayaker Magic: Andrew: I just looked up YEngine stack and heap sizes, the recommended values are 2048 and 1024, This changed at one point, it sounds like you had the old smaller values. [12:11] Ubit Umarov: Grrr [12:11] Ubit Umarov: he now said it is on compile.. [12:12] Ubit Umarov: just need to read the damm message [12:12] Andrew Hellershanks: The first error I ran in to was the Exception that appeared when I told the viewer I wanted to compile the scripts. Once I received an updated version of the code I just got the HeapException error. [12:12] Vincent.Sylvester @hg.zetaworlds.com:8002: ... we be poking in the dark here without script and logs basically, if you cannot figure it out this week just mantis it :) [12:12] Ubit Umarov: sure, some are still criptic.. [12:13] Object: Script running [12:13] Andrew Hellershanks: Now that I have doubled the heap size I will try compiling the scripts once again on the original version of code in case the no ref to object exception is also heap related. [12:14] Ubit Umarov: no ref to object is a more severl thing [12:14] Ubit Umarov: just get the full error [12:15] Andrew Hellershanks: I would discuss it in the IRC channel but I still can't reach the channel since the nationwide Internet outage on Friday. [12:15] Ubit Umarov: oh? [12:16] Vincent.Sylvester @hg.zetaworlds.com:8002: Just mantis it if you get nowhere with it this week or something, better get more brains looking over it and I been cleaning up the mantis as best as I can so feel free to add lol [12:16] Andrew Hellershanks: yea. A major Internet and cell phone provider experience an outage on Friday starting around 5am. I didn't get Internet back until around 10:30pm. Some people were expected to be out as many as 3 days. [12:16] Vincent.Sylvester @hg.zetaworlds.com:8002: BGP again they say [12:16] Ubit Umarov: wel in this case, a mantis without the scropt to repo is a problem [12:16] Ubit Umarov: script.. [12:17] Ubit Umarov: and provide the script, specially a large one, can be a issue [12:17] Andrew Hellershanks: Vincent, sure. My notes say that even with the larger heap size it still throws an exception when I ask for the scripts to be compiled. [12:17] Vincent.Sylvester @hg.zetaworlds.com:8002: Script and logs, well steps to reproduce and logs in most cases [12:17] Ubit Umarov: ohh [12:17] Andrew Hellershanks: If I am allowed to provide the script I will mark it private. [12:18] Ubit Umarov: it may just be a missing script [12:18] Kayaker Magic: Finding a small script to reproduce an error is often difficult. [12:18] Ubit Umarov: that may give a null ref error [12:18] Andrew Hellershanks: To repro the problem just right click object, more, more, scripts, compile (Mono). [12:18] Ubit Umarov: yeap sounds like a missing script asset [12:18] Ubit Umarov: read the error message [12:19] Andrew Hellershanks: at OpenSim.Region.ScriptEngine.Yengine.XMRInstance.PostEvent (OpenSim.Region.ScriptEngine.Shared.EventParams evt) [0x0001f] in XMRInstRun.cs:69 [12:19] Andrew Hellershanks: at (wrapper remoting-invoke-with-check) OpenSim.Region.ScriptEngine.Yengine.XMRInstance.PostEvent(OpenSim.Region.ScriptEngine.Shared.EventParams) [12:19] Andrew Hellershanks: etc... [12:19] Vincent.Sylvester @hg.zetaworlds.com:8002: Most obvious problem with that used to be too small packet size on the database to store the asset [12:19] Andrew Hellershanks: I'll see if I can learn anything more about it and mantis. [12:20] Ubit Umarov: on a post event? [12:20] Ubit Umarov: odd [12:20] Andrew Hellershanks: hm... now that is an intereting thought, Vincent. [12:20] Ubit Umarov: well can't guess [12:20] Ubit Umarov: and that is running it [12:20] Andrew Hellershanks: The text of the script as sitting on my computer is 211772 bytes long. [12:21] Ubit Umarov: or starting [12:21] Vincent.Sylvester @hg.zetaworlds.com:8002: Used to have like 16mb of max packet size on older mysql versions, since then it was increased default value should be much higher now like 1G even on some flavors [12:22] Ubit Umarov: well scripts used to have a limit of 64k bytes on sources [12:22] Vincent.Sylvester @hg.zetaworlds.com:8002: Was big fail on HG assets back then [12:22] Andrew Hellershanks: Ubit, It does look a bit more like it is an issue after the script starts to run. Something to do with llMessageLinked. I am thinking of adding extra code in to LSL_Api.cs to print out what the function receives. [12:23] Ubit Umarov: but that seems is needs a more detailed debug [12:23] Andrew Hellershanks: The Engine is dying and not leaving any useful information about what line of the script it was trying to run at the time it die.d [12:23] Ubit Umarov: that is not on lsl_api [12:23] Andrew Hellershanks: I was hoping one of the Debug settings would help. [12:23] Andrew Hellershanks: llMessageLinked is not in LSL_Api.cs?? [12:23] Ubit Umarov: and it is that one ? [12:24] Andrew Hellershanks: That function is mentioned further down in the backtrace. [12:24] Ubit Umarov: well i can't guess what you didn't told :p [12:24] Vincent.Sylvester @hg.zetaworlds.com:8002: It's like a fox hunt with an invisible fox [12:25] Ubit Umarov: nahh just very bad problem report [12:25] Ubit Umarov: :p [12:26] Vincent.Sylvester @hg.zetaworlds.com:8002: If you don't get anywhere debugging it mantis it, if they don't want to share set it to private so at least Ubit can have a look at it [12:26] Andrew Hellershanks: Yes, the script would have to be marked private. [12:27] Vincent.Sylvester @hg.zetaworlds.com:8002: Dirty secrets lol [12:27] Andrew Hellershanks: Proprietary code that is part of a larger system. [12:28] Vincent.Sylvester @hg.zetaworlds.com:8002: Hopefully not some multi pose animation thing, haven't seen a single one written properly yet they are all massively over complicated and poorly written to the point they consume a ton of resources [12:28] Vincent.Sylvester @hg.zetaworlds.com:8002: Some just don't even work anymore under Y [12:29] Vincent.Sylvester @hg.zetaworlds.com:8002: Or use ossl functions that need to be set true [12:29] Andrew Hellershanks: Vincent, interesting that they some don't work in Y (assuming it isn't just perms). [12:29] Vincent.Sylvester @hg.zetaworlds.com:8002: They fail on the lists they generate or on typecasts [12:29] Andrew Hellershanks: Probably not too surprising though. [12:30] Andrew Hellershanks: oh, right. Probably ones that use the "memory saving trick". [12:30] Vincent.Sylvester @hg.zetaworlds.com:8002: Also prime suspects for that "scripts stopped working" thing I mentioned earlier [12:30] Vincent.Sylvester @hg.zetaworlds.com:8002: Oh yeah when that happens the things like llSetText also don't update so there is a hint at which queue that is [12:31] Vincent.Sylvester @hg.zetaworlds.com:8002: But yeah if you don't get anywhere just make a private mantis ticket for it [12:32] Ubit Umarov: yeha guess that will take some deeper debug [12:33] Misterblue Waves: gotta run... take care all [12:33] Ubit Umarov: cya [12:34] Andrew Hellershanks: As long as I can still access the mantis. Don't know why I still can't access the IRC channels. [12:34] Ubit Umarov: but that means there are more scripts [12:34] Andrew Hellershanks: ok, Misterblue. [12:34] Andrew Hellershanks: We are now half past the hour. Unless there is any last minute items I will wrap up the meeting. [12:34] Ubit Umarov: on the linkset [12:35] Ubit Umarov: so guess the entire object will be needed to debug [12:37] Andrew Hellershanks: ok, I'll call this one done and dusted. :)  Thank you all for coming. See you again next week.