[11:01] Andrew Hellershanks: Hello, everyone.
[11:01] Motoko.Karu @login.digiworldz.com:8002: hihi
[11:02] Ubit Umarov: ohh are those pigs in andrew's hair?
[11:02] Ubit Umarov: :)
[11:02] Ubit Umarov: ( private joke related to irc )
[11:02] Vincent.Sylvester @hg.zetaworlds.com:8002: hehe
[11:03] Vincent.Sylvester @hg.zetaworlds.com:8002: Not much happened this week apart from more bug tracking really, was going to clean up mantis some more, but got busy with other projects
[11:03] Kayaker Magic: My favorite IRC joke is: https://xkcd.com/1782/
[11:04] Motoko.Karu @login.digiworldz.com:8002: ^^
[11:06] Andrew Hellershanks: Nothing much to report regarding changes to the Open Sim code base this past week. Just some minor clean up.
[11:07] Vincent.Sylvester @hg.zetaworlds.com:8002: I did get some strange behavior in offline IM two days ago which I have not quite pinpointed, mainly IM not delivering properly, but since that doesn't log anything be default nothing useful to report
[11:07] Vincent.Sylvester @hg.zetaworlds.com:8002: Suspect it's probably just some issue on timing or locking again as the table has gotten so big I am inclined to start clearing old entries out
[11:08] Vincent.Sylvester @hg.zetaworlds.com:8002: The crashing bug I did mantis as it turns out it was a bit more frequent than I first thought
[11:09] Vincent.Sylvester @hg.zetaworlds.com:8002: 9012 that is
[11:09] Vincent.Sylvester @hg.zetaworlds.com:8002: Will be a pain to track down, but a clue that it started when mono release 179 came out does raise some obvious concerns
[11:11] Andrew Hellershanks nods
[11:11] Vincent.Sylvester @hg.zetaworlds.com:8002: There was one other crash that referenced something in libomv called HexNibble whatever that is, which has me wondering if there is something with the byte conversion to hex that's breaking
[11:11] Andrew Hellershanks: yea, things often break when you install updates. :P
[11:12] Vincent.Sylvester @hg.zetaworlds.com:8002: Rolling back and checking some of the regions that crashed the most frequently is a last resort as rolling back is rather annoying and doesn't really solve the underlying issue whether it be OpenSim or mono itself
[11:12] Andrew Hellershanks: Vincent, it is still broken in 182?
[11:12] Vincent.Sylvester @hg.zetaworlds.com:8002: Yeah far as I can tell
[11:12] Andrew Hellershanks: ok.
[11:12] Vincent.Sylvester @hg.zetaworlds.com:8002: Was hoping that would fix it hence upgraded immediately
[11:12] Ubit Umarov: a nibble is 4 bits
[11:14] Vincent.Sylvester @hg.zetaworlds.com:8002: All these crashes reference script engine so I am hoping I'll eventually find a script that triggers this reliably so it can be debugged
[11:14] Vincent.Sylvester @hg.zetaworlds.com:8002: So far fishing in the dark
[11:14] Vincent.Sylvester @hg.zetaworlds.com:8002: Just too much stuff on those regions to pull apart
[11:14] Ubit Umarov: it just sounds like mono try/catch fails more now
[11:16] Vincent.Sylvester @hg.zetaworlds.com:8002: If only I could figure out how to get some line numbers or other information to point more directly at the point of failure
[11:16] Vincent.Sylvester @hg.zetaworlds.com:8002: Adding log output to it to log all uuids it encounters is going to be really spammy, but maybe a clue, though I have been unable to make it crash on a bad uuid
[11:18] Andrew Hellershanks: You could try running it using a debugger but that has its own issues. If it would seg fault and you have a Debug build you should have a back trace that could provide a starting point for where to look.
[11:18] Ubit Umarov: the low level does throw exceptions if the inputh has invalid chars
[11:19] Ubit Umarov: that is called inside a try/catch, so works as expected on correct mono
[11:19] Vincent.Sylvester @hg.zetaworlds.com:8002: What baffles me is I compiled libomv in debug and OpenSim anyways so not getting any line numbers is odd
[11:21] Vincent.Sylvester @hg.zetaworlds.com:8002: It really does seem like a mono issue, which then makes it even more pressing to find given it might impact a lot of things
[11:21] Vincent.Sylvester @hg.zetaworlds.com:8002: Suppose short of fiddling around with libomv or reading the mono commit log won't get much farther until I find a script that causes this reliably
[11:22] Ubit Umarov: well and on mantis 9012 all those messages sure have more information before the stack dump
[11:22] Ubit Umarov: like telling WHAT
[11:23] Vincent.Sylvester @hg.zetaworlds.com:8002: Most of these start with the stacktrace right after a OAR completes or flotsam cache clear happens, but no failures or errors, immediate stacktrace
[11:24] Ubit Umarov: well strackt trace means nothing withou cause
[11:26] Ubit Umarov: btw listtofloat does check uuids
[11:26] Ubit Umarov: bc someone decided a uuid can mean a float number
[11:26] Ubit Umarov: or a integer expressed as float
[11:27] Vincent.Sylvester @hg.zetaworlds.com:8002: Going through the log on the most crashing region, always happens right after datasnapshot runs
[11:27] Andrew Hellershanks: someone decided a uuid can mean a float number ?? That sounds like someone had some very strange ideas.
[11:28] Ubit Umarov: valid keys seem to get parsed as integers then converted to floats
if (item is LSL_Key)
{
string s = item.ToString();
if(UUID.TryParse(s, out UUID uuidt))
return Convert.ToDouble(new LSL_Integer(s).value);
// we can't do this because a string is also a LSL_Key for now :(
// else
// return 0;
}
[11:28] Ubit Umarov: coments there since "ever"
[11:29] Ubit Umarov: guess it is becasue keys are wrap to string, so ppl got strings reported as keys, so that was needed ???
[11:29] Andrew Hellershanks: That must come down in part to other places where there is little difference between a key and a string.
[11:29] Ubit Umarov: no idea :)
[11:30] Ubit Umarov: auto casting issues
[11:30] Ubit Umarov: possible
[11:30] Andrew Hellershanks: Hello, Selby.
[11:30] Selby.Evans @grid.kitely.com:8002: hi -- internet out till now
[11:30] Ubit Umarov: but well thats like that for ages... even possible it is doing as sl does realy
[11:30] Ubit Umarov: hi ec
[11:31] Ubit Umarov: and wc
[11:32] Kayaker Magic: Glad to hear your Internet is back Selby!
[11:32] Andrew Hellershanks: At least it didn't wait to come back until just as the meeting was winding down. :)
[11:33] Vincent.Sylvester @hg.zetaworlds.com:8002: This one crashed on at OpenSim.Server.Handlers.Simulation.Utils:GetParams <0x001c3>
at OpenSim.Server.Handlers.Simulation.AgentSimpleHandler:ProcessRequest <0x0014f>
[11:34] Ubit Umarov: test on a script
[11:34] Ubit Umarov: string a = "adb23jjd";
llOwnerSay((string)osIsUUID(a));
[11:34] Ubit Umarov: with several variants on a
[11:34] Vincent.Sylvester @hg.zetaworlds.com:8002: Yeah I tried a ton of that, but so far nothing crashing
[11:35] Vincent.Sylvester @hg.zetaworlds.com:8002: What seems really common is that the crashes happen after DataSnapShot runs
[11:35] Vincent.Sylvester @hg.zetaworlds.com:8002: Across three regions they have multiple crashes always right after that
[11:35] Ubit Umarov: that os thing is just direct tryparse :
[11:35] Ubit Umarov: public LSL_Integer osIsUUID(string thing)
{
CheckThreatLevel();
return UUID.TryParse(thing, out UUID test) ? 1 : 0;
}
[11:36] Vincent.Sylvester @hg.zetaworlds.com:8002: at <unknown> <0xffffffff>
at OpenSim.Region.ScriptEngine.Shared.ScriptBase.ScriptBaseClass:llSetRot <0x00061>
at System.Object:CheckSun() <0x009c2>
at System.Object:default timer <0x00452>
[11:36] Ubit Umarov: setrot??
[11:36] Vincent.Sylvester @hg.zetaworlds.com:8002: Yeah I am not making this up
[11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: at OpenSim.Region.ScriptEngine.Yengine.XMRInstAbstract:CallSEH <0x00236>
[11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: Also fairly common that
[11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: Really strange stuff
[11:37] Andrew Hellershanks: That is very odd.
[11:37] Ubit Umarov: are you sure your cpu does not have a hole on the fpu?
[11:37] Ubit Umarov: :)
[11:38] Vincent.Sylvester @hg.zetaworlds.com:8002: I count 154 managed stacktraces across these regions
[11:38] Ubit Umarov: well that last is unrelated
[11:38] Vincent.Sylvester @hg.zetaworlds.com:8002: Possible, who knows what bitflip goes on there, but so frequently right after Datasnapshot is a clue... I hope
[11:40] Ubit Umarov: well its not.. have a region doing it, qith no issues
[11:40] Vincent.Sylvester @hg.zetaworlds.com:8002: It's not all regions that crash, out of 96 only 21 crash regularly now
[11:40] Vincent.Sylvester @hg.zetaworlds.com:8002: Some more some less
[11:41] Ubit Umarov: some things like null refs do happen a bt more now, related to scenepresences, ie avatars
[11:41] Ubit Umarov: since i did add needed explicity delete of them
[11:42] Ubit Umarov: in past they just stayed in memory after the avatar left, so there where no null refs
[11:43] Ubit Umarov: so a lot of code is just broken ie was made with no test for the real presence of the avatar.. so just did.. something with no error
[11:43] Vincent.Sylvester @hg.zetaworlds.com:8002: Suppose I should roll back mono and see if it stops happening, as much trouble as that is, figure out if it just that or OpenSim
[11:43] Vincent.Sylvester @hg.zetaworlds.com:8002: Though I doubt it is OpenSim
[11:43] Ubit Umarov: hopefuly now does give a warning and goes on
[11:43] Ubit Umarov: the uuid thing is not at opensim level
[11:44] Ubit Umarov: as i said it is a mono fail, not doing a legal try/catch catch
[11:44] Ubit Umarov: or so it seems
[11:44] Ubit Umarov: i cant repo
[11:44] Ubit Umarov: ( only have mono 122 )
[11:45] Vincent.Sylvester @hg.zetaworlds.com:8002: Problem is of course rolling back and then not seeing the issue doesn't solve the problem, means crawling through mono commit log and finding what might be the cause, which sounds like fun...
[11:46] Andrew Hellershanks: I have 182.
[11:46] Vincent.Sylvester @hg.zetaworlds.com:8002: Heh maybe need to put the try catch inside another try catch
[11:46] Andrew Hellershanks: :)
[11:47] Vincent.Sylvester @hg.zetaworlds.com:8002: Really weird, always datasnapshot, crash on some lsl function totally unrelated
[11:47] Ubit Umarov: well to know WHAT did mono see could help
[11:47] Vincent.Sylvester @hg.zetaworlds.com:8002: I dread to even go to the commit log fearing for how much I will find there to go through
[11:48] Vincent.Sylvester @hg.zetaworlds.com:8002: Gonna need like ten clones to give that a code review
[11:48] Ubit Umarov: mono guys are now like opensim ones.. do commits like "several bugs fixes" :p
[11:49] Ubit Umarov: "want to know more?"
[11:49] Andrew Hellershanks: :)
[11:49] Ubit Umarov: see the commit
[11:49] Vincent.Sylvester @hg.zetaworlds.com:8002: Not to mention can't actually really tell what went into the release, just guess basically
[11:49] Ubit Umarov: well thye only did publish the release notes for 174 not 179
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: This going to be a fun bug...
[11:50] Andrew Hellershanks: Vincent, you should be able to tell that based on the tags and dates of release.
[11:51] Ubit Umarov: seems there is a even on Metro close
[11:51] Ubit Umarov: but like all modern reporters.. NO DATE on the news :p
[11:51] Ubit Umarov: at least i could not see it
[11:52] Vincent.Sylvester @hg.zetaworlds.com:8002: Sad to see them go like this
[11:52] Vincent.Sylvester @hg.zetaworlds.com:8002: Such a constant just vanishing
[11:52] Ubit Umarov: "
[11:52] Ubit Umarov: The play is titled “The Art Giantess Is The Work: Thea von Harbou,” and will be performed in English, with German subtitles. It will start at 1 p.m. Pacific time, or 10 p.m. German time on the Futurelab region.
The hypergrid address is metro.land:8002:futurelab.
[11:52] Ubit Umarov: "
[11:52] Ubit Umarov: see any date?
[11:53] Vincent.Sylvester @hg.zetaworlds.com:8002: Best show up every day to make sure
[11:54] Ubit Umarov: well this is so typical of news on the net.. jezz
[11:54] Andrew Hellershanks: The web page says the grid will close on June 30.
[11:54] Ubit Umarov: yes june 30 is a guess... but if it closed.. can't be :P
[11:55] Ubit Umarov: other guess is june 27.. the news date ( by miracle there >)
[11:55] Ubit Umarov: but guess june 30 is the date
[11:56] Ubit Umarov: well sounds a nice event... so..
[11:56] Ubit Umarov: https://www.hypergridbusiness.com/2022/06/new-play-premieres-on-metropolis-grid/
[11:57] Ubit Umarov: well that hypergrid site sees a mess now
[11:58] Ubit Umarov: was looking to opensim news, now looking to Home i see "news" abotu opensim, not there
[11:58] Andrew Hellershanks: An image further down that page indicates the play premieres on the 30th.
[11:58] Andrew Hellershanks: Not that unusual a situation for theatre plays. Open and close on the same day. ;)
[11:58] Ubit Umarov: :)
[11:58] Ubit Umarov: well this is on hypergrid home, not news
[11:59] Ubit Umarov: https://www.hypergridbusiness.com/2022/06/diva-distro-seals-vulnerability-grids-need-to-update/
[11:59] Andrew Hellershanks: The hour is almost up once again. Any other questions/comments before we wrap up todays gathering?
[12:01] Ubit Umarov: seems it was a stack overflow, due to un guarded recurrency calls
[12:01] Ubit Umarov: think i did remove one of such tech uses fro opensim
[12:02] Andrew Hellershanks: Thanks for mentioning that article. Important information for anyone using Diva Wifi.
[12:03] Ubit Umarov: recurrent calls are nice way to doe some code, but very easy to go wrong, specialling if using user provided arguments
[12:03] Ubit Umarov: ..specialy ..
[12:03] Kayaker Magic: Duty calls, I need to run.
[12:04] Ubit Umarov: tp is faster
[12:04] Ubit Umarov: )
[12:04] Ubit Umarov: when it does not fail... ofc
[12:04] Selby.Evans @grid.kitely.com:8002: bye all
[12:04] Ubit Umarov: cya
[12:05] Andrew Hellershanks: Bye, Kayaker and Selby.
[12:05] Motoko.Karu @login.digiworldz.com:8002: bye
[12:06] Andrew Hellershanks: If there is nothing more for today I'll call this meeting to a close.
[12:06] Andrew Hellershanks: Thank you all for coming. See you again next week.
