Chat log from the meeting on 2014-07-29

 [11:01] Richardus Raymaker: the latest kokua release is doing weird with mata [11:01] Mata Hari: oh? [11:02] Shez Oyen: you mean the blinking off and on? [11:02] Justin.Clark-Casey @hg.osgrid.org: hi folks [11:02] Shez Oyen: Hi Justin [11:02] Richardus Raymaker: if you move the camera in some angles your avatar you see short flash in the sly above you [11:02] TBG Renfold: hi [11:02]  Richardus Raymaker: hi sahex, justin [11:02] Mata Hari: it's my dazzling skin texture...makes your eyes water [11:03] Shez Oyen: lol [11:03] Richardus Raymaker: \no not blinking. just a flash of the sitting mata but just higher. its very short [11:03] Mata Hari: it's probably because you're transitioning between LOD points [11:03] Richardus Raymaker: oh nebadon i have the same [11:03] Mata Hari: you'll see a brief flash of the different LOD model and then it moves into position [11:03] Richardus Raymaker: maby, hi robert [11:03] Robert Adams: hi all [11:04] Richardus Raymaker: that sounds close to what i see mata [11:04] Mata Hari: I have that happen at times to [11:04]  Mata Hari: *too [11:04] Mata Hari: only solution is to avoid those camera distances [11:04] Shez Oyen: Hi Robert and thank you again! [11:04] Shez Oyen heaps praise and appreciation on Robert [11:05] Shez Oyen: Hi Jazzy :) [11:05]  jazzy chatterbox: hello Shez :) [11:05] jazzy chatterbox: hello Everyone :) [11:05]  Pathfinder.Lester @pathlandia.dlinkddns.com:9000 waves [11:05]  Use a search string with at least 3 characters [11:05]  Nebadon Izumi: hello everyone [11:05]  Shez Oyen: Hi Lester and Neb :) [11:05] Justin.Clark-Casey @hg.osgrid.org: \/me saw a BulletSim hashset exception just now ;) [11:05]  Nebadon Izumi: sorry am quiet was doing a bit of OSCC Load Test prep [11:06]  Robert Adams: BulletSim hashset error? is that a new one? [11:07]  Justin.Clark-Casey @hg.osgrid.org: robert: yes, will pastebin when I can [11:07]  Cyberman Cyberworld: hi all [11:07]  Justin.Clark-Casey @hg.osgrid.org: whacking in special scene settings to kn regions atm [11:07]  Shez Oyen: Hiya Cyberman [11:08]  Cyberman Cyberworld: im just checking how opensim is? [11:08]  Cyberman Cyberworld: is it good [11:08]  Justin.Clark-Casey @hg.osgrid.org: nah, it's bad, real bad [11:08]  Shez Oyen: lol [11:08]  Mata Hari: any clue as to what caused the Event Plaza mass-kick on Sunday? [11:08]  Shez Oyen: ghastly [11:08]  jazzy chatterbox: haha [11:08]  Mata Hari: I've seen that happen occasionaly on Friday nights at Doro's [11:08]  jazzy chatterbox: that was very interesting to say the least Masta [11:08] Nebadon Izumi: I don't know Mata [11:08] Nebadon Izumi: there was a lot of visitors [11:08] jazzy chatterbox: Mata* [11:09] Mata Hari: it's like suddenly the region loses track of who is supposed to be there and kicks everyone except for one or two recent lucky arrivals [11:09] Kayaker Magic: If by 'good' you mean how reliable, OpenSim gets more reliable every day! Thanks to some of the people here. [11:10] Pathfinder.Lester @pathlandia.dlinkddns.com:9000: It's come amazingly far in the past couple years, in my opinion. [11:10] Justin.Clark-Casey @hg.osgrid.org: how many visitors? [11:10] Nebadon Izumi: I don't recall exactly something like 50 or so at peak [11:10] Mata Hari: usually in the 20+ range when it happens at Doros [11:10] Cyberman Cyberworld: help where is my body??? [11:10] Nebadon Izumi: I thought it did very well [11:10] Mata Hari: but I've also seen it happen at as few as only 5-6 in the region [11:10] Cyberman Cyberworld: where is my avatar [11:10] Justin.Clark-Casey @hg.osgrid.org: you know what kind of hardware/network it has? [11:10] Nebadon Izumi: we had 1 crash the entire weekend [11:10] Nebadon Izumi: events like taht at SL bithday parties [11:10] Mata Hari: Akira's....no idea [11:10] Billy.Bradshaw @hg.viewtwo.net:8600: Think there was a mass kick during the HG Safari meeting while visiting OSG7B, I think. Left me with 4 or 5 part processed friends. [11:10] Kayaker Magic: Neb, you look amazingly 3D in those goggles! [11:10] Nebadon Izumi: even those regions crash a couple times a day usually [11:10] Mata Hari: but the identical thing happened sunday at EVent Plaza [11:11] Nebadon Izumi: hehe thanks Kayaker, you can grab a copy o the table next to me [11:11]  Nebadon Izumi: I suspect its probably similar issues we see at load test [11:11] jazzy chatterbox: i know that SAM told me i had 29 listeners when it purged down to 4 in a matter of a couple seconds, but there were about 34 people total on the sim right then I think [11:11] Nebadon Izumi: but a bit more in slow motion [11:11] Mata Hari: and the region doesn't crash...it just kicks most occupants [11:11] Robert Adams: Cyberman: I've inserted a slave trade module into OpenSim. We're into the body trade. (I joke, I joke) [11:11] Pathfinder.Lester @pathlandia.dlinkddns.com:9000: I hate hate that part processed friends thing. I've lost so many friends that way... [11:11] Justin.Clark-Casey @hg.osgrid.org: it's hard to say without seeing logs and perf data. But got my hands very full atm [11:11] Nebadon Izumi: it takes 6-8 hours of lower numbers of avatars [11:11] Mata Hari: nods [11:12] Nebadon Izumi: to trigger what we trigger in 20 minutes of Justins and my bots [11:12] Robert Adams: would have to see the logs when everyone got kicked. That's the place to start [11:12] Mata Hari: well, the logs are there....around 1:30 pm grid time on Event Plaza [11:12] Justin.Clark-Casey @hg.osgrid.org: what we really need to do is fix warp 3d map regen leaking [11:12] Mata Hari: on Sunday [11:12] Nebadon Izumi: we are defnitely not using that though Justin [11:12] Nebadon Izumi: but would be good to fix [11:12] Justin.Clark-Casey @hg.osgrid.org: yeah. too easy for someone to put on and not realise it leaks memory like crazy [11:12] Robert Adams: warp 3d probably needs retirement. I is complex and has problems and no one is around to give it love [11:13] Richardus Raymaker: how do you knows it leaks, maby its because i only generate at startup. anyway if its doing it sometimes i dont see much increase [11:13] Richardus Raymaker: well, come with betetr warp3d robert [11:13] Justin.Clark-Casey @hg.osgrid.org: ideally, but it does produce better images than the default tiler [11:13] Mata Hari: but in all other respects I'd say the weekend was a smashing success [11:13] Richardus Raymaker: not going to use the terrible defaul;t ones [11:13] Billy.Bradshaw @hg.viewtwo.net:8600: I now see friend on line appear in chat, but then look in the people friend dialogue they are not there [11:13] Mata Hari: including consistently 40+ avi on the event plaza on Saturday [11:13] Mata Hari: so nice constant load test [11:13] Justin.Clark-Casey @hg.osgrid.org: richardus: if I manually regen map tiles it leaks/. But not a huge surprise since the code doesn't dispose of any of its graphics objects [11:13] Justin.Clark-Casey @hg.osgrid.org: I fixed a similar problem in the default tiler which now does not leak [11:14] Justin.Clark-Casey @hg.osgrid.org: but fixing warp is more complex [11:14] Justin.Clark-Casey @hg.osgrid.org: it's not written in a very oo style [11:14] Richardus Raymaker: possible more easy to write a nerw one ? [11:14] Justin.Clark-Casey @hg.osgrid.org: maybe even [11:14] Robert Adams: I think both Justin and myself has tried to get Warp3D to not leak (adding disposing). Still not fixed [11:14] Justin.Clark-Casey @hg.osgrid.org: that said, I thnik aurora may have done some work on it [11:14]  Justin.Clark-Casey @hg.osgrid.org: yeah, all the othe rparts are deep in the warp3d code itself [11:15] Richardus Raymaker: is the leaking maby linux / mono problem only ? [11:15] Robert Adams: I ported over some of the Aurora fixes to Warp3D. Still not fixed [11:15] Justin.Clark-Casey @hg.osgrid.org: ok [11:15]  Justin.Clark-Casey @hg.osgrid.org: that all in opensim-libs? [11:15] Robert Adams: not even sure if the warp3d dll is in -libs. I've never looked [11:16] Justin.Clark-Casey @hg.osgrid.org: it is. So are your code fixes somewhere else? [11:17] Robert Adams: my fixes were doing disposes in the C# code. I haven't looked at the dll [11:17] Justin.Clark-Casey @hg.osgrid.org: This is the exception I observed about 20 mins ago on a two region simulator startup btw -> http://pastebin.com/sNS39tnb [11:18] Justin.Clark-Casey @hg.osgrid.org: robert: ah right [11:18] Justin.Clark-Casey @hg.osgrid.org: I'm,not sure the code in opensim-libs actually corresponds exactly to warp3d.dll which is a bad thing [11:18] BlueWall Slade is Online [11:18] Robert Adams: That is a new one Justin. Make a Mantis out of it and assign it to me. Something is not getting locked [11:18] Justin.Clark-Casey @hg.osgrid.org: ok, thanks [11:19] Robert Adams: Or, more likely, the threading has been changed in OpenSim so you're finding lock problems that have always been there :) [11:20]  Robert Adams: should be easy to find from that call stack [11:21]  Justin.Clark-Casey @hg.osgrid.org: yes [11:21]  Mata Hari: path-too-long error :p [11:23]  Justin.Clark-Casey @hg.osgrid.org: anyway, any other opensim topics today? [11:23]  Simulator Version v0.5 shouts: OpenSim 0.8.1 Dev          3a87cce: 2014-07-22 18:04:28 +0100 (Unix/Mono) [11:23]  Cuteulala Artis is Online [11:23]  Kayaker Magic: What are you working on now Justin? [11:23]  Justin.Clark-Casey @hg.osgrid.org: right now, performance under load [11:24]  Mata Hari: nothing special that's new...just more instances of old [11:24]  Billy.Bradshaw @hg.viewtwo.net:8600: mata, I deleted the calling card for the friend I declined but the offers continued, so I accepted in the end. [11:24] Richardus Raymaker: and then its possible safe to delete [11:24] Justin.Clark-Casey @hg.osgrid.org: I have a bunch of xperimental tweaks for the test after to see if they help. I believe the issue atm is just sheer amount of agent updates but it can be quite hard to tell [11:25] Robert Adams: there are queues on top of queues and threads competing with threads... a hard problem to unravel [11:25] Justin.Clark-Casey @hg.osgrid.org: very [11:25] Dahlia Trimble is Online [11:25] Justin.Clark-Casey @hg.osgrid.org: it has grown very organically [11:26] Justin.Clark-Casey @hg.osgrid.org: half the time I'm surprised it works as well as it does [11:26] Kayaker Magic: lol [11:26] Mata Hari: I was glad to see that you were able to reproduce the fetch issue on your own sim...misery loves company [11:26] Justin.Clark-Casey @hg.osgrid.org: yeah. but only for the first time, which was very odd [11:26] Justin.Clark-Casey @hg.osgrid.org: I really need to consistently reproduce it to get to tht ebottom of things [11:27] Mata Hari: I changed the timeout to the much shorter one....still not a single message to log from it [11:27]  Justin.Clark-Casey @hg.osgrid.org: I find that extremely odd - the request should simply timeout whatever happens [11:28] Justin.Clark-Casey @hg.osgrid.org: mata: remind me, are you on windows or mono? [11:28] Mata Hari: win 7 64 and have .net 4.5 [11:28] Justin.Clark-Casey @hg.osgrid.org: and you run at debug log level? [11:29] Mata Hari: usually, yes [11:29] Dahlia Trimble: hi [11:29]  Justin.Clark-Casey @hg.osgrid.org: hi dahlia [11:30] Nebadon Izumi: hello dahlia [11:30] Justin.Clark-Casey @hg.osgrid.org: mata: yeah, I just don't understand why requests don't timeout. THere's no way that should fail [11:31] Mata Hari: maybe it's a case of them not timing out but instead somehow failing to then move on to the next [11:31] Justin.Clark-Casey @hg.osgrid.org: aside from that, I was thinking whether it might be some problematic folder, but I just don't think they refer to each otherin a way that could cause any kind of loop [11:31] Mata Hari: so it jst keeps repeating the same query over and over again [11:32] Justin.Clark-Casey @hg.osgrid.org: that should still send the processed count stats up [11:32]  Billy.Bradshaw @hg.viewtwo.net:8600: If you disable HTTP Inventory in the viewer that would remove the http server?? [11:32] Richardus Raymaker: Mata did you cleared viewer cahce already for test ? [11:32] Justin.Clark-Casey @hg.osgrid.org: billy: yes, it would force fallback to older udp mechanisms which aren't throttled in the same way [11:32] Mata Hari: I try to avoid clearning cache if I can [11:33] Justin.Clark-Casey @hg.osgrid.org: mata: maybe the queueing code is something to look at thought - I've never been very happy with the home-baked mechanisms used [11:33] Mata Hari: I've toggles HTTP fatch too...same issue either way [11:33] Justin.Clark-Casey @hg.osgrid.org: .net 4 has better sdk classes for this kind of thing [11:33] Justin.Clark-Casey @hg.osgrid.org: I would be extremely surprised if that's the case - the udp is an entirely different mechanism [11:34] Justin.Clark-Casey @hg.osgrid.org: are you absolutely sure your viewer is falling back? [11:34] Richardus Raymaker: after i upgraded kokua i copuld not login as example, cleared cache and then it worked. alsoo possible just stupid bad luck the first time [11:34] Nebadon Izumi: network probably is a factor [11:35] Billy.Bradshaw @hg.viewtwo.net:8600: I have seen the slowing under both configs, but never the stall [11:35] Nebadon Izumi: i suspect something between simulators and robust [11:35] Mata Hari: I have no idea how to know for sure what my viewer is or isn't doing [11:35] Nebadon Izumi: because most of these people who have inventory issues [11:35] Nebadon Izumi: they go to a plaza and they are fine [11:35] Justin.Clark-Casey @hg.osgrid.org: on simulator console "debug lludp packet 1" will show the less common packets received by the server [11:35] Richardus Raymaker: Mata what viewer ? [11:35] Nebadon Izumi: plazas are in same data center as robust servers [11:35] Nebadon Izumi: its like zero latency [11:36] Justin.Clark-Casey @hg.osgrid.org: so probably in this case "debug lludp packet 1 --default" and then log in with viewer. If iot's using udp you should see a lot of inentory packets [11:36] Mata Hari: FS 4.6.5 (or whatever that most recent one is) [11:36] Nebadon Izumi: if it was between viewer and simulator they would experience it everywhere [11:36] Justin.Clark-Casey @hg.osgrid.org: it could be windows behave slightly differently than mono here.... [11:36] Richardus Raymaker: i would try sinmgularity. FS not the best one. for me the latest feels a bit more buggy anyway and thats in sl [11:36] Mata Hari: I have never had fetch fail for me on a plaza [11:36] Justin.Clark-Casey @hg.osgrid.org: ai austin is windows as well afair [11:36] Nebadon Izumi: like Ai for instance is in UK [11:36]  Nebadon Izumi: and plazas are in Texas [11:37] Nebadon Izumi: and its ok from his viewer to the plazas, but not his regions to robust in same location [11:37] Mata Hari: in fact usually by the time my viewer shows me the screen the fetch is finished [11:37] Nebadon Izumi: the only thing I can think is on OSgrid [11:37] Nebadon Izumi: we throttle any single IP address to no more than 120 requests per second [11:37] Nebadon Izumi: with nginx [11:38] Nebadon Izumi: could the simulators be exceeding that to Robust? [11:38] Justin.Clark-Casey @hg.osgrid.org: none of this explains the complete lack of timeout for the requests [11:38] Justin.Clark-Casey @hg.osgrid.org: even without any setting changes, they should always teimout after 100 secs and report that fact to the console [11:38] Nebadon Izumi: ya nginx gives a clear erorr message when you are being throttled [11:38] Justin.Clark-Casey @hg.osgrid.org: which should apply to both windows and mono as that's part of the sdk spec [11:38] Mata Hari: I just checked to be sure....     [11:38] Nebadon Izumi: no timeout to me says there is no response at all being processed? [11:39] Justin.Clark-Casey @hg.osgrid.org: even then, timeout should occur [11:39] BlueWall Slade: the console shows the requests very far apart here. [11:39] BlueWall Slade: maybe 2-3 / minute [11:40] Nebadon Izumi: ya and those are always between simulator and robust right? [11:40] Nebadon Izumi: some how high latency disrupts that, very odd [11:40] BlueWall Slade: well, the testing we were doing a couple of nights ago here in ?OSG [11:40] Nebadon Izumi: ya [11:40]  Mata Hari: so if you want me to do something extra/different other than have it set to DEBUG in opensim.exe.config I'd be happy to...even have it spew data for a test or two if that owuld help [11:41] Nebadon Izumi: i also check the osgrid nginx timeout period [11:41] BlueWall Slade: yep, seems like a slow connectiong to the sim triggers something bad [11:41] Nebadon Izumi: its pretty long like 300 seconds or something [11:41] Robert Adams: be sure to minimize the log window or the scrolling slows everything down [11:41] Justin.Clark-Casey @hg.osgrid.org: mata: tbh, that bug eport is already hard to pick through. I would rather have a think about why timeout could possibly be failing altogether [11:41] Justin.Clark-Casey @hg.osgrid.org: that's an extreme anomoly [11:41] Nebadon Izumi: since sometimes requests can genuinely take long time [11:41] Justin.Clark-Casey @hg.osgrid.org: but I probably won't get time again until later this week [11:41] Mata Hari nods [11:42] Mata Hari: well you know where to find me if you need a guinea pig [11:42] Justin.Clark-Casey @hg.osgrid.org: mata: do you have any way to test this stuff under mono? [11:42] Mata Hari: and Ai would I'm sure be happy to test too [11:42] BlueWall Slade: is there any reason that we can't get the inventory as a blob? [11:42] Justin.Clark-Casey @hg.osgrid.org: requests come in fromthe viewer one by one [11:42] Mata Hari: er....I could probably set up something on my old box [11:42] BlueWall Slade: any good reason to get it serialy [11:42] Justin.Clark-Casey @hg.osgrid.org: so you would have to bundle those up [11:42]  BlueWall Slade: ok [11:42]  Mata Hari: it's running ubuntu 10.something [11:43] Justin.Clark-Casey @hg.osgrid.org: mata: I'm wondering if it could possibly be a windows issue as that's what you and ai have in common. Or that could just be a coincidence [11:43] BlueWall Slade: that is a good-enough reason :S [11:43] Richardus Raymaker: Then why is my windows with singularity seems to have no poroiblems ? [11:43] Mata Hari: the other box is pretty underpowered though....Core2Duo with only 4GN RAM [11:43] Justin.Clark-Casey @hg.osgrid.org: I think ubuntu 10 has a high enough versino of mono (2.10) but can't remember for sure [11:44] Justin.Clark-Casey @hg.osgrid.org: richardus: it's not a question of opsys for the viewer but for the simulator [11:44] Richardus Raymaker: k, [11:44]  Justin.Clark-Casey @hg.osgrid.org: mata: do I remember youi saying you also had the issue on a private grid? [11:44] Justin.Clark-Casey @hg.osgrid.org: or rather one where you run your own services? [11:45] Mata Hari: no, I never had that issue when I was hosting a standalone instead [11:45] Mata Hari: and I never had that issue under 0.7.6 [11:45] Mata Hari: in fact I never had that issue until around Christmastime [11:45] Justin.Clark-Casey @hg.osgrid.org: ok, another thing to do may be to put in a debug switch that logs what inve requests are being made and see if it always 'sticks' at the same one [11:46] Mata Hari: this time last year I was crashing like crazy with swapping meshes around and never had troub le logging into my own regions [11:46] Justin.Clark-Casey @hg.osgrid.org: bluewall: actually, now I think about it, the viewer does request a bunch of folders at once so really taht could be bunched. But it means a new inventory service request point, etc. [11:46] Justin.Clark-Casey @hg.osgrid.org: and I'm not sure if it would actually help in this case [11:47] BlueWall Slade: maybe a good future goal then [11:47] Mata Hari: something in OSG throttling could easily be the issue though since both my viewer and regions are the same IP externally [11:47] Justin.Clark-Casey @hg.osgrid.org: I was thinking some caching might be possible simulator side but now I'm not quite so sure [11:47] BlueWall Slade: those serial requests are expensive [11:48] Mata Hari: so if both are making lots of requests there'd be a burst of them [11:48] Justin.Clark-Casey @hg.osgrid.org: mata: if that's the case tyou should still see timeout [11:48] Justin.Clark-Casey @hg.osgrid.org: as the simulator doesn't fulfill the requiest in time or perhaps sends back a 503.... [11:48] Justin.Clark-Casey @hg.osgrid.org: er sorry, service [11:48] BlueWall Slade: it seems to hit a wall, then making requests at about 2-3 / minute [11:48] Mata Hari: hm....well the log and the console are both blissfully empty when it's happening [11:48] BlueWall Slade: I'm not sure though if they are repeat requests [11:48] Justin.Clark-Casey @hg.osgrid.org: bluewall: whjich start are you looking at? [11:49] Mata Hari: the only error message that gets thrown is my CTD if I shut down the sim without wating for at least 10+ mimutes after I log out [11:49] BlueWall Slade: just the console log [11:49] Justin.Clark-Casey @hg.osgrid.org: bluewall: the service log? [11:49] Justin.Clark-Casey @hg.osgrid.org: bluewall: I don't understand since requests aren't loggedatm [11:49] BlueWall Slade: just the console [11:50] BlueWall Slade: it was reporting calls to the OSG inventory server [11:50] Justin.Clark-Casey @hg.osgrid.org: I see way more than 2-3 per min [11:50] BlueWall Slade: hmmm, maybe the slow request logging [11:51] Justin.Clark-Casey @hg.osgrid.org: Though on the robust console for osgrid you can see that some requests take a long time [11:51] Justin.Clark-Casey @hg.osgrid.org: though at the moment not longer than 5 seconds [11:51] Justin.Clark-Casey @hg.osgrid.org: usually/. Could even be poorly optimized db queries... [11:51] Robert Adams: I are going to run... take care all. See you in Mantis land [11:51] Justin.Clark-Casey @hg.osgrid.org: bye robert [11:51] Shez Oyen: BB Robert [11:51] Mata Hari: bye Robert [11:51] BlueWall Slade: I will try to fire it up later and let you know [11:52] BlueWall Slade: bye Robert [11:52] Richardus Raymaker: bye robert [11:52] Dahlia Trimble: bye Robert [11:54] Nebadon Izumi: I have about 1/2 my bots logged in Justin [11:54] Nebadon Izumi: im sitting them down today [11:54] Nebadon Izumi: so taking me a bit longer [11:54] Justin.Clark-Casey @hg.osgrid.org: ok. I'm concenttrating on udp traffic reduction measures today [11:54] Richardus Raymaker: let you guys and girls playing. need to go. bye [11:54] Mata Hari: where to today? [11:55] Richardus Raymaker is Offline [11:55] Mata Hari: (for load test) [11:55] BlueWall Slade: Bye RiRa [11:55] Nebadon Izumi: same place [11:55] Mata Hari: kk [11:55]  Nebadon Izumi: location is http://cc.opensimulator.org:8002 Keynote 1 [11:55] jazzy chatterbox: i need to run, good luck all you Brilliant people :) [11:55]  Nebadon Izumi: thogh if you have alts you can spread them evenly over Keynote 2, Keynote 3, and Keynote 4 [11:56]  Shez Oyen: See ya Jazzy [11:56]  Pathfinder.Lester @pathlandia.dlinkddns.com:9000: seeya on Keynote, folks [11:56]  BlueWall Slade: bye jazzy [11:56]  Nebadon Izumi: yea we can start heading over now [11:56]  Nebs Metal Bar Stool v1.5 (w/sit & launch): Goodbye.. [11:56]  Nebadon Izumi whispers: see you all there :) [11:58] Justin.Clark-Casey @hg.osgrid.org: ok [11:58]  Stephen Xootfly is Online