Chat log from the meeting on 2014-07-29

From OpenSimulator

Jump to: navigation, search
[11:01]  Richardus Raymaker: the latest kokua release is doing weird with mata
[11:01]  Mata Hari: oh?
[11:02]  Shez Oyen: you mean the blinking off and on?
[11:02]  Justin.Clark-Casey hi folks
[11:02]  Shez Oyen: Hi Justin
[11:02]  Richardus Raymaker: if you move the camera in some angles your avatar you see short flash in the sly above you
[11:02]  TBG Renfold: hi
[11:02]  Richardus Raymaker: hi sahex, justin
[11:02]  Mata Hari: it's my dazzling skin texture...makes your eyes water
[11:03]  Shez Oyen: lol
[11:03]  Richardus Raymaker: \no not blinking. just a flash of the sitting mata but just higher. its very short
[11:03]  Mata Hari: it's probably because you're transitioning between LOD points
[11:03]  Richardus Raymaker: oh nebadon i have the same
[11:03]  Mata Hari: you'll see a brief flash of the different LOD model and then it moves into position
[11:03]  Richardus Raymaker: maby, hi robert
[11:03]  Robert Adams: hi all
[11:04]  Richardus Raymaker: that sounds close to what i see mata
[11:04]  Mata Hari: I have that happen at times to
[11:04]  Mata Hari: *too
[11:04]  Mata Hari: only solution is to avoid those camera distances
[11:04]  Shez Oyen: Hi Robert and thank you again!
[11:04]  Shez Oyen heaps praise and appreciation on Robert
[11:05]  Shez Oyen: Hi Jazzy :)
[11:05]  jazzy chatterbox: hello Shez :)
[11:05]  jazzy chatterbox: hello Everyone :)
[11:05]  Pathfinder.Lester waves
[11:05]  Use a search string with at least 3 characters
[11:05]  Nebadon Izumi: hello everyone
[11:05]  Shez Oyen: Hi Lester and Neb :)
[11:05]  Justin.Clark-Casey \/me saw a BulletSim hashset exception just now ;)
[11:05]  Nebadon Izumi: sorry am quiet was doing a bit of OSCC Load Test prep
[11:06]  Robert Adams: BulletSim hashset error? is that a new one?
[11:07]  Justin.Clark-Casey robert: yes, will pastebin when I can
[11:07]  Cyberman Cyberworld: hi all
[11:07]  Justin.Clark-Casey whacking in special scene settings to kn regions atm
[11:07]  Shez Oyen: Hiya Cyberman
[11:08]  Cyberman Cyberworld: im just checking how opensim is?
[11:08]  Cyberman Cyberworld: is it good
[11:08]  Justin.Clark-Casey nah, it's bad, real bad
[11:08]  Shez Oyen: lol
[11:08]  Mata Hari: any clue as to what caused the Event Plaza mass-kick on Sunday?
[11:08]  Shez Oyen: ghastly
[11:08]  jazzy chatterbox: haha
[11:08]  Mata Hari: I've seen that happen occasionaly on Friday nights at Doro's
[11:08]  jazzy chatterbox: that was very interesting to say the least Masta
[11:08]  Nebadon Izumi: I don't know Mata
[11:08]  Nebadon Izumi: there was a lot of visitors
[11:08]  jazzy chatterbox: Mata*
[11:09]  Mata Hari: it's like suddenly the region loses track of who is supposed to be there and kicks everyone except for one or two recent lucky arrivals
[11:09]  Kayaker Magic: If by 'good' you mean how reliable, OpenSim gets more reliable every day! Thanks to some of the people here.
[11:10]  Pathfinder.Lester It's come amazingly far in the past couple years, in my opinion.
[11:10]  Justin.Clark-Casey how many visitors?
[11:10]  Nebadon Izumi: I don't recall exactly something like 50 or so at peak
[11:10]  Mata Hari: usually in the 20+ range when it happens at Doros
[11:10]  Cyberman Cyberworld: help where is my body???
[11:10]  Nebadon Izumi: I thought it did very well
[11:10]  Mata Hari: but I've also seen it happen at as few as only 5-6 in the region
[11:10]  Cyberman Cyberworld: where is my avatar
[11:10]  Justin.Clark-Casey you know what kind of hardware/network it has?
[11:10]  Nebadon Izumi: we had 1 crash the entire weekend
[11:10]  Nebadon Izumi: events like taht at SL bithday parties
[11:10]  Mata Hari: Akira' idea
[11:10]  Billy.Bradshaw Think there was a mass kick during the HG Safari meeting while visiting OSG7B, I think. Left me with 4 or 5 part processed friends.
[11:10]  Kayaker Magic: Neb, you look amazingly 3D in those goggles!
[11:10]  Nebadon Izumi: even those regions crash a couple times a day usually
[11:10]  Mata Hari: but the identical thing happened sunday at EVent Plaza
[11:11]  Nebadon Izumi: hehe thanks Kayaker, you can grab a copy o the table next to me
[11:11]  Nebadon Izumi: I suspect its probably similar issues we see at load test
[11:11]  jazzy chatterbox: i know that SAM told me i had 29 listeners when it purged down to 4 in a matter of a couple seconds, but there were about 34 people total on the sim right then I think
[11:11]  Nebadon Izumi: but a bit more in slow motion
[11:11]  Mata Hari: and the region doesn't just kicks most occupants
[11:11]  Robert Adams: Cyberman: I've inserted a slave trade module into OpenSim. We're into the body trade. (I joke, I joke)
[11:11]  Pathfinder.Lester I hate hate that part processed friends thing. I've lost so many friends that way...
[11:11]  Justin.Clark-Casey it's hard to say without seeing logs and perf data. But got my hands very full atm
[11:11]  Nebadon Izumi: it takes 6-8 hours of lower numbers of avatars
[11:11]  Mata Hari: nods
[11:12]  Nebadon Izumi: to trigger what we trigger in 20 minutes of Justins and my bots
[11:12]  Robert Adams: would have to see the logs when everyone got kicked. That's the place to start
[11:12]  Mata Hari: well, the logs are there....around 1:30 pm grid time on Event Plaza
[11:12]  Justin.Clark-Casey what we really need to do is fix warp 3d map regen leaking
[11:12]  Mata Hari: on Sunday
[11:12]  Nebadon Izumi: we are defnitely not using that though Justin
[11:12]  Nebadon Izumi: but would be good to fix
[11:12]  Justin.Clark-Casey yeah. too easy for someone to put on and not realise it leaks memory like crazy
[11:12]  Robert Adams: warp 3d probably needs retirement. I is complex and has problems and no one is around to give it love
[11:13]  Richardus Raymaker: how do you knows it leaks, maby its because i only generate at startup . anyway if its doing it sometimes i dont see much increase
[11:13]  Richardus Raymaker: well, come with betetr warp3d robert
[11:13]  Justin.Clark-Casey ideally, but it does produce better images than the default tiler
[11:13]  Mata Hari: but in all other respects I'd say the weekend was a smashing success
[11:13]  Richardus Raymaker: not going to use the terrible defaul;t ones
[11:13]  Billy.Bradshaw I now see friend on line appear in chat, but then look in the people friend dialogue they are not there
[11:13]  Mata Hari: including consistently 40+ avi on the event plaza on Saturday
[11:13]  Mata Hari: so nice constant load test
[11:13]  Justin.Clark-Casey richardus: if I manually regen map tiles it leaks/. But not a huge surprise since the code doesn't dispose of any of its graphics objects
[11:13]  Justin.Clark-Casey I fixed a similar problem in the default tiler which now does not leak
[11:14]  Justin.Clark-Casey but fixing warp is more complex
[11:14]  Justin.Clark-Casey it's not written in a very oo style
[11:14]  Richardus Raymaker: possible more easy to write a nerw one ?
[11:14]  Justin.Clark-Casey maybe even
[11:14]  Robert Adams: I think both Justin and myself has tried to get Warp3D to not leak (adding disposing). Still not fixed
[11:14]  Justin.Clark-Casey that said, I thnik aurora may have done some work on it
[11:14]  Justin.Clark-Casey yeah, all the othe rparts are deep in the warp3d code itself
[11:15]  Richardus Raymaker: is the leaking maby linux / mono problem only ?
[11:15]  Robert Adams: I ported over some of the Aurora fixes to Warp3D. Still not fixed
[11:15]  Justin.Clark-Casey ok
[11:15]  Justin.Clark-Casey that all in opensim-libs?
[11:15]  Robert Adams: not even sure if the warp3d dll is in -libs. I've never looked
[11:16]  Justin.Clark-Casey it is. So are your code fixes somewhere else?
[11:17]  Robert Adams: my fixes were doing disposes in the C# code. I haven't looked at the dll
[11:17]  Justin.Clark-Casey This is the exception I observed about 20 mins ago on a two region simulator startup btw ->
[11:18]  Justin.Clark-Casey robert: ah right
[11:18]  Justin.Clark-Casey I'm,not sure the code in opensim-libs actually corresponds exactly to warp3d.dll which is a bad thing
[11:18]  BlueWall Slade is Online
[11:18]  Robert Adams: That is a new one Justin. Make a Mantis out of it and assign it to me. Something is not getting locked
[11:18]  Justin.Clark-Casey ok, thanks
[11:19]  Robert Adams: Or, more likely, the threading has been changed in OpenSim so you're finding lock problems that have always been there :)
[11:20]  Robert Adams: should be easy to find from that call stack
[11:21]  Justin.Clark-Casey yes
[11:21]  Mata Hari: path-too-long error :p
[11:23]  Justin.Clark-Casey anyway, any other opensim topics today?
[11:23]  Simulator Version v0.5 shouts: OpenSim 0.8.1 Dev          3a87cce: 2014-07-22 18:04:28 +0100 (Unix/Mono)
[11:23]  Cuteulala Artis is Online
[11:23]  Kayaker Magic: What are you working on now Justin?
[11:23]  Justin.Clark-Casey right now, performance under load
[11:24]  Mata Hari: nothing special that's new...just more instances of old
[11:24]  Billy.Bradshaw mata, I deleted the calling card for the friend I declined but the offers continued, so I accepted in the end.
[11:24]  Richardus Raymaker: and then its possible safe to delete
[11:24]  Justin.Clark-Casey I have a bunch of xperimental tweaks for the test after to see if they help. I believe the issue atm is just sheer amount of agent updates but it can be quite hard to tell
[11:25]  Robert Adams: there are queues on top of queues and threads competing with threads... a hard problem to unravel
[11:25]  Justin.Clark-Casey very
[11:25]  Dahlia Trimble is Online
[11:25]  Justin.Clark-Casey it has grown very organically
[11:26]  Justin.Clark-Casey half the time I'm surprised it works as well as it does
[11:26]  Kayaker Magic: lol
[11:26]  Mata Hari: I was glad to see that you were able to reproduce the fetch issue on your own sim...misery loves company
[11:26]  Justin.Clark-Casey yeah. but only for the first time, which was very odd
[11:26]  Justin.Clark-Casey I really need to consistently reproduce it to get to tht ebottom of things
[11:27]  Mata Hari: I changed the timeout to the much shorter one....still not a single message to log from it
[11:27]  Justin.Clark-Casey I find that extremely odd - the request should simply timeout whatever happens
[11:28]  Justin.Clark-Casey mata: remind me, are you on windows or mono?
[11:28]  Mata Hari: win 7 64 and have .net 4.5
[11:28]  Justin.Clark-Casey and you run at debug log level?
[11:29]  Mata Hari: usually, yes
[11:29]  Dahlia Trimble: hi
[11:29]  Justin.Clark-Casey hi dahlia
[11:30]  Nebadon Izumi: hello dahlia
[11:30]  Justin.Clark-Casey mata: yeah, I just don't understand why requests don't timeout. THere's no way that should fail
[11:31]  Mata Hari: maybe it's a case of them not timing out but instead somehow failing to then move on to the next
[11:31]  Justin.Clark-Casey aside from that, I was thinking whether it might be some problematic folder, but I just don't think they refer to each otherin a way that could cause any kind of loop
[11:31]  Mata Hari: so it jst keeps repeating the same query over and over again
[11:32]  Justin.Clark-Casey that should still send the processed count stats up
[11:32]  Billy.Bradshaw If you disable HTTP Inventory in the viewer that would remove the http server??
[11:32]  Richardus Raymaker: Mata did you cleared viewer cahce already for test ?
[11:32]  Justin.Clark-Casey billy: yes, it would force fallback to older udp mechanisms which aren't throttled in the same way
[11:32]  Mata Hari: I try to avoid clearning cache if I can
[11:33]  Justin.Clark-Casey mata: maybe the queueing code is something to look at thought - I've never been very happy with the home-baked mechanisms used
[11:33]  Mata Hari: I've toggles HTTP fatch too...same issue either way
[11:33]  Justin.Clark-Casey .net 4 has better sdk classes for this kind of thing
[11:33]  Justin.Clark-Casey I would be extremely surprised if that's the case - the udp is an entirely different mechanism
[11:34]  Justin.Clark-Casey are you absolutely sure your viewer is falling back?
[11:34]  Richardus Raymaker: after i upgraded kokua i copuld not login as example, cleared cache and then it worked. alsoo possible just stupid bad luck the first time
[11:34]  Nebadon Izumi: network probably is a factor
[11:35]  Billy.Bradshaw I have seen the slowing under both configs, but never the stall
[11:35]  Nebadon Izumi: i suspect something between simulators and robust
[11:35]  Mata Hari: I have no idea how to know for sure what my viewer is or isn't doing
[11:35]  Nebadon Izumi: because most of these people who have inventory issues
[11:35]  Nebadon Izumi: they go to a plaza and they are fine
[11:35]  Justin.Clark-Casey on simulator console "debug lludp packet 1" will show the less common packets received by the server
[11:35]  Richardus Raymaker: Mata what viewer ?
[11:35]  Nebadon Izumi: plazas are in same data center as robust servers
[11:35]  Nebadon Izumi: its like zero latency
[11:36]  Justin.Clark-Casey so probably in this case "debug lludp packet 1 --default" and then log in with viewer. If iot's using udp you should see a lot of inentory packets
[11:36]  Mata Hari: FS 4.6.5 (or whatever that most recent one is)
[11:36]  Nebadon Izumi: if it was between viewer and simulator they would experience it everywhere
[11:36]  Justin.Clark-Casey it could be windows behave slightly differently than mono here....
[11:36]  Richardus Raymaker: i would try sinmgularity. FS not the best one. for me the latest feels a bit more buggy anyway and thats in sl
[11:36]  Mata Hari: I have never had fetch fail for me on a plaza
[11:36]  Justin.Clark-Casey ai austin is windows as well afair
[11:36]  Nebadon Izumi: like Ai for instance is in UK
[11:36]  Nebadon Izumi: and plazas are in Texas
[11:37]  Nebadon Izumi: and its ok from his viewer to the plazas, but not his regions to robust in same location
[11:37]  Mata Hari: in fact usually by the time my viewer shows me the screen the fetch is finished
[11:37]  Nebadon Izumi: the only thing I can think is on OSgrid
[11:37]  Nebadon Izumi: we throttle any single IP address to no more than 120 requests per second
[11:37]  Nebadon Izumi: with nginx
[11:38]  Nebadon Izumi: could the simulators be exceeding that to Robust?
[11:38]  Justin.Clark-Casey none of this explains the complete lack of timeout for the requests
[11:38]  Justin.Clark-Casey even without any setting changes, they should always teimout after 100 secs and report that fact to the console
[11:38]  Nebadon Izumi: ya nginx gives a clear erorr message when you are being throttled
[11:38]  Justin.Clark-Casey which should apply to both windows and mono as that's part of the sdk spec
[11:38]  Mata Hari: I just checked to be sure.... <root>
 <level value="DEBUG" />
 <appender-ref ref="Console" />
 <appender-ref ref="LogFileAppender" />
[11:38]  Nebadon Izumi: no timeout to me says there is no response at all being processed?
[11:39]  Justin.Clark-Casey even then, timeout should occur
[11:39]  BlueWall Slade: the console shows the requests very far apart here.
[11:39]  BlueWall Slade: maybe 2-3 / minute
[11:40]  Nebadon Izumi: ya and those are always between simulator and robust right?
[11:40]  Nebadon Izumi: some how high latency disrupts that, very odd
[11:40]  BlueWall Slade: well, the testing we were doing a couple of nights ago here in ?OSG
[11:40]  Nebadon Izumi: ya
[11:40]  Mata Hari: so if you want me to do something extra/different other than have it set to DEBUG in opensim.exe.config I'd be happy to...even have it spew data for a test or two if that owuld help
[11:41]  Nebadon Izumi: i also check the osgrid nginx timeout period
[11:41]  BlueWall Slade: yep, seems like a slow connectiong to the sim triggers something bad
[11:41]  Nebadon Izumi: its pretty long like 300 seconds or something
[11:41]  Robert Adams: be sure to minimize the log window or the scrolling slows everything down
[11:41]  Justin.Clark-Casey mata: tbh, that bug eport is already hard to pick through. I would rather have a think about why timeout could possibly be failing altogether
[11:41]  Justin.Clark-Casey that's an extreme anomoly
[11:41]  Nebadon Izumi: since sometimes requests can genuinely take long time
[11:41]  Justin.Clark-Casey but I probably won't get time again until later this week
[11:41]  Mata Hari nods
[11:42]  Mata Hari: well you know where to find me if you need a guinea pig
[11:42]  Justin.Clark-Casey mata: do you have any way to test this stuff under mono?
[11:42]  Mata Hari: and Ai would I'm sure be happy to test too
[11:42]  BlueWall Slade: is there any reason that we can't get the inventory as a blob?
[11:42]  Justin.Clark-Casey requests come in fromthe viewer one by one
[11:42]  Mata Hari: er....I could probably set up something on my old box
[11:42]  BlueWall Slade: any good reason to get it serialy
[11:42]  Justin.Clark-Casey so you would have to bundle those up
[11:42]  BlueWall Slade: ok
[11:42]  Mata Hari: it's running ubuntu 10.something
[11:43]  Justin.Clark-Casey mata: I'm wondering if it could possibly be a windows issue as that's what you and ai have in common. Or that could just be a coincidence
[11:43]  BlueWall Slade: that is a good-enough reason :S
[11:43]  Richardus Raymaker: Then why is my windows with singularity seems to have no poroiblems ?
[11:43]  Mata Hari: the other box is pretty underpowered though....Core2Duo with only 4GN RAM
[11:43]  Justin.Clark-Casey I think ubuntu 10 has a high enough versino of mono (2.10) but can't remember for sure
[11:44]  Justin.Clark-Casey richardus: it's not a question of opsys for the viewer but for the simulator
[11:44]  Richardus Raymaker: k,
[11:44]  Justin.Clark-Casey mata: do I remember youi saying you also had the issue on a private grid?
[11:44]  Justin.Clark-Casey or rather one where you run your own services?
[11:45]  Mata Hari: no, I never had that issue when I was hosting a standalone instead
[11:45]  Mata Hari: and I never had that issue under 0.7.6
[11:45]  Mata Hari: in fact I never had that issue until around Christmastime
[11:45]  Justin.Clark-Casey ok, another thing to do may be to put in a debug switch that logs what inve requests are being made and see if it always 'sticks' at the same one
[11:46]  Mata Hari: this time last year I was crashing like crazy with swapping meshes around and never had troub le logging into my own regions
[11:46]  Justin.Clark-Casey bluewall: actually, now I think about it, the viewer does request a bunch of folders at once so really taht could be bunched. But it means a new inventory service request point, etc.
[11:46]  Justin.Clark-Casey and I'm not sure if it would actually help in this case
[11:47]  BlueWall Slade: maybe a good future goal then
[11:47]  Mata Hari: something in OSG throttling could easily be the issue though since both my viewer and regions are the same IP externally
[11:47]  Justin.Clark-Casey I was thinking some caching might be possible simulator side but now I'm not quite so sure
[11:47]  BlueWall Slade: those serial requests are expensive
[11:48]  Mata Hari: so if both are making lots of requests there'd be a burst of them
[11:48]  Justin.Clark-Casey mata: if that's the case tyou should still see timeout
[11:48]  Justin.Clark-Casey as the simulator doesn't fulfill the requiest in time or perhaps sends back a 503....
[11:48]  Justin.Clark-Casey er sorry, service
[11:48]  BlueWall Slade: it seems to hit a wall, then making requests at about 2-3 / minute
[11:48]  Mata Hari: hm....well the log and the console are both blissfully empty when it's happening
[11:48]  BlueWall Slade: I'm not sure though if they are repeat requests
[11:48]  Justin.Clark-Casey bluewall: whjich start are you looking at?
[11:49]  Mata Hari: the only error message that gets thrown is my CTD if I shut down the sim without wating for at least 10+ mimutes after I log out
[11:49]  BlueWall Slade: just the console log
[11:49]  Justin.Clark-Casey bluewall: the service log?
[11:49]  Justin.Clark-Casey bluewall: I don't understand since requests aren't loggedatm
[11:49]  BlueWall Slade: just the console
[11:50]  BlueWall Slade: it was reporting calls to the OSG inventory server
[11:50]  Justin.Clark-Casey I see way more than 2-3 per min
[11:50]  BlueWall Slade: hmmm, maybe the slow request logging
[11:51]  Justin.Clark-Casey Though on the robust console for osgrid you can see that some requests take a long time
[11:51]  Justin.Clark-Casey though at the moment not longer than 5 seconds
[11:51]  Justin.Clark-Casey usually/. Could even be poorly optimized db queries...
[11:51]  Robert Adams: I are going to run... take care all. See you in Mantis land
[11:51]  Justin.Clark-Casey bye robert
[11:51]  Shez Oyen: BB Robert
[11:51]  Mata Hari: bye Robert
[11:51]  BlueWall Slade: I will try to fire it up later and let you know
[11:52]  BlueWall Slade: bye Robert
[11:52]  Richardus Raymaker: bye robert
[11:52]  Dahlia Trimble: bye Robert
[11:54]  Nebadon Izumi: I have about 1/2 my bots logged in Justin
[11:54]  Nebadon Izumi: im sitting them down today
[11:54]  Nebadon Izumi: so taking me a bit longer
[11:54]  Justin.Clark-Casey ok. I'm concenttrating on udp traffic reduction measures today
[11:54]  Richardus Raymaker: let you guys and girls playing. need to go. bye
[11:54]  Mata Hari: where to today?
[11:55]  Richardus Raymaker is Offline
[11:55]  Mata Hari: (for load test)
[11:55]  BlueWall Slade: Bye RiRa
[11:55]  Nebadon Izumi: same place
[11:55]  Mata Hari: kk
[11:55]  Nebadon Izumi: location is Keynote 1
[11:55]  jazzy chatterbox: i need to run, good luck all you Brilliant people :)
[11:55]  Nebadon Izumi: thogh if you have alts you can spread them evenly over Keynote 2, Keynote 3, and Keynote 4
[11:56]  Shez Oyen: See ya Jazzy
[11:56]  Pathfinder.Lester seeya on Keynote, folks
[11:56]  BlueWall Slade: bye jazzy
[11:56]  Nebadon Izumi: yea we can start heading over now
[11:56]  Nebs Metal Bar Stool v1.5 (w/sit & launch): Goodbye..
[11:56]  Nebadon Izumi whispers: see you all there :)
[11:58]  Justin.Clark-Casey ok
[11:58]  Stephen Xootfly is Online
Personal tools
About This Wiki