Chat log from the meeting on 2016-05-24

[11:02] Robert Adams: I've been seeing the sit position discussion... I thought there were now 2 modes: SL compatible (default) and OpenSim legacy
[11:02] Andrew Hellershanks: Crashed the viewer again by switching workspaces. grrr
[11:02] Robert Adams: is that not true any more?
[11:02] Gavin.Hird @grid.xmir.org:8002: at least I can sit normal again at XMIR with the code change that was pushed last night
[11:03] Gavin.Hird @grid.xmir.org:8002: so that's good
[11:03] Andrew Hellershanks: Robert, there is a big discussion going on as to whether master is compatible with SL, 0.8, or neither.
[11:04] Andrew Hellershanks: There is still the setting where you can enable legacy sit
[11:06] Gavin.Hird @grid.xmir.org:8002: For those who have not seen it, here is a link to the changes that LL will do to the viewer code and their infrastructure that effectively removed UDP. You can read it at your own leasure. http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Release/4.0.6.315555
[11:06] Robert Adams: so the problem is that the default sit position is possibly not like SL
[11:07] Gavin.Hird @grid.xmir.org:8002: I thought the code push last night was going to fix that?
[11:07] Gavin.Hird @grid.xmir.org:8002: for 0.8.x it is not identical AFAIK
[11:08] Gavin.Hird @grid.xmir.org:8002: What changed with your viewer since you keep crashing Andrew?
[11:09] Robert Adams: humm... sounds like we need some test cases
[11:10] Andrew Hellershanks: I have 3 workspaces in my desktop. I run the viewer in one of them. When I switch workspace it often crashes the viewer.
[11:10] Andrew Hellershanks: I haven't had a reliable viewer in a long time.
[11:11] George Equus: FS seem rock solid
[11:11] Gavin.Hird @grid.xmir.org:8002: I don't think the renderer code knows how to handle that well TBH
[11:11] Andrew Hellershanks: George, try it in Linux and switch workspaces. I bet it will crash eventually.
[11:11] Andrew Hellershanks: Doesn't happen every time.
[11:11] George Equus: ahh Linux..
[11:11] George Equus: Not for me
[11:12] Andrew Hellershanks: Gavin, The SL viewer never crashes on me. It is all the third party viewers that are buggy in how they handle workspace changes.
[11:12] Gavin.Hird @grid.xmir.org:8002: I can see that, haha
[11:12] Andrew Hellershanks: Robert, Tests need to be done with SL, OpenSim, and in Avination.
[11:13] James.Atlloud @grid.kitely.com:8002: That sounds amazing - 3 workspaces. Seems like there would be intense memory management in that.
[11:13] Andrew Hellershanks: For the tests to be useful the avatar needs to be the same in height for all 3 grids.
[11:13] Robert Adams: yes... it would be nice to create some reference chairs and run them in all three
[11:14] Andrew Hellershanks: James, it is very handy having multiple workspaces. I used to have 4 but I found I rarely need even 3
[11:14] Robert Adams: there is also the problem of different sized AVs
[11:14] James.Atlloud @grid.kitely.com:8002: cool
[11:14] Andrew Hellershanks: Robert, References avs are also needed
[11:15] James.Atlloud @grid.kitely.com:8002: My last regular use of Linux was 10+ years ago.. lots has changed I'm sure.
[11:15] Gavin.Hird @grid.xmir.org:8002: Linux and OS X both have double buffered frame buffers, while Windows just writes to the screen. So much more is going on behind the scenes when you switch a workspace on those systems..
[11:15] Andrew Hellershanks: James, yes. A lot has changed in those years. I've been mainly running Linux for the past 22 years.
[11:16] Andrew Hellershanks: Gavin, the viewers didn't crash on me in the past.
[11:16] Gavin.Hird @grid.xmir.org:8002: define past? :-)
[11:16] Robert Adams: The Release Notes referenced above mostly says they are not going to use the UDP inventory protocol any more... I believe OS implements all the new Inventory2 protocol (CAPS, etc) so we should be OK
[11:17] Andrew Hellershanks: Gavin, hard to pin down. Viewer code has changed. Video drivers have changed. Viewers may be making more use of features of video cards.
[11:17] Gavin.Hird @grid.xmir.org:8002: we are not OK Robert
[11:17] Gavin.Hird @grid.xmir.org:8002: the code from 4.02 crashes on opensim
[11:17] Robert Adams: how so?
[11:17] Gavin.Hird @grid.xmir.org:8002: their code changes behind the scene are massive
[11:17] Simulator Version v0.5 ruft: OpenSim 0.9.0.0 Dev 057d1b2: 2016-05-16 21:57:19 -0700 (Unix/Mono)
[11:17] Andrew Hellershanks: Gavin, umm... 5 years, maybe? I don't remember how long it's been since I could switch work spaces in a TPV and not crash it.
[11:17] Gavin.Hird @grid.xmir.org:8002: it started in 2014
[11:18] Gavin.Hird @grid.xmir.org:8002: so on a version 1.x viewer?
[11:20] Andrew Hellershanks: Gavin, No idea.
[11:20] Gavin.Hird @grid.xmir.org:8002: it must have been 2009 they released the version 2.0 viewer and it took a while for the TPVs to catch up
[11:21] Andrew Hellershanks: Gavin, this has been going on for quite some time now. I'm not sure how long. At least a year minimum. Probably more like 2 years. Possibly longer.
[11:22] Andrew Hellershanks: Robert, what's new and exciting in the world of BulletSim physics?
[11:23] Gavin.Hird @grid.xmir.org:8002: there was a lot of instability introduced in the code the summer that OSG crashed
[11:23] Robert Adams: I haven't been working on BulletSim... I'm looking at adding Bullet3 since it has hardware acceleration
[11:24] Andrew Hellershanks: Does this mean we are going to be seeing another (new?) physics engine for OpenSim soon?
[11:24] Gavin.Hird @grid.xmir.org:8002: after that the code changes in the LL code that everyone uses have been massive
[11:24] Robert Adams: but I'm mostly interested in fixing master since no one has been fixing the Big Merge problems
[11:24] Kayaker Magic: YAPE Yet Another Physics Engine
[11:24] James.Atlloud @grid.kitely.com:8002: good name
[11:24] Andrew Hellershanks: Kayaker, just what I was thinking. :)
[11:25] Gavin.Hird @grid.xmir.org:8002: hardware acceleration is most likely a good thing!
[11:25] Robert Adams: very true... I've been thinking that physics could be done a lot differently... but that's a whole different discussion
[11:25] Andrew Hellershanks: I've forgotten who was working on BS other than you, Robert. I haven't looked at it in a while.
[11:26] Kayaker Magic: But physics is done on the server, so the acceleration would have to be done there, not in the viewer or the client computers.
[11:26] Andrew Hellershanks: Is Bullet3 available in a public repo yet or is it not that far along in development at this stage?
[11:27] Robert Adams: it's on github
[11:27] Robert Adams: https://github.com/bulletphysics/bullet3/
[11:27] Gavin.Hird @grid.xmir.org:8002: is there any reason why some of the physics cannot be done in the clinet, Robert?
[11:28] Andrew Hellershanks: Robert, ty
[11:28] Andrew Hellershanks: Gavin, I can see one of the problems with client side physics is getting the updates to other people in the region. Imagine if you just broken through a wall of stacked prims?
[11:28] Robert Adams: no reason to not do some on client.... some games and other VMs (like Eve Online) do it on both
[11:30] Gavin.Hird @grid.xmir.org:8002: often the avatar only collides with objects in the path or ground that does not affect anyone else
[11:30] Andrew Hellershanks: Gavin, right. Most of the time that would be fine.
[11:31] Kayaker Magic: I saw something about a shooter videogame recently that had a rule that if you shot someone and later updates showed that they had moved before you shot them, you took a hit anyway.
[11:31] Andrew Hellershanks: Robert, other than hardware acceleration, are there any other differences between BS and B3 worth noting?
[11:31] Andrew Hellershanks: Do you need people to help test it?
[11:32] Robert Adams: Bullet3 is supposed to be quicker than Bullet2 even when not doing hardware acceleration
[11:33] Robert Adams: Bullet2 is very slow in mesh-to-mesh collisions which is the main performance problem with BulletSim
[11:33] Andrew Hellershanks: nice
[11:33] Sheera.Khan @hypergrid.org:8002: Does the server need a special CPU/GPU for the hardware acceleration?
[11:33] Robert Adams: Sheera: yes
[11:33] Andrew Hellershanks: CPU or GPU?
[11:34] George Equus: GPU
[11:34] Robert Adams: acceleration interfacs like OpenCV (used by Bullet3) have drivers for GPUs (like NVidia's CUDA) as well as versions for Intel's wide instructions (SSE3, ...)
[11:35] Kayaker Magic: Many people (like Ilan at Kitely) tell their users not to use BulletSim because it has occasional runnaway CPU hogging behavior. Any progress on identifying that?
[11:35] Gavin.Hird @grid.xmir.org:8002: most datacenter servers don't have much GPU to write home about
[11:35] Robert Adams: NVidia wants every server to have one of their cards :)
[11:35] Andrew Hellershanks: Robert, thought so. Just wanted to clarify that.
[11:35] George Equus: We are talking Graphics cards here right? GPU
[11:35] Andrew Hellershanks: yes
[11:35] Robert Adams: Intel is also adding stripped down GPUs to their servers (the compute part and not the display portion)
[11:35] Gavin.Hird @grid.xmir.org:8002: to triple power and cooling requirements :-))
[11:36] Andrew Hellershanks: Robert, I went with nVidia a while back so I could try out CUDA
[11:36] Robert Adams: But my OSGrid regions run on a desktop system so I have a GPU for my regions :)
[11:37] Andrew Hellershanks: Gavin, Some data centers can be using headless servers that would have no GPU.
[11:37] Gavin.Hird @grid.xmir.org:8002: for anyone who is not hosting in a datacenter it will probably be a great
[11:37] Andrew Hellershanks: If Bullet3 is still faster without a GPU available that is still a plus.
[11:38] Gavin.Hird @grid.xmir.org:8002: Andres, most of them are barely able to drive a terminal session
[11:38] Andrew Hellershanks: Gavin, :)
[11:38] Robert Adams: Intel's new generation serves have the compute portion of a GPU as well as an ASIC... servers are getting much more completed
[11:38] Andrew Hellershanks nods
[11:39] Andrew Hellershanks: Robert, any numbers as to how much faster the new engine is compared to BS, both with and without use of GPU?
[11:39] Kayaker Magic: I wonder if the BulletSim CPU hogging behavior has to do with the problem of objects that change their physics shape. ubODE has a bad case of that, see mantis 7853
[11:40] Gavin.Hird @grid.xmir.org:8002: Kayaker, the CPU increase between 0.8.x and 0.9 is the same for both bullet and ubODE as far as I can measure it
[11:41] Andrew Hellershanks: Gavin, you are getting a 50% CPU increase with master when using either physics engine?
[11:41] Gavin.Hird @grid.xmir.org:8002: yup
[11:42] Kayaker Magic: This is not a 0.8 0.9 issue. Bullet sometimes goes crazy and hogs the CPU in 0.8, so Kitely tells everyone DON'T USE BULLETSIM
[11:42] Andrew Hellershanks: Gavin, have you tried turning off physics to eliminate it as the source of the CPU increase?
[11:42] Gavin.Hird @grid.xmir.org:8002: I have actually never seen that
[11:42] Gavin.Hird @grid.xmir.org:8002: I have not
[11:42] Robert Adams: it's unlikely I'll attack Bullet3 in the short term... I think my first tasks will be making 0.9 viable
[11:42] Andrew Hellershanks: eliminate or confirm physics as the source of the CPU increase.
[11:43] Andrew Hellershanks: Gavin, It would be worth trying. I haven't gotten any more information as to why there would be any more CPU usage with master. Certainly not 50% more.
[11:43] Kayaker Magic: I'm just hypothesizing that it is content related and the script in mantis 7853 might be a way to make it happen. Robert you should try that out on BulletSim.
[11:43] Gavin.Hird @grid.xmir.org:8002: region to region cross chatter
[11:44] Andrew Hellershanks: Robert, Is there much that needs to be done to BS for 0.9?
[11:44] Gavin.Hird @grid.xmir.org:8002: and as I said before observed when you have more than one simulator on the same IP
[11:44] Kayaker Magic: I think we are talking about two different CPU increases here.
[11:44] Gavin.Hird @grid.xmir.org:8002: or most pronounced when
[11:45] Andrew Hellershanks: Kayaker, Gavin has stated he sees a 50% increase in CPU load with git master of OpenSim.
[11:45] Robert Adams: It doesn't seem so... most of the work is correcting what git did when it merged the code bases
[11:45] Andrew Hellershanks: Robert, ok.
[11:47] Dahlia Trimble: I heard there were cookies here
[11:49] George Equus: viewers got those Gavin
[11:49] Andrew Hellershanks: Gavin, I would be surprised if region to region chatter has changed to the extent it is causing such a large increase in CPU usage. With physics disabled you can determine if physics is contributing to the load or not.
[11:49] Gavin.Hird @grid.xmir.org:8002: there is something there in the web browser
[11:49] Robert Adams: the "unknown user" bug is still around
[11:49] Dahlia Trimble: lol I see
[11:50] George Equus: Yes Gavin :)
[11:50] Gavin.Hird @grid.xmir.org:8002: Andrew, the code changes in Framework/scene between 0.8.4 and 0.9 was massive
[11:50] Gavin.Hird @grid.xmir.org:8002: same with entity transfer
[11:50] Robert Adams: the region-to-region protocols are pretty adhoc... wouldn't surprise me if there was a problem
[11:50] Andrew Hellershanks: Robert, Either that or someone is messing with us by being cute in the select of their avatar name.
[11:51] Jeff.Kelley @88.161.20.136:9000: there is worse than the unknown user : the mystery user, has no name on friend list, has no name over his head
[11:51] Robert Adams: I've had a problem that a particular alt I have causes continuous rebakes in 0.9
[11:51] Robert Adams: that causes CPU load at both ends
[11:51] Kayaker Magic: Ha! I built a particle cloud named Ruth once...
[11:53] Dahlia Trimble: speaking of clouds, what was were the keystrokes to rebake? I cant remember
[11:53] Andrew Hellershanks: Ctrl-Alt-R ?
[11:53] George Equus: Y
[11:53] Dahlia Trimble: tried that
[11:53] Gavin.Hird @grid.xmir.org:8002: region/framework/scenes
[11:54] Andrew Hellershanks: We need more of those load tests that were held at the OSCC regions.
[11:57] Gavin.Hird @grid.xmir.org:8002: with mesh clothes what people seem to wear can get really interesting at times, hehe
[11:57] Andrew Hellershanks: Robert, Do you have a problem with client side or server side baking with that avatar?
[11:58] Dahlia Trimble: I remember when I was working on IdealistViewer years ago, the avatar looked fine to me but appeared topless to everyone else and I kept getting lots of friend requests whenever I would log in to Wright Plaza
[11:58] Robert Adams: I'm as far as noticing that the simulator spews continuous rebake debug log messages
[11:58] Andrew Hellershanks: ok.
[11:59] Gavin.Hird @grid.xmir.org:8002: what viewer are you on Dahlia
[11:59] Dahlia Trimble: Singularity
[12:00] Gavin.Hird @grid.xmir.org:8002: we have just taken some code changes from LL that trottles rebake requests after some attempts
[12:00] Andrew Hellershanks: Robert, it would be worth noting at some point when you take a break from your work on the physics engine.
[12:00] Gavin.Hird @grid.xmir.org:8002: after some failed attempts that is
[12:00] Dahlia Trimble: didnt LL get rid of all the rebake stuff when they went to server side baking?
[12:01] Gavin.Hird @grid.xmir.org:8002: I am talking about server side rebaking
[12:01] Dahlia Trimble: oh opensim has that now?
[12:01] Gavin.Hird @grid.xmir.org:8002: sure
[12:01] Dahlia Trimble: nice
[12:01] Sheera.Khan @hypergrid.org:8002: well, we would have xbakes for that matter... But it doesn't help in all cases either
[12:01] Sheera.Khan @hypergrid.org:8002: I see Gavin grey i.e.
[12:02] Gavin.Hird @grid.xmir.org:8002: I am grey, haha
[12:02] Sheera.Khan @hypergrid.org:8002: ah ^^
[12:03] Andrew Hellershanks: SSB may need to be enabled in the grid. I haven't looked that closely at that recently.
[12:03] Andrew Hellershanks: wow. The hour has gone by. Any last minute items before we wrap things up?
[12:04] Sheera.Khan @hypergrid.org:8002: for SSB you might want to look for xbakes in OpenSim.ini
[12:05] Andrew Hellershanks: yes, that is the settings I was thinking about.
[12:05] Gavin.Hird @grid.xmir.org:8002: you just have to turn it on in Robust
[12:05] Gavin.Hird @grid.xmir.org:8002: it puts some load on the server though and storage
[12:05] Gavin.Hird @grid.xmir.org:8002: if the grid is busy of course
[12:06] Andrew Hellershanks: Gavin, there you go. That's some of your CPU load increase if you have SSB enabled. :)
[12:06] Gavin.Hird @grid.xmir.org:8002: no
[12:06] Gavin.Hird @grid.xmir.org:8002: Robust is running an a separate machine without any regions
[12:06] Sheera.Khan @hypergrid.org:8002: XBakes is enabled by default I think
[12:06] Gavin.Hird @grid.xmir.org:8002: and my grid is not busy
[12:06] Gavin.Hird @grid.xmir.org:8002: :-))
[12:07] Andrew Hellershanks: There is an Xbakes section in OpenSim.ini
[12:08] Andrew Hellershanks: ah, that just points to the Robust server that is handling it.
[12:08] Gavin.Hird @grid.xmir.org:8002: exactly
[12:08] Dahlia Trimble: well I was just popping in to say hi. Nice to see you all :)
[12:09] Andrew Hellershanks: Good to see you again, Dahlia.
[12:09] Andrew Hellershanks: It is time to end this meeting for today. Thank you all for coming.