Chat log from the meeting on 2016-05-24

[11:02] Robert Adams: I've been seeing the sit position discussion... I thought there were now 2 modes: SL compatible (default) and OpenSim legacy [11:02] Andrew Hellershanks: Crashed the viewer again by switching workspaces. grrr [11:02] Robert Adams: is that not true any more? [11:02] Gavin.Hird @grid.xmir.org:8002: at least I can sit normal again at XMIR with the code change that was pushed last night [11:03] Gavin.Hird @grid.xmir.org:8002: so that's good [11:03] Andrew Hellershanks: Robert, there is a big discussion going on as to whether master is compatible with SL, 0.8, or neither. [11:04] Andrew Hellershanks: There is still the setting where you can enable legacy sit [11:06] Gavin.Hird @grid.xmir.org:8002: For those who have not seen it, here is a link to the changes that LL will do to the viewer code and their infrastructure that effectively removed UDP. You can read it at your own leasure. http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Release/4.0.6.315555 [11:06] Robert Adams: so the problem is that the default sit position is possibly not like SL [11:07] Gavin.Hird @grid.xmir.org:8002: I thought the code push last night was going to fix that? [11:07] Gavin.Hird @grid.xmir.org:8002: for 0.8.x it is not identical AFAIK [11:08] Gavin.Hird @grid.xmir.org:8002: What changed with your viewer since you keep crashing Andrew? [11:09] Robert Adams: humm... sounds like we need some test cases [11:10] Andrew Hellershanks: I have 3 workspaces in my desktop. I run the viewer in one of them. When I switch workspace it often crashes the viewer. [11:10] Andrew Hellershanks: I haven't had a reliable viewer in a long time. [11:11] George Equus: FS seem rock solid [11:11] Gavin.Hird @grid.xmir.org:8002: I don't think the renderer code knows how to handle that well TBH [11:11] Andrew Hellershanks: George, try it in Linux and switch workspaces. I bet it will crash eventually. [11:11] Andrew Hellershanks: Doesn't happen every time. [11:11] George Equus: ahh Linux.. [11:11] George Equus: Not for me [11:12] Andrew Hellershanks: Gavin, The SL viewer never crashes on me. It is all the third party viewers that are buggy in how they handle workspace changes. [11:12] Gavin.Hird @grid.xmir.org:8002: I can see that, haha [11:12] Andrew Hellershanks: Robert, Tests need to be done with SL, OpenSim, and in Avination. [11:13] James.Atlloud @grid.kitely.com:8002: That sounds amazing - 3 workspaces. Seems like there would be intense memory management in that. [11:13] Andrew Hellershanks: For the tests to be useful the avatar needs to be the same in height for all 3 grids. [11:13] Robert Adams: yes... it would be nice to create some reference chairs and run them in all three [11:14] Andrew Hellershanks: James, it is very handy having multiple workspaces. I used to have 4 but I found I rarely need even 3 [11:14] Robert Adams: there is also the problem of different sized AVs [11:14] James.Atlloud @grid.kitely.com:8002: cool [11:14] Andrew Hellershanks: Robert, References avs are also needed [11:15] James.Atlloud @grid.kitely.com:8002: My last regular use of Linux was 10+ years ago.. lots has changed I'm sure. [11:15] Gavin.Hird @grid.xmir.org:8002: Linux and OS X both have double buffered frame buffers, while Windows just writes to the screen. So much more is going on behind the scenes when you switch a workspace on those systems.. [11:15] Andrew Hellershanks: James, yes. A lot has changed in those years. I've been mainly running Linux for the past 22 years. [11:16] Andrew Hellershanks: Gavin, the viewers didn't crash on me in the past. [11:16] Gavin.Hird @grid.xmir.org:8002: define past? :-) [11:16] Robert Adams: The Release Notes referenced above mostly says they are not going to use the UDP inventory protocol any more... I believe OS implements all the new Inventory2 protocol (CAPS, etc) so we should be OK [11:17] Andrew Hellershanks: Gavin, hard to pin down. Viewer code has changed. Video drivers have changed. Viewers may be making more use of features of video cards. [11:17] Gavin.Hird @grid.xmir.org:8002: we are not OK Robert [11:17] Gavin.Hird @grid.xmir.org:8002: the code from 4.02 crashes on opensim [11:17] Robert Adams: how so? [11:17] Gavin.Hird @grid.xmir.org:8002: their code changes behind the scene are massive [11:17] Simulator Version v0.5 ruft: OpenSim 0.9.0.0 Dev       057d1b2: 2016-05-16 21:57:19 -0700 (Unix/Mono) [11:17] Andrew Hellershanks: Gavin, umm... 5 years, maybe? I don't remember how long it's been since I could switch work spaces in a TPV and not crash it. [11:17] Gavin.Hird @grid.xmir.org:8002: it started in 2014 [11:18] Gavin.Hird @grid.xmir.org:8002: so on a version 1.x viewer? [11:20] Andrew Hellershanks: Gavin, No idea. [11:20] Gavin.Hird @grid.xmir.org:8002: it must have been 2009 they released the version 2.0 viewer and it took a while for the TPVs to catch up [11:21] Andrew Hellershanks: Gavin, this has been going on for quite some time now. I'm not sure how long. At least a year minimum. Probably more like 2 years. Possibly longer. [11:22] Andrew Hellershanks: Robert, what's new and exciting in the world of BulletSim physics? [11:23] Gavin.Hird @grid.xmir.org:8002: there was a lot of instability introduced in the code the summer that OSG crashed [11:23] Robert Adams: I haven't been working on BulletSim... I'm looking at adding Bullet3 since it has hardware acceleration [11:24] Andrew Hellershanks: Does this mean we are going to be seeing another (new?) physics engine for OpenSim soon? [11:24] Gavin.Hird @grid.xmir.org:8002: after that the code changes in the LL code that everyone uses have been massive [11:24] Robert Adams: but I'm mostly interested in fixing master since no one has been fixing the Big Merge problems [11:24] Kayaker Magic: YAPE Yet Another Physics Engine [11:24] James.Atlloud @grid.kitely.com:8002: good name [11:24] Andrew Hellershanks: Kayaker, just what I was thinking. :) [11:25] Gavin.Hird @grid.xmir.org:8002: hardware acceleration is most likely a good thing! [11:25] Robert Adams: very true... I've been thinking that physics could be done a lot differently... but that's a whole different discussion [11:25] Andrew Hellershanks: I've forgotten who was working on BS other than you, Robert. I haven't looked at it in a while. [11:26] Kayaker Magic: But physics is done on the server, so the acceleration would have to be done there, not in the viewer or the client computers. [11:26] Andrew Hellershanks: Is Bullet3 available in a public repo yet or is it not that far along in development at this stage? [11:27] Robert Adams: it's on github [11:27] Robert Adams: https://github.com/bulletphysics/bullet3/ [11:27] Gavin.Hird @grid.xmir.org:8002: is there any reason why some of the physics cannot be done in the clinet, Robert? [11:28] Andrew Hellershanks: Robert, ty [11:28] Andrew Hellershanks: Gavin, I can see one of the problems with client side physics is getting the updates to other people in the region. Imagine if you just broken through a wall of stacked prims? [11:28] Robert Adams: no reason to not do some on client.... some games and other VMs (like Eve Online) do it on both [11:30] Gavin.Hird @grid.xmir.org:8002: often the avatar only collides with objects in the path or ground that does not affect anyone else [11:30] Andrew Hellershanks: Gavin, right. Most of the time that would be fine. [11:31] Kayaker Magic: I saw something about a shooter videogame recently that had a rule that if you shot someone and later updates showed that they had moved before you shot them, you took a hit anyway. [11:31] Andrew Hellershanks: Robert, other than hardware acceleration, are there any other differences between BS and B3 worth noting? [11:31] Andrew Hellershanks: Do you need people to help test it? [11:32] Robert Adams: Bullet3 is supposed to be quicker than Bullet2 even when not doing hardware acceleration [11:33] Robert Adams: Bullet2 is very slow in mesh-to-mesh collisions which is the main performance problem with BulletSim [11:33] Andrew Hellershanks: nice [11:33] Sheera.Khan @hypergrid.org:8002: Does the server need a special CPU/GPU for the hardware acceleration? [11:33] Robert Adams: Sheera: yes [11:33] Andrew Hellershanks: CPU or GPU? [11:34] George Equus: GPU [11:34] Robert Adams: acceleration interfacs like OpenCV (used by Bullet3) have drivers for GPUs (like NVidia's CUDA) as well as versions for Intel's wide instructions (SSE3, ...) [11:35] Kayaker Magic: Many people (like Ilan at Kitely) tell their users not to use BulletSim because it has occasional runnaway CPU hogging behavior. Any progress on identifying that? [11:35] Gavin.Hird @grid.xmir.org:8002: most datacenter servers don't have much GPU to write home about [11:35] Robert Adams: NVidia wants every server to have one of their cards :) [11:35] Andrew Hellershanks: Robert, thought so. Just wanted to clarify that. [11:35] George Equus: We are talking Graphics cards here right? GPU [11:35] Andrew Hellershanks: yes [11:35] Robert Adams: Intel is also adding stripped down GPUs to their servers (the compute part and not the display portion) [11:35] Gavin.Hird @grid.xmir.org:8002: to triple power and cooling requirements :-)) [11:36] Andrew Hellershanks: Robert, I went with nVidia a while back so I could try out CUDA [11:36] Robert Adams: But my OSGrid regions run on a desktop system so I have a GPU for my regions :) [11:37] Andrew Hellershanks: Gavin, Some data centers can be using headless servers that would have no GPU. [11:37] Gavin.Hird @grid.xmir.org:8002: for anyone who is not hosting in a datacenter it will probably be a great [11:37] Andrew Hellershanks: If Bullet3 is still faster without a GPU available that is still a plus. [11:38] Gavin.Hird @grid.xmir.org:8002: Andres, most of them are barely able to drive a terminal session [11:38] Andrew Hellershanks: Gavin, :) [11:38] Robert Adams: Intel's new generation serves have the compute portion of a GPU as well as an ASIC... servers are getting much more completed [11:38] Andrew Hellershanks nods [11:39] Andrew Hellershanks: Robert, any numbers as to how much faster the new engine is compared to BS, both with and without use of GPU? [11:39] Kayaker Magic: I wonder if the BulletSim CPU hogging behavior has to do with the problem of objects that change their physics shape. ubODE has a bad case of that, see mantis 7853 [11:40] Gavin.Hird @grid.xmir.org:8002: Kayaker, the CPU increase between 0.8.x and 0.9 is the same for both bullet and ubODE as far as I can measure it [11:41] Andrew Hellershanks: Gavin, you are getting a 50% CPU increase with master when using either physics engine? [11:41] Gavin.Hird @grid.xmir.org:8002: yup [11:42] Kayaker Magic: This is not a 0.8 0.9 issue. Bullet sometimes goes crazy and hogs the CPU in 0.8, so Kitely tells everyone DON'T USE BULLETSIM [11:42] Andrew Hellershanks: Gavin, have you tried turning off physics to eliminate it as the source of the CPU increase? [11:42] Gavin.Hird @grid.xmir.org:8002: I have actually never seen that [11:42] Gavin.Hird @grid.xmir.org:8002: I have not [11:42] Robert Adams: it's unlikely I'll attack Bullet3 in the short term... I think my first tasks will be making 0.9 viable [11:42] Andrew Hellershanks: eliminate or confirm physics as the source of the CPU increase. [11:43] Andrew Hellershanks: Gavin, It would be worth trying. I haven't gotten any more information as to why there would be any more CPU usage with master. Certainly not 50% more. [11:43] Kayaker Magic: I'm just hypothesizing that it is content related and the script in mantis 7853 might be a way to make it happen. Robert you should try that out on BulletSim. [11:43] Gavin.Hird @grid.xmir.org:8002: region to region cross chatter [11:44] Andrew Hellershanks: Robert, Is there much that needs to be done to BS for 0.9? [11:44] Gavin.Hird @grid.xmir.org:8002: and as I said before observed when you have more than one simulator on the same IP [11:44] Kayaker Magic: I think we are talking about two different CPU increases here. [11:44] Gavin.Hird @grid.xmir.org:8002: or most pronounced when [11:45] Andrew Hellershanks: Kayaker, Gavin has stated he sees a 50% increase in CPU load with git master of OpenSim. [11:45] Robert Adams: It doesn't seem so... most of the work is correcting what git did when it merged the code bases [11:45] Andrew Hellershanks: Robert, ok. [11:47] Dahlia Trimble: I heard there were cookies here [11:49] George Equus: viewers got those Gavin [11:49] Andrew Hellershanks: Gavin, I would be surprised if region to region chatter has changed to the extent it is causing such a large increase in CPU usage. With physics disabled you can determine if physics is contributing to the load or not. [11:49] Gavin.Hird @grid.xmir.org:8002: there is something there in the web browser [11:49] Robert Adams: the "unknown user" bug is still around [11:49] Dahlia Trimble: lol I see [11:50] George Equus: Yes Gavin :) [11:50] Gavin.Hird @grid.xmir.org:8002: Andrew, the code changes in Framework/scene between 0.8.4 and 0.9 was massive [11:50] Gavin.Hird @grid.xmir.org:8002: same with entity transfer [11:50] Robert Adams: the region-to-region protocols are pretty adhoc... wouldn't surprise me if there was a problem [11:50] Andrew Hellershanks: Robert, Either that or someone is messing with us by being cute in the select of their avatar name. [11:51] Jeff.Kelley @88.161.20.136:9000: there is worse than the unknown user : the mystery user, has no name on friend list, has no name over his head [11:51] Robert Adams: I've had a problem that a particular alt I have causes continuous rebakes in 0.9 [11:51] Robert Adams: that causes CPU load at both ends [11:51] Kayaker Magic: Ha! I built a particle cloud named Ruth once... [11:53] Dahlia Trimble: speaking of clouds, what was were the keystrokes to rebake? I cant remember [11:53] Andrew Hellershanks: Ctrl-Alt-R ? [11:53] George Equus: Y [11:53] Dahlia Trimble: tried that [11:53] Gavin.Hird @grid.xmir.org:8002: region/framework/scenes [11:54] Andrew Hellershanks: We need more of those load tests that were held at the OSCC regions. [11:57] Gavin.Hird @grid.xmir.org:8002: with mesh clothes what people seem to wear can get really interesting at times, hehe [11:57] Andrew Hellershanks: Robert, Do you have a problem with client side or server side baking with that avatar? [11:58] Dahlia Trimble: I remember when I was working on IdealistViewer years ago, the avatar looked fine to me but appeared topless to everyone else and I kept getting lots of friend requests whenever I would log in to Wright Plaza [11:58] Robert Adams: I'm as far as noticing that the simulator spews continuous rebake debug log messages [11:58] Andrew Hellershanks: ok. [11:59] Gavin.Hird @grid.xmir.org:8002: what viewer are you on Dahlia [11:59] Dahlia Trimble: Singularity [12:00] Gavin.Hird @grid.xmir.org:8002: we have just taken some code changes from LL that trottles rebake requests after some attempts [12:00] Andrew Hellershanks: Robert, it would be worth noting at some point when you take a break from your work on the physics engine. [12:00] Gavin.Hird @grid.xmir.org:8002: after some failed attempts that is [12:00] Dahlia Trimble: didnt LL get rid of all the rebake stuff when they went to server side baking? [12:01] Gavin.Hird @grid.xmir.org:8002: I am talking about server side rebaking [12:01] Dahlia Trimble: oh opensim has that now? [12:01] Gavin.Hird @grid.xmir.org:8002: sure [12:01] Dahlia Trimble: nice [12:01] Sheera.Khan @hypergrid.org:8002: well, we would have xbakes for that matter... But it doesn't help in all cases either [12:01] Sheera.Khan @hypergrid.org:8002: I see Gavin grey i.e. [12:02] Gavin.Hird @grid.xmir.org:8002: I am grey, haha [12:02] Sheera.Khan @hypergrid.org:8002: ah ^^ [12:03] Andrew Hellershanks: SSB may need to be enabled in the grid. I haven't looked that closely at that recently. [12:03] Andrew Hellershanks: wow. The hour has gone by. Any last minute items before we wrap things up? [12:04] Sheera.Khan @hypergrid.org:8002: for SSB you might want to look for xbakes in OpenSim.ini [12:05] Andrew Hellershanks: yes, that is the settings I was thinking about. [12:05] Gavin.Hird @grid.xmir.org:8002: you just have to turn it on in Robust [12:05] Gavin.Hird @grid.xmir.org:8002: it puts some load on the server though and storage [12:05] Gavin.Hird @grid.xmir.org:8002: if the grid is busy of course [12:06] Andrew Hellershanks: Gavin, there you go. That's some of your CPU load increase if you have SSB enabled. :) [12:06] Gavin.Hird @grid.xmir.org:8002: no [12:06] Gavin.Hird @grid.xmir.org:8002: Robust is running an a separate machine without any regions [12:06] Sheera.Khan @hypergrid.org:8002: XBakes is enabled by default I think [12:06] Gavin.Hird @grid.xmir.org:8002: and my grid is not busy [12:06] Gavin.Hird @grid.xmir.org:8002: :-)) [12:07] Andrew Hellershanks: There is an Xbakes section in OpenSim.ini [12:08] Andrew Hellershanks: ah, that just points to the Robust server that is handling it. [12:08] Gavin.Hird @grid.xmir.org:8002: exactly [12:08] Dahlia Trimble: well I was just popping in to say hi. Nice to see you all :) [12:09] Andrew Hellershanks: Good to see you again, Dahlia. [12:09] Andrew Hellershanks: It is time to end this meeting for today. Thank you all for coming.