Chat log from the meeting on 2016-11-08

[11:02] Kayaker.Magic @grid.kitely.com:8002: I re-tested my problems with Ubits llCastRay and got him to look at it again. [11:03] Jak Daniels: is it working better now on Vars? [11:03] Kayaker.Magic @grid.kitely.com:8002: A little better, but my hope is that it will get below 500 ms per call, and we are not there yet. [11:04] James.atLLOUD @hg.osgrid.org: How far from 500ms are you? [11:04] Kayaker.Magic @grid.kitely.com:8002: It is down to 700ms from my tests showing 1300ms before.... [11:04] James.atLLOUD @hg.osgrid.org: whoa - still good improvement [11:04] Jak Daniels: does the v3 raycast only work on ubOde? [11:05] Kayaker.Magic @grid.kitely.com:8002: V3 can be set withoug ubODE [11:05] Kayaker.Magic @grid.kitely.com:8002: Even works in 0.8 [11:05] Jak Daniels: does which physics engine is in use have any bearing on response times? [11:06] Kayaker.Magic @grid.kitely.com:8002: Well, ubODE has it's own llCastray, which I hoped would speed it up because llCastRay and the Physics engine both need a similar model to work. [11:07] Kayaker.Magic @grid.kitely.com:8002: I'm told some versions of llCastRay have to construct a physics model first in order to work. [11:08] Jak Daniels: I just wondered if there might be timing differnces, e.g. with bullet running in it's own thread [11:08] Kayaker.Magic @grid.kitely.com:8002: I havn't tested llCastRay time with both Bullet and ODE... [11:10] Kayaker.Magic @grid.kitely.com:8002: The discouraging thing is that llCastRay on SL or InWorlds is blindingly fast by compaison to OpenSim [11:10] Jak Daniels: does inwor;ds use physx as physics engine? [11:11] Kayaker.Magic @grid.kitely.com:8002: Yes, InWorldz is using Physx [11:11] Kayaker.Magic @grid.kitely.com:8002: And I'm told it has llCastRay integrated into PhysX [11:12] Jak Daniels: And SL uses havoc on the client, maybe those are the differences. Physx I assume runs on a gpu processor for inworldz and havoc runs on the viewer in SL? That might explain the difference [11:12] George Equus: This is ridiculous.... can't someone do something about these sits? [11:13] Kayaker.Magic @grid.kitely.com:8002: No, Physx has a software only mode and I'm told that is what is running on the servers. [11:13] Jak Daniels: just sit on the sofa anywhere.... you'll get one of about 4 poses ;) [11:13] Andrew.Hellershanks @hg.osgrid.org: What's wrong with the sits? [11:13] James.atLLOUD @hg.osgrid.org: Do you have AO George. [11:13] George Equus: All turned off [11:13] George Equus: I'm not the only one having trouble here [11:14] James.atLLOUD @hg.osgrid.org: Looks like it's on in my viewer - very causal stand, eh? [11:14] Sheera Khan: @ George: I try to catch the same seat every time... the one where my sit position fits ^^ [11:14] Kayaker.Magic @grid.kitely.com:8002: I never have problems sitting here. [11:15] Jak Daniels: nor me.... my viewer AO is still on too [11:15] George Equus: Ha! got a "sit" in one out of four tries tonight...   sigh [11:15] Andrew.Hellershanks @hg.osgrid.org: In the past week or two questions have been asked about the OSCC for this year. I received emails yesterday saying that registration is now open for it. [11:15] George Equus: I just did my registration today [11:15] Kayaker.Magic @grid.kitely.com:8002: Just saw the OSCC announcement this morning! [11:16] George Equus: very late this year [11:17] Kayaker.Magic @grid.kitely.com:8002: Ha: Orthopaedic Surgery Center of Clearwater [11:18] James.atLLOUD @hg.osgrid.org: lol [11:18] Jak Daniels: that reminds me, I must go into OSCC and change all the flags and banners. Good think they are scripted or it would be a very long and tedious job ;) [11:18] Andrew.Hellershanks @hg.osgrid.org: The URL is https://www.eventbrite.com/e/opensimulator-community-conference-2016-registration-28824725530 [11:18] Andrew.Hellershanks @hg.osgrid.org: George, it is still about a month off. [11:18] Andrew.Hellershanks @hg.osgrid.org: Saturday, December 10, 2016 at 7:00 AM - Sunday, December 11, 2016 at 7:00 PM (EST) [11:18] Kayaker.Magic @grid.kitely.com:8002: approaching like a freight train IMHO [11:20] Andrew.Hellershanks @hg.osgrid.org: :) [11:20] James.atLLOUD @hg.osgrid.org: Events always seem to accelerate on approach. [11:21] George Equus: I mis read  lol   last year was about beginning of November too [11:22] James.atLLOUD @hg.osgrid.org: Does Physx require Nvidia hardware for the client? [11:22] Kayaker.Magic @grid.kitely.com:8002: Nope [11:22] James.atLLOUD @hg.osgrid.org: OK, whew. [11:23] Andrew.Hellershanks @hg.osgrid.org: Ubit has been busy this past week. He has made a lot more commits than in the past couple of weeks. [11:23] James.atLLOUD @hg.osgrid.org: 22! W00t [11:23] Kayaker.Magic @grid.kitely.com:8002: The biggest problem with Physx on the server is that it doesn't run under Linux, only Windows server [11:23] James.atLLOUD @hg.osgrid.org: OH yes, big problem. [11:24] Kayaker.Magic @grid.kitely.com:8002: Apparenly it could run under Linux, but hasn't been tested much there yet. [11:24] Jak Daniels: it really was designed to be run on the GPU not on x86 cpus ;) [11:25] James.atLLOUD @hg.osgrid.org: sure - thus the speedy llcastRay [11:25] Kayaker.Magic @grid.kitely.com:8002: I've watched it work on InWorldz without a GPU and it works well. [11:28] Andrew Hellershanks: Welcome, Ubit [11:28] Alicia Raven: i was just going to a test region when i saw the meeting was in progress [11:29] Ubit Umarov: well i was lost at rl sorry :) [11:29] Jak Daniels: I arrived an hour ago thinking the meeting would be early again due to daylight times etc ;) [11:29] Andrew Hellershanks: :) [11:29] Jak Daniels: last week it was at 18:00 UTC [11:29] James atLLOUD: oh no - do not like DST [11:29] Andrew Hellershanks: The time change is always confusing. [11:30] Ubit Umarov: ( hidding at rl from alicia ) [11:30] Andrew Hellershanks: I don't either, James. It gets dark so early now. [11:30] Alicia Raven: lol i found a nice bug to confuse ubit :P [11:30] Andrew Hellershanks: :) [11:30] Jak Daniels: it says 19:00 UTC in the wiki and the other times are in xST not xDT, so I think during the summer we hold it at the wrong time [11:30] Kayaker.Magic @grid.kitely.com:8002: As a protest of DST I always refer to it as Daylight Wastings Time [11:30] James atLLOUD: It wreaks havoc with global internet events and once again, the US dominates for the rest of the world. [11:31] James atLLOUD: lol [11:31] Ubit Umarov: wel should use UTC ( sorry us ppl ) [11:31] Andrew Hellershanks: I'm not sure which part of the year is the "savings" time. [11:31] Jak Daniels: everyone should use UTC or GMT. That's time ground zero :) [11:31] James atLLOUD: Oh come on, let's use metric time! [11:32] Ubit Umarov: utc is basicly gmt [11:32] Andrew Hellershanks: yes [11:32] Alicia Raven: star dates would be best [11:32] Jak Daniels: lol [11:32] Andrew Hellershanks laughs [11:32] Ubit Umarov: just greenwitch no longer runs the standard... NIST and others do it [11:32] Jak Daniels: I bet star dates would be based on UTC [11:33] George Equus: I tried proposing UTC for years now, in the viewer too. No interest... [11:33] Ubit Umarov: UTC is used in all science astronomy included [11:33] George Equus: ppl seem to like confusion [11:33] James atLLOUD: It's 804 millidays right now. [11:33] Andrew Hellershanks: Alicia, IIRC, you mentioned you are running OpenSim under Linux. Which version of mono are you using? How do you find the memory usage compares to running OS under Windows? [11:33] James atLLOUD: EVERYWHERE :) [11:33] Alicia Raven: another VR system i used had its own time zone VRT which was in the middle of atlantic (halfway between US and UK) [11:34] Ubit Umarov: UTC is just a simple time convertion for everyone [11:34] Alicia Raven: using 3.2.8 mono and i havnt compared memory for long time, i would be interested to see tho [11:34] Jak Daniels: I've tried three versions of mono.... 3.10 3.12 and 4.6. In all case the memory uses just spirals upwards until mono eventually crashes :( [11:34] Ubit Umarov: i only need to add one hour lol [11:34] Andrew Hellershanks: Ubit, simple for those that know the difference between their local time and UTC [11:35] Alicia Raven: 3.2.8 has no memory issues like that, atleased not for me [11:35] Ubit Umarov: actually also forgot if my local time is as utc or one hour dif [11:35] Alicia Raven: i just have a strange UDP issue that no one else seems to [11:35] George Equus: yes Andrew, much simpler for everyone because UTC never ever change [11:35] George Equus: each do the maths on one fixed time [11:36] Andrew Hellershanks: Alicia, I've found that the memory use with mono 3.8.0 is double what it is under Windows and it will increase over time. [11:36] Jak Daniels: I've been graphing my instances: http://host3-3.ateb.co.uk/MIOS/?set=Clunbar01:21 [11:36] Andrew Hellershanks: George, UTC never changes but if you live in a country where the clocks change you have to know how much to add/subtract from local to get UTC. [11:36] Alicia Raven: have u tried 3.2.8 or that too low? [11:36] Jak Daniels: even with the regular GC it still creeps up [11:37] Alicia Raven: its not the warp3d leak is it? [11:37] Jak Daniels: I used to run on 3.2.8. I might try going back to that [11:37] Jak Daniels: Not using warp3d [11:37] Jak Daniels: static map tiles [11:37] Andrew Hellershanks: Jak, yes but with mono it creeps up fairly quickly. In less than a week all the RAM in the machine will be in use. [11:37] George Equus: Yes Andrew.... but you only need to know your own time zone diff from UTC  not truing to mess with us central time changes as well [11:37] Andrew Hellershanks: Alicia, no. I only have warp3d set to create tiles once on startup. [11:38] George Equus: This comunity is after all global, not a US thing anymore [11:38] Alicia Raven: since i upgraded to master code i havnt left sims running as long as that so i will keep my eye on it from now on [11:38] Andrew Hellershanks: George, right. With the time change my offset is now 5 hours when its normally 4. [11:38] James atLLOUD: no bites on millidays. ok. [11:38] James atLLOUD: too soon. [11:38] Andrew Hellershanks: Alicia, ok. Let me know what you notice. [11:38] Alicia Raven: anyone here noticed problems with slow UDP [11:38] Alicia Raven: ? [11:38] George Equus: them Swizz made a nice try :) [11:39] James atLLOUD: Yes - Swatch beats.  lol [11:39] Ubit Umarov: nice my local time now is UTC :) [11:39] George Equus: hehe, think you can buy those watches still [11:40] Andrew Hellershanks: Alicia, I may have to try 3.2.8. I had a copy of it but I was having issues trying to use it. I don't remember the exact nature of the problems off hand. [11:40] Jak Daniels: Hmmm, there are some spikes on the graphs occasionally on my instances: http://host3-3.ateb.co.uk/MIOS/?set=Clunbar01:0 [11:40] Jak Daniels: for UDP average process time [11:40] James atLLOUD: not thrilled to have metric time be a brand, but it was an effert. [11:40] Jak Daniels: meh, sometimes the graphs don't load properly.... that's another issue :) [11:41] Ubit Umarov: what is that udp time ? [11:41] Jak Daniels: Client Stack: Average UDP Process Time [11:41] Ubit Umarov: collected from where ? [11:42] Jak Daniels: from the json stats interface from each of the instances of opensim that I'm running on that server [11:42] Jak Daniels: I graph the stats with RRDTool [11:42] Ubit Umarov: yeack i need to remove those cpu eating things :p [11:43] Ubit Umarov: what os version ? [11:43] Jak Daniels: 0.9.0 dev just before the last release.... maybe 2 weeks old [11:44] Ubit Umarov: ok ty [11:44] Ubit Umarov: now one does need to know what udp process time means :P [11:45] Andrew Hellershanks: Ubit, Do tell us oh wise one. :) [11:45] Jak Daniels: Yes.... It's taken me a while to figure out what some of the stats are and how they change over time... e.g. some are like counters, other are gauges... some are scaled ...it's a bit of a mess [11:46] Ubit Umarov: i did ignore those stats so far.. only changed the viewer ones [11:46] Ubit Umarov: bc i do have a strong bias to just delete them ;) [11:46] Ubit Umarov: well not just bc [11:46] Jak Daniels: noooo.... don't do that. Just make them work properly.... like SNMP from a router ;p [11:47] Ubit Umarov: but they are expensive and code is pretty confusing [11:47] Ubit Umarov: we do have several stats running... all diferent [11:47] Jak Daniels: to be honest there is too many.... I think Justin was a great fan of stats, I remember he used to make lots of graphs... [11:48] Jak Daniels: But we should keep some basic ones like mem usuage, cpu usage, threads etc [11:48] Ubit Umarov: they are important of course.. [11:48] Ubit Umarov: but a cleanup is need one of this days [11:48] Andrew Hellershanks: One option is to only have some of the stats when running in debug mode. [11:48] Ubit Umarov: also some are pretty hard to understand [11:48] Jak Daniels: yeah, that might work. I always run in release mode tho lol [11:49] Ubit Umarov: remember the 55fps normalized value discution [11:49] Andrew Hellershanks: Jak, that would let you have basic stats as you just mentioned but not all the ones currently available. [11:49] Ubit Umarov: OS is a multitask thing.. err 200 threads jak? outch [11:49] Jak Daniels: yes. I'd have to refactor my stats reporting framework for that ;) [11:50] Alicia Raven: some code ive seen takes longer to calculate the stats than to do the work the stats represent [11:50] Ubit Umarov: so timing stats etc are hard to relate to actually performance [11:50] Andrew Hellershanks: Ubit, yea but that was about changing how the stat was calculated. Turning on/off some stats is not as major a discussion. [11:50] Alicia Raven: not referring to os there btw [11:50] Jak Daniels: yep I see it runs around 140-200 threads most of the time in mono [11:50] Ubit Umarov: you all seen papers published with that mistake [11:50] Ubit Umarov: how many regions per instance? [11:51] Jak Daniels: one [11:51] Ubit Umarov: ohh [11:51] Jak Daniels: on my test server sometimes I run two per instance [11:51] Jak Daniels: but on this server each Instance=1 region [11:51] Ubit Umarov: ive about 114 on my empty test set with 3 regions on a instance [11:51] Andrew Hellershanks: I have had 9 in one instance in the past. That was back in the 0.6.9 days. [11:52] Ubit Umarov: ( 144 as windoes count not opensim ) [11:52] Jak Daniels: ok ubit.... here my empty regions on my test server: [11:52] Jak Daniels: http://host3-5.ateb.co.uk/MIOS/?set=JDTest1:27 [11:52] Jak Daniels: about 100 threads [11:52] Ubit Umarov: ok thats close to what i see [11:53] Ubit Umarov: i did want to reduce threads [11:53] Ubit Umarov: ended up adding more :p [11:53] Jak Daniels: yes... I always thought we use too many. It gets to a point where the cpu is mostly context switching [11:54] Jak Daniels: and each thread is doing very little [11:54] Andrew Hellershanks: jak: Nice use of RRDTool [11:54] Ubit Umarov: ms says a application should not use more than 6 [11:54] Ubit Umarov: and it is a nonse not only opensim [11:54] Ubit Umarov: seen a vidoe codec using 500 [11:55] Ubit Umarov: with a cpu with only 4 real cores ( plus hyperth bad ones ) [11:55] Jak Daniels: thanks Andrew.... I added stats reporting to my MIOS project on github. MIOS= Multiple Instances (of) Open Simulator [11:56] James atLLOUD: nice [11:56] Jak Daniels: it records the json stats from each instance every 60 seconds [11:56] Ubit Umarov: what hell are slow frames ? [11:56] Jak Daniels: You tell me lol [11:57] Andrew Hellershanks: Jak, How do you access all those stats? I'm thinking by multiple means. [11:57] Jak Daniels: I might not be graphing that stat right as I didn't know what it meant or what the units were [11:57] Jak Daniels: I don't understand the question Andrew [11:57] Ubit Umarov: 1000 thread pools ?? [11:57] Ubit Umarov: what hell [11:58] Simulator Version v0.5 ruft: OpenSim 0.9.1.0 Dev       6956ada: 2016-11-06 04:29:01 +0000 (Unix/Mono) [11:58] Andrew Hellershanks: Jak, jsonsimstats doesn't provide all those statistics on the page. Neither does the WebStats, IIRC. [11:58] Andrew Hellershanks: jak, How are you gathering all that data? [11:59] Jak Daniels: No.... json stats returns all the stats as json when you call http://simhost:9000/ManagedStats [11:59] Jak Daniels: Everything... [11:59] Andrew Hellershanks: ManagedStats? I haven't seen that mentioned anywhere before now. [11:59] Jak Daniels: So MIOS knows how split those stats up and send them as datasources to multiple RRD databases [11:59] Ubit Umarov: justin things [12:00] Jak Daniels: I also create RRAs (round robin archives) that store the data over a longer period at reduced resoultion [12:00] Ubit Umarov: like jak said justin loved "instrumentation" [12:00] Andrew Hellershanks: Is that documented anywhere? [12:00] Jak Daniels: so it can store up to 3 years [12:00] Ubit Umarov: at justin blog ? [12:00] Jak Daniels: Yes there is a section for it in opensim.ini .... just a mo.... [12:01] Jak Daniels: brb [12:01] Andrew Hellershanks: oh, I just found it buried in the OpenSimDefaults.ini file. [12:01] Ubit Umarov: i did see them in code [12:02] Jak Daniels: its in [startup] [12:02] Jak Daniels: ManagedStatsRemoteFetchURI=ManagedStats [12:02] Andrew Hellershanks: yea, Its right after the entry for jsonSimStats [12:02] Andrew Hellershanks: I don't have that enabled in the standalone I use for testing. [12:02] Ubit Umarov: well as i said we do need to review all those [12:03] Ubit Umarov: simplify etc [12:03] Jak Daniels: try this Andrew... this is what it gives: [12:03] Jak Daniels: http://host3-5.ateb.co.uk:9000/ManagedStats/ [12:03] James atLLOUD: What creates a thread? How do you reduce that? [12:03] Jak Daniels: Right I gotta go..... got band practise now its 8pm here. See ya all :) [12:04] Andrew Hellershanks: That is a lot of data. [12:04] Ubit Umarov: hmm some of those mb affect by the stats normalization also [12:04] James atLLOUD: Bye Jak - great stuff. [12:04] Andrew Hellershanks: Not easy to read all that json data. [12:04] Ubit Umarov: as i said i only payed attention well mainly.. to the ones sent to viewers [12:04] Andrew Hellershanks: Hm... there is a bit of chat lag here today. [12:05] Andrew Hellershanks: Seeing how many stats are in ManagedStats I would think that is something that could be reduced when OS is in release mode. [12:06] Ubit Umarov: fear some are intrusive on execution timing/load [12:06] Andrew Hellershanks: Ubit, All the more reason to disable some of the stats in release mode. [12:06] Ubit Umarov: prob of all this instrumentation was that it was done as a deep debug tool [12:07] Andrew Hellershanks nods [12:07] Ubit Umarov: now devs can do it another way... [12:07] Ubit Umarov: yeap chat lag :( [12:07] Andrew Hellershanks: Anything else for todays meeting before we wrap it up? [12:08] Andrew Hellershanks: Ubit, I think it is something that can be discussed after 0.9 is released. [12:08] Ubit Umarov: for example on that raycasts issue i just places timing in my test code to see what was been slow [12:08] Ubit Umarov: and removed when done [12:09] Andrew Hellershanks nods [12:09] Ubit Umarov: ubOde may have some of that commented out still [12:09] Andrew Hellershanks: Some stats code doesn't get removed after the original reason for adding it no longer exists. [12:11] Andrew Hellershanks: If there isn't anything else for today we can call this one done and dusted. [12:11] Andrew Hellershanks: Good to hear that there has been some improvements to llCastRay. Kayaker will be happy about that. He has been complaining about it for some time. :) [12:12] Ubit Umarov: hmm Kayaker you did report the raycast issue right ? [12:12] Kayaker.Magic @grid.kitely.com:8002: yeah, that was me [12:12] Ubit Umarov: is it better ? [12:13] Kayaker.Magic @grid.kitely.com:8002: Difficult for me to test, I can't build it myself. Been using the latest OSGrid releases [12:13] Ubit Umarov: this afternoon i did start writing a replacement code for that v3 terrain collider.. but it will take a while [12:14] Kayaker.Magic @grid.kitely.com:8002: Cool, I will look forward to it.