Chat log from the meeting on 2016-11-08

[11:02] Kayaker.Magic @grid.kitely.com:8002: I re-tested my problems with Ubits llCastRay and got him to look at it again.
[11:03] Jak Daniels: is it working better now on Vars?
[11:03] Kayaker.Magic @grid.kitely.com:8002: A little better, but my hope is that it will get below 500 ms per call, and we are not there yet.
[11:04] James.atLLOUD @hg.osgrid.org: How far from 500ms are you?
[11:04] Kayaker.Magic @grid.kitely.com:8002: It is down to 700ms from my tests showing 1300ms before....
[11:04] James.atLLOUD @hg.osgrid.org: whoa - still good improvement
[11:04] Jak Daniels: does the v3 raycast only work on ubOde?
[11:05] Kayaker.Magic @grid.kitely.com:8002: V3 can be set withoug ubODE
[11:05] Kayaker.Magic @grid.kitely.com:8002: Even works in 0.8
[11:05] Jak Daniels: does which physics engine is in use have any bearing on response times?
[11:06] Kayaker.Magic @grid.kitely.com:8002: Well, ubODE has it's own llCastray, which I hoped would speed it up because llCastRay and the Physics engine both need a similar model to work.
[11:07] Kayaker.Magic @grid.kitely.com:8002: I'm told some versions of llCastRay have to construct a physics model first in order to work.
[11:08] Jak Daniels: I just wondered if there might be timing differnces, e.g. with bullet running in it's own thread
[11:08] Kayaker.Magic @grid.kitely.com:8002: I havn't tested llCastRay time with both Bullet and ODE...
[11:10] Kayaker.Magic @grid.kitely.com:8002: The discouraging thing is that llCastRay on SL or InWorlds is blindingly fast by compaison to OpenSim
[11:10] Jak Daniels: does inwor;ds use physx as physics engine?
[11:11] Kayaker.Magic @grid.kitely.com:8002: Yes, InWorldz is using Physx
[11:11] Kayaker.Magic @grid.kitely.com:8002: And I'm told it has llCastRay integrated into PhysX
[11:12] Jak Daniels: And SL uses havoc on the client, maybe those are the differences. Physx I assume runs on a gpu processor for inworldz and havoc runs on the viewer in SL? That might explain the difference
[11:12] George Equus: This is ridiculous.... can't someone do something about these sits?
[11:13] Kayaker.Magic @grid.kitely.com:8002: No, Physx has a software only mode and I'm told that is what is running on the servers.
[11:13] Jak Daniels: just sit on the sofa anywhere.... you'll get one of about 4 poses ;)
[11:13] Andrew.Hellershanks @hg.osgrid.org: What's wrong with the sits?
[11:13] James.atLLOUD @hg.osgrid.org: Do you have AO George.
[11:13] George Equus: All turned off
[11:13] George Equus: I'm not the only one having trouble here
[11:14] James.atLLOUD @hg.osgrid.org: Looks like it's on in my viewer - very causal stand, eh?
[11:14] Sheera Khan: @ George: I try to catch the same seat every time... the one where my sit position fits ^^
[11:14] Kayaker.Magic @grid.kitely.com:8002: I never have problems sitting here.
[11:15] Jak Daniels: nor me.... my viewer AO is still on too
[11:15] George Equus: Ha! got a "sit" in one out of four tries tonight... sigh
[11:15] Andrew.Hellershanks @hg.osgrid.org: In the past week or two questions have been asked about the OSCC for this year. I received emails yesterday saying that registration is now open for it.
[11:15] George Equus: I just did my registration today
[11:15] Kayaker.Magic @grid.kitely.com:8002: Just saw the OSCC announcement this morning!
[11:16] George Equus: very late this year
[11:17] Kayaker.Magic @grid.kitely.com:8002: Ha: Orthopaedic Surgery Center of Clearwater
[11:18] James.atLLOUD @hg.osgrid.org: lol
[11:18] Jak Daniels: that reminds me, I must go into OSCC and change all the flags and banners. Good think they are scripted or it would be a very long and tedious job ;)
[11:18] Andrew.Hellershanks @hg.osgrid.org: The URL is https://www.eventbrite.com/e/opensimulator-community-conference-2016-registration-28824725530
[11:18] Andrew.Hellershanks @hg.osgrid.org: George, it is still about a month off.
[11:18] Andrew.Hellershanks @hg.osgrid.org: Saturday, December 10, 2016 at 7:00 AM - Sunday, December 11, 2016 at 7:00 PM (EST)
[11:18] Kayaker.Magic @grid.kitely.com:8002: approaching like a freight train IMHO
[11:20] Andrew.Hellershanks @hg.osgrid.org: :)
[11:20] James.atLLOUD @hg.osgrid.org: Events always seem to accelerate on approach.
[11:21] George Equus: I mis read lol last year was about beginning of November too
[11:22] James.atLLOUD @hg.osgrid.org: Does Physx require Nvidia hardware for the client?
[11:22] Kayaker.Magic @grid.kitely.com:8002: Nope
[11:22] James.atLLOUD @hg.osgrid.org: OK, whew.
[11:23] Andrew.Hellershanks @hg.osgrid.org: Ubit has been busy this past week. He has made a lot more commits than in the past couple of weeks.
[11:23] James.atLLOUD @hg.osgrid.org: 22! W00t
[11:23] Kayaker.Magic @grid.kitely.com:8002: The biggest problem with Physx on the server is that it doesn't run under Linux, only Windows server
[11:23] James.atLLOUD @hg.osgrid.org: OH yes, big problem.
[11:24] Kayaker.Magic @grid.kitely.com:8002: Apparenly it could run under Linux, but hasn't been tested much there yet.
[11:24] Jak Daniels: it really was designed to be run on the GPU not on x86 cpus ;)
[11:25] James.atLLOUD @hg.osgrid.org: sure - thus the speedy llcastRay
[11:25] Kayaker.Magic @grid.kitely.com:8002: I've watched it work on InWorldz without a GPU and it works well.
[11:28] Andrew Hellershanks: Welcome, Ubit
[11:28] Alicia Raven: i was just going to a test region when i saw the meeting was in progress
[11:29] Ubit Umarov: well i was lost at rl sorry :)
[11:29] Jak Daniels: I arrived an hour ago thinking the meeting would be early again due to daylight times etc ;)
[11:29] Andrew Hellershanks: :)
[11:29] Jak Daniels: last week it was at 18:00 UTC
[11:29] James atLLOUD: oh no - do not like DST
[11:29] Andrew Hellershanks: The time change is always confusing.
[11:30] Ubit Umarov: ( hidding at rl from alicia )
[11:30] Andrew Hellershanks: I don't either, James. It gets dark so early now.
[11:30] Alicia Raven: lol i found a nice bug to confuse ubit :P
[11:30] Andrew Hellershanks: :)
[11:30] Jak Daniels: it says 19:00 UTC in the wiki and the other times are in xST not xDT, so I think during the summer we hold it at the wrong time
[11:30] Kayaker.Magic @grid.kitely.com:8002: As a protest of DST I always refer to it as Daylight Wastings Time
[11:30] James atLLOUD: It wreaks havoc with global internet events and once again, the US dominates for the rest of the world.
[11:31] James atLLOUD: lol
[11:31] Ubit Umarov: wel should use UTC ( sorry us ppl )
[11:31] Andrew Hellershanks: I'm not sure which part of the year is the "savings" time.
[11:31] Jak Daniels: everyone should use UTC or GMT. That's time ground zero :)
[11:31] James atLLOUD: Oh come on, let's use metric time!
[11:32] Ubit Umarov: utc is basicly gmt
[11:32] Andrew Hellershanks: yes
[11:32] Alicia Raven: star dates would be best
[11:32] Jak Daniels: lol
[11:32] Andrew Hellershanks laughs
[11:32] Ubit Umarov: just greenwitch no longer runs the standard... NIST and others do it
[11:32] Jak Daniels: I bet star dates would be based on UTC
[11:33] George Equus: I tried proposing UTC for years now, in the viewer too. No interest...
[11:33] Ubit Umarov: UTC is used in all science astronomy included
[11:33] George Equus: ppl seem to like confusion
[11:33] James atLLOUD: It's 804 millidays right now.
[11:33] Andrew Hellershanks: Alicia, IIRC, you mentioned you are running OpenSim under Linux. Which version of mono are you using? How do you find the memory usage compares to running OS under Windows?
[11:33] James atLLOUD: EVERYWHERE :)
[11:33] Alicia Raven: another VR system i used had its own time zone VRT which was in the middle of atlantic (halfway between US and UK)
[11:34] Ubit Umarov: UTC is just a simple time convertion for everyone
[11:34] Alicia Raven: using 3.2.8 mono and i havnt compared memory for long time, i would be interested to see tho
[11:34] Jak Daniels: I've tried three versions of mono.... 3.10 3.12 and 4.6. In all case the memory uses just spirals upwards until mono eventually crashes :(
[11:34] Ubit Umarov: i only need to add one hour lol
[11:34] Andrew Hellershanks: Ubit, simple for those that know the difference between their local time and UTC
[11:35] Alicia Raven: 3.2.8 has no memory issues like that, atleased not for me
[11:35] Ubit Umarov: actually also forgot if my local time is as utc or one hour dif
[11:35] Alicia Raven: i just have a strange UDP issue that no one else seems to
[11:35] George Equus: yes Andrew, much simpler for everyone because UTC never ever change
[11:35] George Equus: each do the maths on one fixed time
[11:36] Andrew Hellershanks: Alicia, I've found that the memory use with mono 3.8.0 is double what it is under Windows and it will increase over time.
[11:36] Jak Daniels: I've been graphing my instances: http://host3-3.ateb.co.uk/MIOS/?set=Clunbar01:21
[11:36] Andrew Hellershanks: George, UTC never changes but if you live in a country where the clocks change you have to know how much to add/subtract from local to get UTC.
[11:36] Alicia Raven: have u tried 3.2.8 or that too low?
[11:36] Jak Daniels: even with the regular GC it still creeps up
[11:37] Alicia Raven: its not the warp3d leak is it?
[11:37] Jak Daniels: I used to run on 3.2.8. I might try going back to that
[11:37] Jak Daniels: Not using warp3d
[11:37] Jak Daniels: static map tiles
[11:37] Andrew Hellershanks: Jak, yes but with mono it creeps up fairly quickly. In less than a week all the RAM in the machine will be in use.
[11:37] George Equus: Yes Andrew.... but you only need to know your own time zone diff from UTC not truing to mess with us central time changes as well
[11:37] Andrew Hellershanks: Alicia, no. I only have warp3d set to create tiles once on startup.
[11:38] George Equus: This comunity is after all global, not a US thing anymore
[11:38] Alicia Raven: since i upgraded to master code i havnt left sims running as long as that so i will keep my eye on it from now on
[11:38] Andrew Hellershanks: George, right. With the time change my offset is now 5 hours when its normally 4.
[11:38] James atLLOUD: no bites on millidays. ok.
[11:38] James atLLOUD: too soon.
[11:38] Andrew Hellershanks: Alicia, ok. Let me know what you notice.
[11:38] Alicia Raven: anyone here noticed problems with slow UDP
[11:38] Alicia Raven: ?
[11:38] George Equus: them Swizz made a nice try  :)
[11:39] James atLLOUD: Yes - Swatch beats. lol
[11:39] Ubit Umarov: nice my local time now is UTC :)
[11:39] George Equus: hehe, think you can buy those watches still
[11:40] Andrew Hellershanks: Alicia, I may have to try 3.2.8. I had a copy of it but I was having issues trying to use it. I don't remember the exact nature of the problems off hand.
[11:40] Jak Daniels: Hmmm, there are some spikes on the graphs occasionally on my instances: http://host3-3.ateb.co.uk/MIOS/?set=Clunbar01:0
[11:40] Jak Daniels: for UDP average process time
[11:40] James atLLOUD: not thrilled to have metric time be a brand, but it was an effert.
[11:40] Jak Daniels: meh, sometimes the graphs don't load properly.... that's another issue :)
[11:41] Ubit Umarov: what is that udp time ?
[11:41] Jak Daniels: Client Stack: Average UDP Process Time
[11:41] Ubit Umarov: collected from where ?
[11:42] Jak Daniels: from the json stats interface from each of the instances of opensim that I'm running on that server
[11:42] Jak Daniels: I graph the stats with RRDTool
[11:42] Ubit Umarov: yeack i need to remove those cpu eating things :p
[11:43] Ubit Umarov: what os version ?
[11:43] Jak Daniels: 0.9.0 dev just before the last release.... maybe 2 weeks old
[11:44] Ubit Umarov: ok ty
[11:44] Ubit Umarov: now one does need to know what udp process time means :P
[11:45] Andrew Hellershanks: Ubit, Do tell us oh wise one. :)
[11:45] Jak Daniels: Yes.... It's taken me a while to figure out what some of the stats are and how they change over time... e.g. some are like counters, other are gauges... some are scaled ...it's a bit of a mess
[11:46] Ubit Umarov: i did ignore those stats so far.. only changed the viewer ones
[11:46] Ubit Umarov: bc i do have a strong bias to just delete them ;)
[11:46] Ubit Umarov: well not just bc
[11:46] Jak Daniels: noooo.... don't do that. Just make them work properly.... like SNMP from a router ;p
[11:47] Ubit Umarov: but they are expensive and code is pretty confusing
[11:47] Ubit Umarov: we do have several stats running... all diferent
[11:47] Jak Daniels: to be honest there is too many.... I think Justin was a great fan of stats, I remember he used to make lots of graphs...
[11:48] Jak Daniels: But we should keep some basic ones like mem usuage, cpu usage, threads etc
[11:48] Ubit Umarov: they are important of course..
[11:48] Ubit Umarov: but a cleanup is need one of this days
[11:48] Andrew Hellershanks: One option is to only have some of the stats when running in debug mode.
[11:48] Ubit Umarov: also some are pretty hard to understand
[11:48] Jak Daniels: yeah, that might work. I always run in release mode tho lol
[11:49] Ubit Umarov: remember the 55fps normalized value discution
[11:49] Andrew Hellershanks: Jak, that would let you have basic stats as you just mentioned but not all the ones currently available.
[11:49] Ubit Umarov: OS is a multitask thing.. err 200 threads jak? outch
[11:49] Jak Daniels: yes. I'd have to refactor my stats reporting framework for that ;)
[11:50] Alicia Raven: some code ive seen takes longer to calculate the stats than to do the work the stats represent
[11:50] Ubit Umarov: so timing stats etc are hard to relate to actually performance
[11:50] Andrew Hellershanks: Ubit, yea but that was about changing how the stat was calculated. Turning on/off some stats is not as major a discussion.
[11:50] Alicia Raven: not referring to os there btw
[11:50] Jak Daniels: yep I see it runs around 140-200 threads most of the time in mono
[11:50] Ubit Umarov: you all seen papers published with that mistake
[11:50] Ubit Umarov: how many regions per instance?
[11:51] Jak Daniels: one
[11:51] Ubit Umarov: ohh
[11:51] Jak Daniels: on my test server sometimes I run two per instance
[11:51] Jak Daniels: but on this server each Instance=1 region
[11:51] Ubit Umarov: ive about 114 on my empty test set with 3 regions on a instance
[11:51] Andrew Hellershanks: I have had 9 in one instance in the past. That was back in the 0.6.9 days.
[11:52] Ubit Umarov: ( 144 as windoes count not opensim )
[11:52] Jak Daniels: ok ubit.... here my empty regions on my test server:
[11:52] Jak Daniels: http://host3-5.ateb.co.uk/MIOS/?set=JDTest1:27
[11:52] Jak Daniels: about 100 threads
[11:52] Ubit Umarov: ok thats close to what i see
[11:53] Ubit Umarov: i did want to reduce threads
[11:53] Ubit Umarov: ended up adding more :p
[11:53] Jak Daniels: yes... I always thought we use too many. It gets to a point where the cpu is mostly context switching
[11:54] Jak Daniels: and each thread is doing very little
[11:54] Andrew Hellershanks: jak: Nice use of RRDTool
[11:54] Ubit Umarov: ms says a application should not use more than 6
[11:54] Ubit Umarov: and it is a nonse not only opensim
[11:54] Ubit Umarov: seen a vidoe codec using 500
[11:55] Ubit Umarov: with a cpu with only 4 real cores ( plus hyperth bad ones )
[11:55] Jak Daniels: thanks Andrew.... I added stats reporting to my MIOS project on github. MIOS= Multiple Instances (of) Open Simulator
[11:56] James atLLOUD: nice
[11:56] Jak Daniels: it records the json stats from each instance every 60 seconds
[11:56] Ubit Umarov: what hell are slow frames ?
[11:56] Jak Daniels: You tell me lol
[11:57] Andrew Hellershanks: Jak, How do you access all those stats? I'm thinking by multiple means.
[11:57] Jak Daniels: I might not be graphing that stat right as I didn't know what it meant or what the units were
[11:57] Jak Daniels: I don't understand the question Andrew
[11:57] Ubit Umarov: 1000 thread pools ??
[11:57] Ubit Umarov: what hell
[11:58] Simulator Version v0.5 ruft: OpenSim 0.9.1.0 Dev 6956ada: 2016-11-06 04:29:01 +0000 (Unix/Mono)
[11:58] Andrew Hellershanks: Jak, jsonsimstats doesn't provide all those statistics on the page. Neither does the WebStats, IIRC.
[11:58] Andrew Hellershanks: jak, How are you gathering all that data?
[11:59] Jak Daniels: No.... json stats returns all the stats as json when you call http://simhost:9000/ManagedStats
[11:59] Jak Daniels: Everything...
[11:59] Andrew Hellershanks: ManagedStats? I haven't seen that mentioned anywhere before now.
[11:59] Jak Daniels: So MIOS knows how split those stats up and send them as datasources to multiple RRD databases
[11:59] Ubit Umarov: justin things
[12:00] Jak Daniels: I also create RRAs (round robin archives) that store the data over a longer period at reduced resoultion
[12:00] Ubit Umarov: like jak said justin loved "instrumentation"
[12:00] Andrew Hellershanks: Is that documented anywhere?
[12:00] Jak Daniels: so it can store up to 3 years
[12:00] Ubit Umarov: at justin blog ?
[12:00] Jak Daniels: Yes there is a section for it in opensim.ini .... just a mo....
[12:01] Jak Daniels: brb
[12:01] Andrew Hellershanks: oh, I just found it buried in the OpenSimDefaults.ini file.
[12:01] Ubit Umarov: i did see them in code
[12:02] Jak Daniels: its in [startup]
[12:02] Jak Daniels: ManagedStatsRemoteFetchURI=ManagedStats
[12:02] Andrew Hellershanks: yea, Its right after the entry for jsonSimStats
[12:02] Andrew Hellershanks: I don't have that enabled in the standalone I use for testing.
[12:02] Ubit Umarov: well as i said we do need to review all those
[12:03] Ubit Umarov: simplify etc
[12:03] Jak Daniels: try this Andrew... this is what it gives:
[12:03] Jak Daniels: http://host3-5.ateb.co.uk:9000/ManagedStats/
[12:03] James atLLOUD: What creates a thread? How do you reduce that?
[12:03] Jak Daniels: Right I gotta go..... got band practise now its 8pm here. See ya all :)
[12:04] Andrew Hellershanks: That is a lot of data.
[12:04] Ubit Umarov: hmm some of those mb affect by the stats normalization also
[12:04] James atLLOUD: Bye Jak - great stuff.
[12:04] Andrew Hellershanks: Not easy to read all that json data.
[12:04] Ubit Umarov: as i said i only payed attention well mainly.. to the ones sent to viewers
[12:04] Andrew Hellershanks: Hm... there is a bit of chat lag here today.
[12:05] Andrew Hellershanks: Seeing how many stats are in ManagedStats I would think that is something that could be reduced when OS is in release mode.
[12:06] Ubit Umarov: fear some are intrusive on execution timing/load
[12:06] Andrew Hellershanks: Ubit, All the more reason to disable some of the stats in release mode.
[12:06] Ubit Umarov: prob of all this instrumentation was that it was done as a deep debug tool
[12:07] Andrew Hellershanks nods
[12:07] Ubit Umarov: now devs can do it another way...
[12:07] Ubit Umarov: yeap chat lag :(
[12:07] Andrew Hellershanks: Anything else for todays meeting before we wrap it up?
[12:08] Andrew Hellershanks: Ubit, I think it is something that can be discussed after 0.9 is released.
[12:08] Ubit Umarov: for example on that raycasts issue i just places timing in my test code to see what was been slow
[12:08] Ubit Umarov: and removed when done
[12:09] Andrew Hellershanks nods
[12:09] Ubit Umarov: ubOde may have some of that commented out still
[12:09] Andrew Hellershanks: Some stats code doesn't get removed after the original reason for adding it no longer exists.
[12:11] Andrew Hellershanks: If there isn't anything else for today we can call this one done and dusted.
[12:11] Andrew Hellershanks: Good to hear that there has been some improvements to llCastRay. Kayaker will be happy about that. He has been complaining about it for some time. :)
[12:12] Ubit Umarov: hmm Kayaker you did report the raycast issue right ?
[12:12] Kayaker.Magic @grid.kitely.com:8002: yeah, that was me
[12:12] Ubit Umarov: is it better ?
[12:13] Kayaker.Magic @grid.kitely.com:8002: Difficult for me to test, I can't build it myself. Been using the latest OSGrid releases
[12:13] Ubit Umarov: this afternoon i did start writing a replacement code for that v3 terrain collider.. but it will take a while
[12:14] Kayaker.Magic @grid.kitely.com:8002: Cool, I will look forward to it.