[11:03] Andrew Hellershanks is online.
[11:03] Second Life: Region found!
[11:03] Second Life: Region found!
[11:03] Andrew Hellershanks: There is some glue missing that prevents event markers from showing on the maps.
[11:03] Second Life: Teleport completed from hop://login.osgrid.org/jump8000/128/124/28
[11:03] Second Life: The region you have entered is running a different simulator version.
Current simulator: OpenSim 0.7.6 Dev 6da50d3: 2013-07-16 07:19:13 -0700 (Unix/Mono)
Previous simulator: OpenSim 0.7.6 Dev f3b3e21: 2013-07-14 07:28:40 -0700 (Unix/Mono)
[11:04] Justin Clark-Casey: hi folks
[11:04] Nebadon Izumi: hello Justin
[11:04] Vivian Klees: hi Justin
[11:04] Shaun.Emerald @sanctuary.homelinux.org:8012: Hi Justin cloud.
[11:04] Richardus Raymaker: hi justin
[11:04] Justin Clark-Casey: Just cleared my cache
[11:04] Andrew Hellershanks: I don't know enough about OS to feel inclined to tackle figuring out how to apply the glue (if that is what is needed).
[11:04] Nebadon Izumi: your actually a cloud to me too Shaun
[11:05] Shaun.Emerald @sanctuary.homelinux.org:8012: I just rebaked. Maybe that will help.
[11:05] Richardus Raymaker: Oh. is tehre some document that explains the xml codes remote admin use ? and groups etc. ?
[11:05] Vivian Klees: JCC rezzed
[11:05] Nebadon Izumi: there you guys are :)
[11:05] Shaun.Emerald @sanctuary.homelinux.org:8012: I'm seeing two unknowns typing in local.
[11:05] Nebadon Izumi: I just updated Wright Plaza right before everyone got here
[11:05] Justin Clark-Casey: GUN7?
[11:05] Andrew Hellershanks: what's strange about event markers on map is that I see the query for event data going out. I see it calling the routine that would make the event markers appear. Its just the data that needs to be obtained by the code that sends out the event marker locations.
[11:05] Shaun.Emerald @sanctuary.homelinux.org:8012: and AU3
[11:06] Nebadon Izumi: am I coming up GUN7?
[11:06] Justin Clark-Casey: AU3 is less surprising
[11:06] Justin Clark-Casey: GUN7 is the one that would interest me most
[11:06] Richardus Raymaker: talking about the guns. how are things going justin ?
[11:06] Shaun.Emerald @sanctuary.homelinux.org:8012: there is one of each
[11:06] Nebadon Izumi: you probably just need to clear cache
[11:06] Justin Clark-Casey: Was hoping that dvia's changes had fixed the issue but clearly not
[11:06] Nebadon Izumi: ya its not quite perfect yet
[11:06] Shaun.Emerald @sanctuary.homelinux.org:8012: GUN7 just spoke about guns and asked how justin was
[11:06] Nebadon Izumi: i dont think Last Owner has been fixed yet Justin
[11:06] Justin Clark-Casey: Was looking forensically at the code just now. Clearly something is amiss but it's not obvious what
[11:07] Richardus Raymaker: gun7 ? thats maby me Richardus
[11:07] Nebadon Izumi: most of the GUN7 i see are from Last Owner on prims
[11:07] Justin Clark-Casey: well, I wa slooking at some anomolies in last tues load test
[11:07] Nebadon Izumi: its not tied into the HG stuff
[11:07] Nebadon Izumi: so they comeback unknown
[11:07] Justin Clark-Casey: where a griduser entry should have been present but for some reason was not picked up
[11:07] Shaun.Emerald @sanctuary.homelinux.org:8012: yes Richardus
[11:07] Nebadon Izumi: speaking of Load test we are not going to have one today at the conclusion of the meeting
[11:07] Shaun.Emerald @sanctuary.homelinux.org:8012: AU3 is Neb I think
[11:08] Nebadon Izumi: diva and I have really been pounding on some things, and were about to change up some stuff
[11:08] Nebadon Izumi: i was hoping maybe to have it in before meeting today, but she is not back yet
[11:08] Richardus Raymaker: i logged in on snoopy's sim and teleported to here. no HG at all
[11:08] Justin Clark-Casey: what else is going to change?
[11:08] Andrew Hellershanks: You still have 52 minutes
[11:08] Nebadon Izumi: so I am thinking either Thursday or Saturday
[11:08] Nebadon Izumi: it seems the Blocking Queue Justin
[11:08] Justin Clark-Casey: richardus: remote admin presentation is at http://opensimulator.org/wiki/RemoteAdmin
[11:08] Nebadon Izumi: she wants to do a totally new type of Blocking queue
[11:08] Justin Clark-Casey: richardus: but nothing on groups
[11:08] Nebadon Izumi: diva and dahlia and myself were doing some testing last night
[11:09] Nebadon Izumi: dahlia noticed that the high cpu during tests was system cpu
[11:09] Nebadon Izumi: not mono
[11:09] Nebadon Izumi: and her and diva figured out its the blocking queue
[11:09] Nebadon Izumi: holding up threads I guess
[11:09] Justin Clark-Casey: through actual testing?
[11:09] Nebadon Izumi: yesz
[11:09] Richardus Raymaker: Yes justin. but ineed to decode how the cml looks.. not abig problem as soon i get the remote admoin data to the php code. fails with my C# program. need to test that different
[11:09] Nebadon Izumi: a shitload of testing
[11:09] Justin Clark-Casey: well, the blocking queue should pause the thread, it should not be in a busy loop or anything like that
[11:09] Shaun.Emerald @sanctuary.homelinux.org:8012: enough testing to frazzle Neb's nerves
[11:09] Nebadon Izumi: thats pretty much all i did yesterday
[11:09] Justin Clark-Casey: ok
[11:09] Richardus Raymaker: whats thew blocking queu doing ?
[11:10] Justin Clark-Casey: maybe a weakness in mono?
[11:10] Nebadon Izumi: I dont know yet we havent done it yet
[11:10] Nebadon Izumi: but diva has some ideas we are going to test this afternoon
[11:10] Justin Clark-Casey: the blocking queue is now being more heavily used to handle incoming udp requests instead of putting each on its own threadpool thread
[11:10] Nebadon Izumi: I cant fully explain the extent of the problem
[11:10] Nebadon Izumi: we would need diva or dahlia
[11:10] Nebadon Izumi: im sure dahlia will show up shortly
[11:10] Justin Clark-Casey: fair enough - I'm juts a little surprised that's the cause
[11:10] Justin Clark-Casey: but with mono it's not completely out of the question
[11:10] Nebadon Izumi: ya
[11:10] Nebadon Izumi: diva seems to think though
[11:11] Nebadon Izumi: that if we can do this right, it will be a huge performance boost for lots of things
[11:11] Nebadon Izumi: just quoting what she said
[11:11] Justin Clark-Casey: ok
[11:11] Justin Clark-Casey: hope springs eternal, eh?
[11:11] Nebadon Izumi: hehe
[11:11] Shaun.Emerald @sanctuary.homelinux.org:8012: Sometimes she counts her chickens before they've hatched, but we can hope. :)
[11:12] Richardus Raymaker: good she dont use sion chickens
[11:12] Nebadon Izumi: ya it does seem to be the sticky spot though
[11:12] Nebadon Izumi: and dahlia seemed to concur
[11:12] Justin Clark-Casey: fair enough. glad to hear a lot of testing went on
[11:12] Nebadon Izumi: yes lots more to come too
[11:12] Nebadon Izumi: we are seeing some improvements with a patch dahlia put in last night
[11:13] Nebadon Izumi: but diva seems to think we can do even better
[11:13] Richardus Raymaker: sounds like its good to hold the 0.7.6 release a bit back.. :O
[11:13] Nebadon Izumi: were running the latest code here
[11:13] Nebadon Izumi: i updated just before everyone came in
[11:13] Nebadon Izumi: cpu is at 46%
[11:13] Nebadon Izumi: for this region
[11:13] Nebadon Izumi: thats pretty good
[11:13] Justin Clark-Casey: how do you know if something is high cpu btw?
[11:13] Justin Clark-Casey: do we have any kind of rule of thumb?
[11:13] Nebadon Izumi: were using top
[11:14] Richardus Raymaker: sounds high for 13 avatars ?
[11:14] Nebadon Izumi: and dahlia had some other tools we used
[11:14] logger sewell: hey everyone
[11:14] Justin Clark-Casey: hey logger
[11:14] Nebadon Izumi: a shell script to log some stuff
[11:14] Nebadon Izumi: also we have been using htop
[11:14] Nebadon Izumi: which is pretty neat
[11:14] Shaun.Emerald @sanctuary.homelinux.org:8012: that script used vmstat, Neb
[11:14] Nebadon Izumi: ya dahlias shell script was logging stuff with vmstat
[11:14] Justin Clark-Casey: you might expect, say, 5% cpu per avatar once everyghing has settled down?
[11:14] Richardus Raymaker: iftop can be usefu;ll maby to, but thats for network stuff
[11:15] Nebadon Izumi: that sounds pretty reasonable
[11:15] Justin Clark-Casey: of course, not necessarily a linear load
[11:15] Nebadon Izumi: though id like to see it even lower than that
[11:15] Nebadon Izumi: I think we can
[11:15] Justin Clark-Casey: yeah, because that would still be 1100% cpu for 220 avs :)
[11:15] Nebadon Izumi: there was a point in the last few days though
[11:15] Justin Clark-Casey: without aking into account child agent comms, etc.
[11:15] Nebadon Izumi: if i was logging in 19 avatars to Keynote 1
[11:15] Nebadon Izumi: cpu was like 350-450% in KN1
[11:15] Nebadon Izumi: and 300% in KN2-4
[11:15] Nebadon Izumi: and KN2-4 were empty
[11:16] Nebadon Izumi: we got most of that worked out now
[11:16] Justin Clark-Casey: that is pretty odd, as most child comms would be message from the sim about other avatars/ojbjects
[11:16] Nebadon Izumi: and have it down considerably from those numbers
[11:16] Nebadon Izumi: but the cpu still seems to high to me for regions with no avatars
[11:16] Nebadon Izumi: my last test i did with 9 avatars
[11:16] Justin Clark-Casey: so dahlia;s commit 42e2a0d really made a difference?
[11:16] Nebadon Izumi: the cpu in KN1 was about 75-80%
[11:16] Justin Clark-Casey: she did have another one but I see it was revert
[11:16] Justin Clark-Casey: ah, never mind, I can ask directly later
[11:16] Nebadon Izumi: and the KN2-4 was like 65%
[11:17] Nebadon Izumi: that seems high for no avtars
[11:17] Nebadon Izumi: to be at 65%
[11:17] Justin Clark-Casey: yes
[11:17] Nebadon Izumi: we might not be able to get the KN1 region down a whole lot more with 9
[11:17] Nebadon Izumi: but we should be able to get the child agents to be less harsh
[11:17] Andrew Hellershanks wonders what the next thing will be that will hold back the 0.7.6 release ;)
[11:17] Justin Clark-Casey: should improve if avs are sitting
[11:18] Nebadon Izumi: its hardly noticeable
[11:18] Nebadon Izumi: at this point
[11:18] Nebadon Izumi: its maybe 5% cpu overall
[11:18] Justin Clark-Casey: andrew: well, I much prefer to see this kind of thing settle downa bit first
[11:18] Nebadon Izumi: the difference between sitting and standing
[11:18] Nebadon Izumi: that is standing still
[11:18] Justin Clark-Casey: though tbh, it's the unknown users issue that still concerns me
[11:18] Nebadon Izumi: if everyone is moving things change
[11:18] Andrew Hellershanks: Definitely. If it will make a big improvement to grids, it is worth waiting.
[11:18] Justin Clark-Casey: yeah, standing still should be better since all the incoming AgentUpdate UDP is being thrown away if no av changes
[11:18] Justin Clark-Casey: instead of being processed
[11:19] Justin Clark-Casey: well, ideally I like a period of calm but that just hasn't happened recently
[11:19] Nebadon Izumi: next few days should be interesting
[11:19] Justin Clark-Casey: in theory, could cut a release from an earlier point - it's this unknown users thing that I really want to see improved
[11:19] Nebadon Izumi: ya that would be good
[11:19] Justin Clark-Casey: I don't think it's all down to the last owner thing from log analysis
[11:19] Richardus Raymaker: the unknown user is a big problem yes
[11:20] Nebadon Izumi: no its probably not just that im sure too Justin
[11:20] Justin Clark-Casey: might even be better to make no reply at all in that case
[11:20] Nebadon Izumi: but im sure its a big one people wearing attachments
[11:20] Nebadon Izumi: coming in from HG
[11:20] Justin Clark-Casey: which would leave the owner as perpectually "Loading...."
[11:20] Nebadon Izumi: but there are definitely other wierdnesses still
[11:21] Justin Clark-Casey: the thing is, the HG code now inserts a grid user entry before the av gets onto the region
[11:21] Justin Clark-Casey: so the regio nshould always pick it up - but for some reason this definitely failed for some people in the logs
[11:21] Justin Clark-Casey: from the messages last Tues
[11:21] Nebadon Izumi: yea
[11:21] Nebadon Izumi: maybe some kind of delay and timing gets wierd
[11:21] Nebadon Izumi: gets missed
[11:21] Justin Clark-Casey: yeah
[11:22] Nebadon Izumi: we have found a bunch of odd things for sure
[11:22] Justin Clark-Casey: also, I added stats measumrenet for FetchInventoryDescendents2 yesterday
[11:22] Nebadon Izumi: i still need to get OSgrid robust updated
[11:22] Nebadon Izumi: so we store session data in the db now
[11:22] Nebadon Izumi: turns out we were previously only storing it in memory
[11:22] Nebadon Izumi: and when robust was restarted all session data lost
[11:22] Nebadon Izumi: and OSgrid restarts robust daily
[11:22] Nebadon Izumi: via a cron task
[11:22] Nebadon Izumi: hence lots of problems
[11:22] Richardus Raymaker: justin, robust still have only rest console ?
[11:23] Nebadon Izumi: i will try to get that update in later
[11:23] Justin Clark-Casey: huh, some what surprisingly have only been 31 fetch inventory descendnets 2 requets on wp
[11:23] Nebadon Izumi: that does seem odd, my inventory was already cached though im sure
[11:23] Justin Clark-Casey: 32 now I opened a folder
[11:24] Justin Clark-Casey: yeah, maybe everyone has it cached already
[11:24] Justin Clark-Casey: was expecting a lot higher
[11:24] Nebadon Izumi: most of the people here are in the grid a lot
[11:24] Dahlia Trimble is online.
[11:24] Nebadon Izumi: oh here is Dahlia
[11:24] Justin Clark-Casey: yeah, would only really trigger for people coming directly into this region
[11:24] Richardus Raymaker: hi dahlia
[11:24] Justin Clark-Casey: nebadon: you're still planning on trying to offload texture requests from the conference sims?
[11:24] Nebadon Izumi: i dont think shes here yet
[11:24] Nebadon Izumi: she just came online
[11:24] Nebadon Izumi: im sure she will show up shortly
[11:25] Nebadon Izumi: already done
[11:25] Richardus Raymaker: lol. ok
[11:25] Nebadon Izumi: we have 4 texture robust consoles
[11:25] Justin Clark-Casey: I gues any change would have been masked by the other events
[11:25] Nebadon Izumi: and each of the Keynote regions has its own dedicated Robust texture server
[11:25] Justin Clark-Casey: nebadon: oh, you're not using sras?
[11:25] Nebadon Izumi: no
[11:25] Nebadon Izumi: were totally stock opensim
[11:26] Justin Clark-Casey: okay, so will be able to see number of incoming requets via stats
[11:26] Nebadon Izumi: it should be fine
[11:26] Nebadon Izumi: yes
[11:26] Justin Clark-Casey: richardus: I'm not sure if rest console is active for robust
[11:26] Nebadon Izumi: i didnt want to overcomplicate things with sras
[11:26] Nebadon Izumi: it seemed like overkill for cc grid
[11:26] Justin Clark-Casey: probably considerably more efficient that robust asset
[11:26] Nebadon Izumi: ya maybe
[11:26] Justin Clark-Casey: but never measured so that's something of an assumption
[11:26] Dahlia Trimble: hi
[11:26] Nebadon Izumi: i think it will do fine for the conference
[11:26] Nebadon Izumi: hello Dahlia
[11:27] Justin Clark-Casey: hi dahlia
[11:27] Nebadon Izumi: maybe you can explain to Justin a little bit about what we found last night
[11:27] Nebadon Izumi: i couldnt fully explain it
[11:27] Dahlia Trimble: me?
[11:27] Nebadon Izumi: ya :)
[11:27] Dahlia Trimble: about the blocking queue?
[11:27] Nebadon Izumi: the Blocking Queue stuff
[11:27] Nebadon Izumi: yes
[11:27] Dahlia Trimble: oh
[11:27] Dahlia Trimble: well there was high system load in the server
[11:28] Richardus Raymaker: Justin, i dont need rest console. i just hope soemday for emote admin interface on robust. so it can be shutdown remote with program
[11:28] Dahlia Trimble: which I thought could be related to thread management
[11:28] Dahlia Trimble: so diva thought it was in BlockingQueue
[11:28] Richardus Raymaker: hi dahlia
[11:29] Dahlia Trimble: so I made some tweaks to BLockingQueue that removed some stuff that could have caused erronous use
[11:29] Dahlia Trimble: specifically, I redued use of locks that didnt affect the internal state of the queue
[11:29] Dahlia Trimble: *reduced
[11:29] Dahlia Trimble: did that help?
[11:30] Nebadon Izumi: yes
[11:30] Nebadon Izumi: a lot
[11:30] Dahlia Trimble: cool
[11:30] Dahlia Trimble: I couldnt really test
[11:30] Nebadon Izumi: diva seems to think we can go a bit further even still
[11:30] Nebadon Izumi: but the testing this morning went considerably better
[11:30] Dahlia Trimble: oh Im sure there are more problems
[11:30] Justin Clark-Casey: afaik, msdn doc makes no thread safety guarantees about any method on System.Collections.Generic.Queue, including Count
[11:31] Dahlia Trimble: count doesnt affect the queue tho
[11:31] Dahlia Trimble: MSDN examples dont lock for count
[11:31] Justin Clark-Casey: they don't? any links?
[11:31] Dahlia Trimble: ya sec
[11:31] Justin Clark-Casey: yes, but conceivably you could catch it in some intermiediate state when someing else is updating the queue
[11:32] Justin Clark-Casey: though for count I guess there's not so much that can go wrong as it's just return a value type
[11:33] Vivian Klees: >
[11:33] Grid.Master @75.144.98.145:8002 smiles
[11:33] Nebadon Izumi: :) just waiting on dahlia to get examples :)
[11:34] Justin Clark-Casey: did just see wright plaza complain about a thread timeout though, which was curious
[11:34] Nebadon Izumi: no this just happened?
[11:34] Nebadon Izumi: i would say thats probably not all that unusual
[11:34] Justin Clark-Casey: yes
[11:35] Nebadon Izumi: the maintenance thread?
[11:35] Justin Clark-Casey: well, more the warning above that
[11:35] Richardus Raymaker: hmm maintance thread. i have seen that one last time to i think
[11:35] Justin Clark-Casey: about the thread watchdog, which should really never exceed the interval by that much unless something just caused a very heavy load
[11:35] Justin Clark-Casey: was also a transient chat lag
[11:36] Nebadon Izumi: I didnt notice the chat lag
[11:36] Nebadon Izumi: im sure things get a bit wierd when multiple people are logging in simultanously to this many avatars
[11:37] Justin Clark-Casey: I wonder if repeated requests to the surrounding unresponsive sims are having an impact
[11:37] Nebadon Izumi: it might
[11:37] Nebadon Izumi: that has happened in the past
[11:37] Nebadon Izumi: e
[11:37] Nebadon Izumi: i see most of the regions
[11:37] Nebadon Izumi: 1 looks not visible
[11:37] Nebadon Izumi: NE corner
[11:38] Justin Clark-Casey: yeah, that's probably the bad one
[11:38] Nebadon Izumi: feels nice and fluid here though
[11:38] Justin Clark-Casey: yeah, it was just a transient lag - haven't seen it again since
[11:38] Dahlia Trimble: justin firefox isnt working with my clipboard but its in the example for Monitor.Pulse method in MSDN docs
[11:38] Richardus Raymaker: the NE seems gone for me to
[11:39] Nebadon Izumi: as much lag as Child Agents seem to cause
[11:39] Shaun.Emerald @sanctuary.homelinux.org:8012: for me too
[11:39] Nebadon Izumi: it wouldnt surprise me if a missing sim makes things even worse
[11:39] Dahlia Trimble: its also often recommended to test Monitor.Wait() in a while loop due to false triggering
[11:40] Dahlia Trimble: so I added that
[11:40] Justin Clark-Casey: hmm, not example code in http://msdn.microsoft.com/en-us/library/system.threading.monitor.pulse.aspx
[11:40] Justin Clark-Casey nods
[11:40] Justin Clark-Casey: nebadon: and the funk that the mapitem thread tosses out
[11:40] Andrew Hellershanks: hehe
[11:40] Simulator Version v0.5 shouts: OpenSim 0.7.6 Dev 6da50d3: 2013-07-16 07:19:13 -0700 (Unix/Mono)
[11:41] Nebadon Izumi: blacklist stuff?
[11:41] Justin Clark-Casey: yes
[11:41] Dahlia Trimble: http://msdn.microsoft.com/en-us/library/system.threading.monitor.pulse(v=vs.71).aspx
[11:41] Nebadon Izumi: ya the map fetches a lot of data
[11:42] Andrew Hellershanks: yea, it does.
[11:42] Dahlia Trimble hand typed that url
[11:42] Andrew Hellershanks: It even fetches some data now that it doesn't know what to do with.
[11:43] Nebadon Izumi: ha
[11:43] Nebadon Izumi: have you noticed how low the CPU is here
[11:43] Nebadon Izumi: its crazy low
[11:44] Nebadon Izumi: like 25% of 1 core at times
[11:44] Richardus Raymaker: whats low ? still 46% ?
[11:44] Andrew Hellershanks: What was it like before?
[11:44] Richardus Raymaker: oh thats not bad neb
[11:44] Andrew Hellershanks: yea, good for this region
[11:44] Nebadon Izumi: it was probably like 80-100% with this kind of avatar load
[11:44] Andrew Hellershanks: Big improvement.
[11:44] Nebadon Izumi: ya really big this is good, we are on the right track I think
[11:45] Justin Clark-Casey: dahlia: interesting, thanks. I see there the Count is only called when first and second threads have finished, so no other activity anyway
[11:45] Justin Clark-Casey: but I will loko further - always interested in such examples
[11:45] Nebadon Izumi: don't you know im loko!
[11:46] Nebadon Izumi: lol
[11:46] Andrew Hellershanks laughs
[11:46] Dahlia Trimble: shouldnt really need to lock when reading a field if it doesnt modify, and count *shouldnt* modify or even iterate
[11:46] Dahlia Trimble: it does iteriate in STL lists but those arent managed
[11:46] Justin Clark-Casey: depends how its implemented internally
[11:46] Nebadon Izumi: cpu is at 17%
[11:46] Andrew Hellershanks: How is the memory usage for WP these days?
[11:46] Nebadon Izumi: amazing
[11:47] Dahlia Trimble: anyway the MSDN docs really are lacking for all that threading stuff
[11:47] Justin Clark-Casey: yeah, they're pretty shitty for this
[11:48] Dahlia Trimble: OTOH, there was a crash yesterday (before my patch) that could be from a lack of locking in a queue
<-- At this point Nebadon Izumi accidentally rezzed a very large and complicated set of objects, which unacked everyone from the sim due to starvation of cpu resources -->
[11:51] Andrew Hellershanks is online.
[11:51] Nebadon Izumi is offline.
[11:52] Nebadon Izumi is online.
[11:52] Justin Clark-Casey: well, I have to imagine that one was related to that huge rez
[11:52] Andrew Hellershanks is offline.
[11:52] Richardus Raymaker is online.
[11:52] Nebadon Izumi: wow sorry about that
[11:52] Nebadon Izumi: i didnt mean to rez that
[11:52] Andrew Hellershanks is online.
[11:52] Nebadon Izumi: totally logged me out
[11:52] Dahlia Trimble: lol
[11:52] Richardus Raymaker: bad neb.. bad.. :)
[11:52] Richardus Raymaker: hi marcus
[11:52] Justin Clark-Casey: it does demo thought that some kidn of throttling is really needed there as well
[11:52] Grid.Master @75.144.98.145:8002 wonders why everybody poofed
[11:53] Vivian Klees shakes a fist at demoscene
[11:53] Nebadon Izumi: ya my viewer lagged on me
[11:53] Richardus Raymaker: 1 mesh to far neb
[11:53] Dahlia Trimble: anyway justin....
[11:53] Nebadon Izumi: and all the sudden it rezzed
[11:53] Grid.Master @75.144.98.145:8002 blinks
[11:53] Grid.Master @75.144.98.145:8002: hmmm
[11:53] Andrew Hellershanks: I got logged out too when nebadon did whatever he did.
[11:53] Nebadon Izumi: i had my inventory open
[11:53] Grid.Master @75.144.98.145:8002: I hypered in.. wonder if that helped me NOT go poof
[11:53] Nebadon Izumi: i think when my viewer lagged i must have clicked and drug it out
[11:53] Nebadon Izumi: lol
[11:53] Dahlia Trimble: I think that crash yesterday was in DoubleQueue
[11:53] Richardus Raymaker: some emsh reezzed, but not compleet. then lagg.. and then [quit]
[11:53] Nebadon Izumi: i was scrolling my inventory
[11:53] Dahlia Trimble: but I didnt trace into it, and trying to lock DoubleQueue deadlocked
[11:53] Nebadon Izumi: sorry about that
[11:53] Vivian Klees: it was Neb's demoscene
[11:54] Justin Clark-Casey: dahlia: yeah,. so other commenters like http://stackoverflow.com/questions/8457243/is-queue-count-thread-safe would agree that Count is not guaranteedthread-safe. though a particular implementation might be
[11:54] Dahlia Trimble: so something in DoubleQueue internal locking might be amis
[11:54] Justin Clark-Casey: what we really want is something like ConcurrentQueue but that 's not available yet
[11:54] Justin Clark-Casey nods
[11:54] Dahlia Trimble: ya thats in 4.0 I think
[11:54] Justin Clark-Casey: yeah
[11:54] Richardus Raymaker: time for 4.0 :)
[11:56] Nebadon Izumi: well I am definitely interested to see what Diva has in mind
[11:56] Nebadon Izumi: hopefully we can come up with something clever that works well until mono has matured a bit more
[11:56] Nebadon Izumi: and allows us to run the fancy .net 4 / 4.5 stuff
[11:56] Justin Clark-Casey: well, one has to remember that this also has to run in .net
[11:57] Nebadon Izumi: yea
[11:57] Justin Clark-Casey: so one cannot really rely on particular mono implementation details, which may even change between mono versions
[11:57] Nebadon Izumi: right
[11:57] Dahlia Trimble: well justin I had an alternative implementation to the BlockingQueue fix that maintained a count internal property but it was a bit more complex
[11:57] Nebadon Izumi: I think diva knows that
[11:57] Nebadon Izumi: I think she uses windows mostly
[11:57] Dahlia Trimble: so I went for the simpler version
[11:57] Nebadon Izumi: it want until a few weeks ago she started testing on Linux
[11:57] Nebadon Izumi: mostly because i was complaining so much
[11:57] Nebadon Izumi: haha
[11:57] Richardus Raymaker: useing windows here. to and you need to install special .net 3.5 for opensim
[11:57] Justin Clark-Casey: dahlia: I guess it epends on whether a wrong count could cause a problem. If not then perhaps not so bad, though it still makes me nervous
[11:58] Nebadon Izumi: its a catch 22 though too
[11:58] Dahlia Trimble: well its better than it was :)
[11:58] Nebadon Izumi: it seems like having the lock in place might also cause problems
[11:58] Nebadon Izumi: now we just need to figure out which is less problems
[11:58] Nebadon Izumi: haha
[11:58] Justin Clark-Casey: well, that commit wasn't just the lock, it was also the other bug fixes dahlia did
[11:59] Justin Clark-Casey: or rather,m the other one
[11:59] Justin Clark-Casey: because some of the code outside the locks could also be inconsistent with the two m_quque.Count and m_pqeue.Count
[11:59] Dahlia Trimble: accessing the Count could trigger the Wait
[11:59] Justin Clark-Casey: in Contains it looks like it would be possible to return false when in fact the item is there
[11:59] Dahlia Trimble: when the count was locked
[12:00] Justin Clark-Casey: yes. The lock should be only taken for a short time though?
[12:01] Dahlia Trimble: Im also not happy with the Wai with the timeout, which could trigger on a timeout early but with still an empty queue
[12:01] Dahlia Trimble: so that whole class is a mess
[12:01] Justin Clark-Casey: it is possible for Contains() to return the wrong result now
[12:01] Justin Clark-Casey: yes, I don't think we should really be writing our own concurrent structures
[12:01] Dahlia Trimble: depends
[12:01] Justin Clark-Casey: there are libraries like C5.dll around
[12:01] Dahlia Trimble: but its not likely
[12:02] Justin Clark-Casey: yeah, but thats' jus tthe kidn of thing that causes mysterious unsolveable bugs
[12:02] Nebadon Izumi: I would suggest hopping on IRC in a bit so we can discuss this with diva too
[12:02] Nebadon Izumi: so we get an idea of where she is planning on heading
[12:02] Dahlia Trimble: no problem :)
[12:02] Nebadon Izumi: she might be back now
[12:02] Nebadon Izumi: i just pinged her on iRc
[12:02] Justin Clark-Casey: sure. well, I need to eat first but will be back later
[12:03] Nebadon Izumi: ok sounds good, ya I could use some food myself
[12:03] Nebadon Izumi: starving
[12:03] Dahlia Trimble: mmmmm food
[12:03] Nebadon Izumi: haha
[12:04] Nebadon Izumi: seems like divas not back yet anyway
[12:04] Fred.Appleby @hg.viewtwo.net:8654: Fred audits the cakes in the display cabinet
[12:04] Nebadon Izumi: something else i want to add
[12:04] Nebadon Izumi: Singularity Alpha viewer now has the ability to export prims as collada
[12:04] Nebadon Izumi: it will retain texture mapping and colors
[12:04] Dahlia Trimble: anyway justin Im not claiming that BlockingQueue is perfect now, Im only suggesting that its a mess and what I did last night was try to address *some* of that mess
[12:04] Nebadon Izumi: but not the textures themselves yet
[12:05] Justin Clark-Casey: dahlia: oh sure, commits definitely appreciated
[12:05] Nebadon Izumi: https://files.streamgrid.net/singularity/
[12:05] Justin Clark-Casey: I jam just paranoid about introducing race conditions since I've had to spend so many hours tracking these things down in the past
[12:05] Nebadon Izumi: you can grab the latest Singulrity alpha here
[12:06] Nebadon Izumi: ya the good thing now is we have a pretty good testing area setup
[12:06] Nebadon Izumi: so we can really pound on these changes
[12:06] Dahlia Trimble: well I think there was a big race condition *before* my commit
[12:06] Nebadon Izumi: were not just guessing
[12:06] Justin Clark-Casey: dahlia: how so?
[12:06] Nebadon Izumi: being able to mass teleport real avatars was huge too
[12:06] Dahlia Trimble: but I'll add the internal count logic
[12:07] Dahlia Trimble: how so? Wait was falsely exiting
[12:07] Dahlia Trimble: also I do believe Contains is unused
[12:07] Dahlia Trimble: I was going to remove it
[12:08] Justin Clark-Casey: ok, I need to read further. I believe lock() is simply synatactic sugar for Monitor Enter() and Monitor.Exit()
[12:08] Dahlia Trimble: probably
[12:08] Andrew Hellershanks: I seem to remember reading some like that too when I was cleaning up some code I had worked on.
[12:09] Dahlia Trimble: well revert the commit if you dont like it
[12:09] Justin Clark-Casey: it's not eh whole commit - just the parts where count is accessed outside the lock that makes me nervous
[12:09] Dahlia Trimble: yes but the lock is the problem
[12:10] Justin Clark-Casey: you'r saying because the lock exit somehow caused the other threads to exit Monitor.Wait(m_queueSync)?
[12:10] Dahlia Trimble: Im saying MSDN docs are unclear about that
[12:11] Dahlia Trimble: and if another thread really needs a transaction level Count well ya then it needs further attention
[12:11] Justin Clark-Casey: it shouldn't happen because only Pulse/PulseAll should do that
[12:12] Dahlia Trimble: shouldnt
[12:12] Dahlia Trimble: well revert it then
[12:13] Dahlia Trimble is going to eat lunch, bye all :)
[12:14] Nebadon Izumi: see you dahlia, thanks a bunch
[12:14] Nebadon Izumi whispers: see you on IRC im sure :)
[12:14] Shaun.Emerald @sanctuary.homelinux.org:8012: If what she did really helped performance, then it clearly shows *something* there isn't right and needs much closer looking at, yes (says the non-programmer sysadmin).?
[12:14] Fred.Appleby @hg.viewtwo.net:8654: If the commit is reverted, and the load isssues return, game over?
[12:14] Justin Clark-Casey: well no, that means further work
[12:14] Nebadon Izumi: I think diva plans on changing all of this
[12:14] Richardus Raymaker: No fred. then just back to the drawingboard
[12:14] Nebadon Izumi: so we really need to see what she has in store
[12:14] Dahlia Trimble is offline.
[12:15] Andrew Hellershanks: I'm going to head out.
[12:15] Nebadon Izumi: ok ya I think the meeting has wrapped at this point
[12:15] Nebadon Izumi: thanks everyone
[12:15] Shaun.Emerald @sanctuary.homelinux.org:8012: Neb, I finally installed htop. Does 31 threads per mono sound right to you?
[12:15] Nebadon Izumi: keep an eye on IRC for more fun later :)
[12:15] Andrew Hellershanks: I'll be commiting the changes to ossearch to fix some issues with events in the next day or so.
[12:15] Andrew Hellershanks: Possibly as soon as later today.
[12:15] Nebadon Izumi: i was seeing between 30-50 Shaun
[12:15] Nebadon Izumi: sounds about right
[12:15] Shaun.Emerald @sanctuary.homelinux.org:8012: okay
[12:16] Shaun.Emerald @sanctuary.homelinux.org:8012: Is Andrew ++ Plugh?
[12:16] Nebadon Izumi: htop is pretty cool
[12:16] Shaun.Emerald @sanctuary.homelinux.org:8012: ==
[12:16] Nebadon Izumi: yes
[12:16] Andrew Hellershanks: Shaun, yes
[12:16] Shaun.Emerald @sanctuary.homelinux.org:8012: Ah!
[12:16] Andrew Hellershanks: And the incantation for me is "Xyzzy" but it doesn't always work. ;)
[12:17] Fred.Appleby @hg.viewtwo.net:8654: The latest commit I have seen my reference thread count increase by 11
[12:17] Andrew Hellershanks waves
[12:17] Andrew Hellershanks is offline.
[12:18] Richardus Raymaker: heading out to.
[12:18] Richardus Raymaker: bye
[12:18] Nebadon Izumi: justin Diva is back on IRC
[12:18] Nebadon Izumi: see you Richardus
[12:18] Richardus Raymaker: suprissed justin still here..
[12:18] Justin Clark-Casey: well, I'm reading the code
[12:18] Nebadon Izumi: hehe
[12:18] Shaun.Emerald @sanctuary.homelinux.org:8012: you know with htop, if you have lsof installed it can tell you what files are open and if you have strace installed it'll let you attch to a process.
[12:19] Justin Clark-Casey: and there is now a race where a poll request could get wrongly discarded
[12:19] Justin Clark-Casey: anyway, if we were to try a revert later, would you be able to do a comparison test?
[12:19] Nebadon Izumi: depends
[12:19] Nebadon Izumi: on what diva has me doing
[12:19] Nebadon Izumi: but probably
[12:19] Justin Clark-Casey: ok
[12:20] Justin Clark-Casey: in theory, the performnace impact here should be very low, so I would be very surprised if this really made a huge difference
[12:20] Justin Clark-Casey: but one always has to be prepared to be surprised
[12:21] Nebadon Izumi: ya
[12:21] Nebadon Izumi: it sounds like this whole area is quite the mess
[12:21] Justin Clark-Casey: I'm not sure I see the mess tbh
[12:21] Nebadon Izumi: im sure its going to need some going over with fine tooth comb
[12:21] Justin Clark-Casey: but anyway
[12:21] Shaun.Emerald @sanctuary.homelinux.org:8012: Back to work for me. ttyl
[12:21] Nebadon Izumi: ok see you all on IRC
[12:21] Richardus Raymaker: bye shaun
[12:21] Nebadon Izumi: im going to find some meat to bite into
[12:21] Nebadon Izumi: haha
[12:22] Nebadon Izumi: starving
[12:22] Justin Clark-Casey: ok, bye folks
[12:22] Justin Clark-Casey wqves
[12:22] Richardus Raymaker: make sure you bite first
[12:22] Nebadon Izumi is offline.
[12:22] Marcus Llewellyn smiles and waves. :)
