[11:00] Selby.Evans @grid.kitely.com:8002: Hi everyone
[11:00] Arielle Popstar: Hi Selby
[11:01] Arielle Popstar: Jamie. Jagga, Vincent
[11:02] Selby.Evans @grid.kitely.com:8002: Hi Arielle
[11:03] Vincent.Sylvester @hg.zetaworlds.com:8002: No amount of throwing random household objects is gonna get Andrew in here unless Ubit knows a secret tunnel to him, cause he ain't on IRC
[11:04] Ubit Umarov: hi
[11:04] Arielle Popstar: hi
[11:05] Vincent.Sylvester @hg.zetaworlds.com:8002: Not that there is much this week anyways, no commits far as I can see
[11:05] Ubit Umarov: what you mean? there all the news you bring
[11:06] Jamie.Jordan @grid.kitely.com:8002: hi everybody
[11:06] Vincent.Sylvester @hg.zetaworlds.com:8002: I did ponder over groups v2 a bit in regards to adding offline IM to that, but I am not sure the effort would be worth it
[11:06] Arielle Popstar: whynot worth it?
[11:07] Vincent.Sylvester @hg.zetaworlds.com:8002: Not to mention sending over 800 emails on each message when groups get large
[11:07] Ubit Umarov: wel its ims do need improvement
[11:07] Ubit Umarov: groups scale very badly
[11:08] Ubit Umarov: also did at irc, ofc
[11:08] Ubit Umarov: back in the day i had to limit number of users per irc channel to 1000
[11:08] Ubit Umarov: or everything whouls just colapse
[11:08] Ubit Umarov: ( 2000 )
[11:09] Vincent.Sylvester @hg.zetaworlds.com:8002: I haven't looked deep enough to know why groups is so sluggish, but I have a feeling some of it is down to poorly optimized SQL queries
[11:09] Ubit Umarov: hmm or 1998.. around that time
[11:09] Ubit Umarov: its a lot of work
[11:09] Vincent.Sylvester @hg.zetaworlds.com:8002: Can be a lot faster to join data and prepare things on the SQL side of things rather than doing hundreds of queries and joining in code
[11:09] Vincent.Sylvester @hg.zetaworlds.com:8002: But that is mostly just a guess from looking at the db connector
[11:10] Ubit Umarov: just to fidn what members are online, then where.. to send to them
[11:10] Ubit Umarov: offline saves a messages
[11:10] Ubit Umarov: like grousp with 5000 users, only 10 online..
[11:10] Vincent.Sylvester @hg.zetaworlds.com:8002: Which in SQL is just inner joins, but I don't think it uses those at all instead checking each user individually or something like that
[11:11] Ubit Umarov: thats 4990 less messages to worrie abotu
[11:12] Ubit Umarov: ims should be a service like irc
[11:12] Ubit Umarov: in fact irc
[11:13] Vincent.Sylvester @hg.zetaworlds.com:8002: It's pretty similar in requirements yeah
[11:13] Ubit Umarov: and centralized
[11:13] Vincent.Sylvester @hg.zetaworlds.com:8002: Well centralized and hypergrid don't mix
[11:13] Ubit Umarov: so users presence is easier to keep
[11:13] Ubit Umarov: hypergrid is a exception
[11:14] Ubit Umarov: and will never work as internal grid
[11:14] Vincent.Sylvester @hg.zetaworlds.com:8002: It has a presence cache already, set to expire every 20 seconds
[11:14] Ubit Umarov: thats nothing
[11:14] Vincent.Sylvester @hg.zetaworlds.com:8002: Yup
[11:14] Ubit Umarov: iand its 20s of extra trouble
[11:14] Ubit Umarov: but of is needed
[11:14] Ubit Umarov: ..ofc..
[11:15] Vincent.Sylvester @hg.zetaworlds.com:8002: It's still an "addon" so in a way it still uses indirect ways to get certain data making it slower
[11:16] Vincent.Sylvester @hg.zetaworlds.com:8002: Fixing that up, well pain
[11:16] Vincent.Sylvester @hg.zetaworlds.com:8002: Plus need to be careful not to make it too dependent so that it can still be split into its own Robust instance
[11:17] Vincent.Sylvester @hg.zetaworlds.com:8002: I hopefully have some time later this week to look at how it builds certain data, maybe some things can be changed with better SQL queries and less code
[11:18] Vincent.Sylvester @hg.zetaworlds.com:8002: Good example would be instead of getting first a list of members and then doing a query for each profile, just join the member list to the profile table, which should be faster, but then needs different code to pull the data apart again
[11:18] Jagga Meredith: you need to be cognizant of how much load you put on the SQL server. Joins are expensive
[11:19] Ubit Umarov: what profile query?
[11:19] Vincent.Sylvester @hg.zetaworlds.com:8002: If proper indexes exist joins should not cause much trouble, it's only when there are no indexes or other type conversions things get slow
[11:19] Jagga Meredith: yup
[11:20] Vincent.Sylvester @hg.zetaworlds.com:8002: That's been a problem with search actually, rewriting the ossearch to use more SQL than php code for combining data I found some joins to be really slow due to difference in charset and collation
[11:21] Jagga Meredith: exactly
[11:21] Vincent.Sylvester @hg.zetaworlds.com:8002: Not without cause there are courses and certifications for SQL, some wizardry involved
[11:21] Jagga Meredith: been there, done that
[11:22] Jagga Meredith: got the migraine
[11:22] Vincent.Sylvester @hg.zetaworlds.com:8002: And it's another area that using large tables for its data isn't exactly ideal design either. There are a few things that benefit from SQL, but also things that might work better in a document style DB, but that's a whole nother can of worms lol
[11:22] Vincent.Sylvester @hg.zetaworlds.com:8002: Small steps, first getting the C# code brought into this century might help
[11:23] Ubit Umarov: Andrew's cat arrived
[11:24] Andrew Hellershanks: Hello, everyone.
[11:24] Jagga Meredith: You're late. Current topic SQL optimization
[11:24] Ubit Umarov: hi Andrew, welcome to this week opensim meeting
[11:24] Andrew Hellershanks: I was working on some software that needs to be ready for a trade show in about a weeks time. The data format got changed on me at the last minute and now I'm still trying to get the code to realiably read the data. :P
[11:25] Vincent.Sylvester @hg.zetaworlds.com:8002: I'll add the earlier conversation to the log no biggie
[11:25] Ubit Umarov: "realiably" is optimisitic :)
[11:27] Vincent.Sylvester @hg.zetaworlds.com:8002: In other news no more TryParse crashes, some issues with avsitter I cannot reproduce, so something might be awry still, but just as lost to the cause as before. At least doesn't crash, which is positive
[11:32] Andrew Hellershanks: You finished the discussion about SQL optimization?
[11:33] Jagga Meredith: I'm done
[11:33] Vincent.Sylvester @hg.zetaworlds.com:8002: Most of that is down to doing research on the current queries and how code then combines the data etc. Figuring out the "is" to plan the "should"
[11:35] Andrew Hellershanks: There are online query analyzer tools that help you optimize queries.
[11:36] Vincent.Sylvester @hg.zetaworlds.com:8002: From the few glances I had at the db connector there are not many complex ones in there so I think most of problems is code that fetches tons of smaller pieces of data and then combines them rather than letting SQL do the heavy lifting
[11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: So it's down to figuring out what data it needs, how it currently fetches it and if that can be combined to a single query then to be broken up into the format it needs going forward
[11:37] Andrew Hellershanks: That seems likely.
[11:38] Vincent.Sylvester @hg.zetaworlds.com:8002: It's "get this than these things for each" vs. "get that inner joined with this" or some form of that
[11:39] Vincent.Sylvester @hg.zetaworlds.com:8002: The code itself also hasn't been touched in a while and probably has some slower methods in there or could benefit from better integration with other code and modules
[11:40] Vincent.Sylvester @hg.zetaworlds.com:8002: I only started looking a bit at that over the weekend, no deep diving yet, submarine is still out for delivery heh
[11:41] Vincent.Sylvester @hg.zetaworlds.com:8002: Hasn't been much happening otherwise for me at least
[11:42] Andrew Hellershanks: That level of deep diving is going to take time before it yields any useful information. Thank you for taking the time to do the diving.
[11:44] Vincent.Sylvester @hg.zetaworlds.com:8002: Yeah it will. Groups is one of those things that seems slow despite having no real reason for it, given the amount of data other parts juggle just fine, so something is awry in there either conceptually or just old code. It "shouldn't" be this slow even considering the data structure
[11:45] Andrew Hellershanks: Groups is one area of the code that could be doing multiple queries to get to the wanted data due to the database structure.
[11:47] Vincent.Sylvester @hg.zetaworlds.com:8002: It's programmers intuition perhaps telling me something is not quite right in there somewhere, finding it is another matter and I certainly don't want to rewrite the whole thing if I can help it
[11:48] Andrew Hellershanks nods
[11:48] Andrew Hellershanks: I've always thought the groups data structures seemed a little odd.
[11:49] Ubit Umarov: adn we do have 2 groups services in use
[11:49] Vincent.Sylvester @hg.zetaworlds.com:8002: They are not that outlandish given the data it needs, but I think it does nothing to handle it gracefully and just brute forces what it needs
[11:49] Ubit Umarov: some grids use the older flotsam one
[11:49] Ubit Umarov: like osgrid
[11:49] Andrew Hellershanks: Right
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: I thought they moved away from that?
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: Or was that just flotsam profiles?
[11:52] Ubit Umarov: core profiles are ok
[11:52] Ubit Umarov: they moved to them
[11:53] Andrew Hellershanks: None core profiles module doesn't use flotsam.
[11:54] Andrew Hellershanks: I just looked at the phpxmlrpc based groups code and it does have some queries that use JOIN.
[11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: It's remarkable how much faster those can be compared to individual queries and joining data in code. I noticed that with ossearch getting the additional data from classifieds and such to show profile data was ten times faster in SQL compared to two queries and php doing the work
[11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: Sure it puts more load on the database, but there are query caches and indexes that help with that a little bit
[11:57] Vincent.Sylvester @hg.zetaworlds.com:8002: Also a bit easier to throw hardware at SQL than OpenSim given the way things operate
[11:59] Andrew Hellershanks nods
[11:59] Vincent.Sylvester @hg.zetaworlds.com:8002: Oh right, for anyone that has taken a copy of the Hypergrid Wall I made a while back, I been working on a new version that greatly reduces the script load from 600 down to a single script. So a new copy should be available on the ZetaWorlds welcome region soon(ish).
[11:59] Andrew Hellershanks: We are at the top of the hour. Is there anyone who has a question they wanted to ask before they need to leave?
[12:00] Jagga Meredith: I'm done RL calls
[12:01] Andrew Hellershanks: ok, Jagga. Thanks for dropping by.
[12:01] Jamie.Jordan @grid.kitely.com:8002: thanks yall good meetinh
[12:02] Andrew Hellershanks: yw, Jamie.
[12:02] Jamie.Jordan @grid.kitely.com:8002: meeting :-)
[12:02] Andrew Hellershanks: :)
[12:04] Ubit Umarov: ok, rl also calls me. I was a bit quiet bc a bit indisposed rl
[12:04] Andrew Hellershanks: Ubit, np. Hope you feel better soon.
[12:04] Ubit Umarov: see u next week
[12:05] Ubit Umarov: thnx
[12:05] Andrew Hellershanks: Bye, Ubit.
[12:05] Ubit Umarov: seems osgridn groups is having issues
[12:06] Ubit Umarov: strange how that still did impact local chat and other region http
[12:06] Ubit Umarov: strange, need to take a look
[12:06] Andrew Hellershanks: Oh? Do they say what sort of issues? They have been having a number of problems.
[12:06] Vincent.Sylvester @hg.zetaworlds.com:8002: There is a mantis about it basically stopping a region dead if it takes long to fetch group data
[12:06] Ubit Umarov: failed commes on console
[12:06] Vincent.Sylvester @hg.zetaworlds.com:8002: Forgot the id
[12:06] Andrew Hellershanks: Hm... that seems a bit strange.
[12:07] Ubit Umarov: mb just a net glitch
[12:07] Ubit Umarov: also to presence
[12:07] Ubit Umarov: possible just tempory glitch
[12:07] Ubit Umarov: the strange part is how that did block local chat
[12:07] Andrew Hellershanks: That's possible.
[12:08] Ubit Umarov: shoud be o dif threads
[12:08] Ubit Umarov: but well all using the .net http code.. that may do t
[12:09] Vincent.Sylvester @hg.zetaworlds.com:8002: groups refresh isn't async it's a block
[12:09] Ubit Umarov: they had limitation lik eonly allow 2 connections etc
[12:09] Andrew Hellershanks: If the code related to those functions has not changed the next question is to ask what has changed if it isn't some temporary network issue. Did they recently update .NET, for example?
[12:09] Ubit Umarov: wel dunno.. filed on TODO :)
[12:10] Ubit Umarov: we are on mono
[12:10] Ubit Umarov: cya ppl
[12:10] Andrew Hellershanks: ok. But what version is osgrid using and did they do a recent update? Perhaps there is another bad version of mono.
[12:11] Andrew Hellershanks: With it being the top of the hour and with Ubit just left I'll close the meeting for today.
[12:11] Andrew Hellershanks: Thank you all for coming. See you again next week.
[12:19] Vincent.Sylvester @hg.zetaworlds.com:8002: 5795 is the mantis on that group not being async issue
