[11:02] Andrew Hellershanks: All I need now is a working AO. I have 10 and they aren't quite working 100% yet.
[11:02] Justin Clark-Casey: hi folks
[11:02] Mata Hari waves to Justin
[11:02] Andrew Hellershanks: hm... I have three or four AO's here to check out.
[11:02] Andrew Hellershanks: Just what I need. more of them.
[11:03] Andrew Hellershanks: :P
[11:03] Andrew Hellershanks: Hey, Justin
[11:03] Richardus Raymaker: hi justin
[11:03] Nebadon Izumi: hello Justin
[11:04] Nebadon Izumi: did Gwenette ever get in touch with you at Moses meeting?
[11:04] Justin Clark-Casey: I don't think so - I wasn't there this week
[11:04] Nebadon Izumi: ah ok
[11:04] Nebadon Izumi: there are issues with OAR/IAR imports etc.. nothing new really
[11:05] Nebadon Izumi: but perms are getting clobbered and she messaged me on skype and I recommended she talk with you
[11:05] Justin Clark-Casey: yes, this perms thing needs to be sorted out
[11:05] Nebadon Izumi: I have same issues myself, very difficult to move stuff around with IARs
[11:05] Nebadon Izumi: some of the stuff I move and its almost impossible to fix
[11:06] Andrew Hellershanks: I've seen some minor other perms issues but not related to iar/oar.
[11:06] Nebadon Izumi: like I have a box of stuff, i move it to a different grid with IAR and it says 'No Copy / No Mod'
[11:06] Justin Clark-Casey: ok, Maybe you could ask her to create a mantis so we can track this?
[11:06] fine.Dreadlow @come.2fine.de:8002: hello
[11:06] Nebadon Izumi: and it seems like the only way i could fix it was to unbox it and move it into a new box and then do god permissions fix
[11:06] Justin Clark-Casey: yeah, it's a real pita.
[11:06] Andrew Hellershanks: I sometimes find that I can take other peoples things by accident (even when locked) when my avatar has level 200 even though I'm not in god mode.
[11:06] Cuteulala Artis is Online
[11:06] Nebadon Izumi: but if i keep it in the original box its unfixable
[11:07] Andrew Hellershanks: Didn't even have admin menu showing.
[11:07] Richardus Raymaker: aha, that perm problem. oh yes. sometimes you need to ignore as exampel soem thing and create a new script and copy the code over to get perms back
[11:07] Justin Clark-Casey: ok. Most helpful thing for me is to have good test cases in a mantis, even better with example IARs/OARs linked
[11:07] Nebadon Izumi: she seemed to have the opposite problem though, she said things that shut not be full perm, were turning full perm after transfered
[11:07] Nebadon Izumi: shut=should
[11:07] Justin Clark-Casey: then I can write regression tests to probe the issues in detail and prevent future regressions
[11:07] Nebadon Izumi: ok i'll ask her to file a mantis
[11:07] Justin Clark-Casey: have to be careful in the premissions area
[11:07] Justin Clark-Casey: thanks
[11:07] Nebadon Izumi: there might even already be some I'll take a look
[11:08] Justin Clark-Casey: I don'[t remember seeing anything but there's a lot of bug reports
[11:08] Nebadon Izumi: ok
[11:08] Nebadon Izumi: another thing I think might be useful is with oars to have the ability to set an owner
[11:08] Nebadon Izumi: so when your importing / merging an oar you can say make this person own all of this content
[11:09] Justin Clark-Casey: somebody other than estate manager?
[11:09] Nebadon Izumi: instead of relying on region owner to do it
[11:09] Nebadon Izumi: correct
[11:09] Richardus Raymaker: or at least option to correct wrong set urls in the oar.
[11:09] Nebadon Izumi: for an event like OSCC that just doesnt work
[11:09] Nebadon Izumi: we had to spend a lot of time sorting that out last year
[11:09] Justin Clark-Casey nods
[11:09] Justin Clark-Casey: ok
[11:10] Nebadon Izumi: it makes it impossible to isolate the newly merged content from stuff that already existed and is owned by the estate owner
[11:10] Nebadon Izumi: also another thing I need to open a mantis on that would be nice to have is the ability to change a region owner
[11:10] Nebadon Izumi: without having to database dive
[11:11] Nebadon Izumi: a lot of people ask me about that
[11:11] Justin Clark-Casey: isn't that equivalent to changing estate owner?
[11:11] Nebadon Izumi: ya same thing
[11:11] Nebadon Izumi: but right now we dont have the ability
[11:11] Justin Clark-Casey: yeah, we need commands in general to manipulate estates - somebody did put in a mantis about that
[11:11] Nebadon Izumi: ok cool
[11:11] Justin Clark-Casey: rather than owner being able to do this in the db
[11:11] Nebadon Izumi: ya
[11:12] Nebadon Izumi: Fleep brought it up other day
[11:12] Nebadon Izumi: I am setting up a new Avacon grid
[11:12] Nebadon Izumi: and I made my avatar owner of all the regions
[11:12] Nebadon Izumi: and she wanted to change it to a neutral account
[11:12] Nebadon Izumi: now I have to edit the database on 20 regions manually
[11:12] Justin Clark-Casey: they're not all using a commen estate db?
[11:12] fine.Dreadlow @come.2fine.de:8002: it would be nice if there where a webside to handle those things
[11:13] Nebadon Izumi: they are, but its still a pain
[11:13] Justin Clark-Casey: but doesn't that mean you would only need to change it once there?
[11:13] Richardus Raymaker: Bad experience with a central esate database. its more easyh in maintance to keep every region seperate
[11:13] Nebadon Izumi: well depends on if she wants same owner for everything
[11:13] Andrew Hellershanks: You have to do them one at a time?
[11:13] Justin Clark-Casey: fine: opensimulator deliberatly doesn't have a web interface to prevent increasing the complexity of the project
[11:13] Nebadon Izumi: she actually hasnt said that yet
[11:13] Justin Clark-Casey: and to allow multiple external web interfaces
[11:14] Nebadon Izumi: still, its risky to be database diving, and it prevents her from being able to do it
[11:14] Justin Clark-Casey: yeah, I think it's one of the few main tasks where you still have to db diving
[11:14] Nebadon Izumi: basically makes it so I have to, or in general someone with good database manipulation skills, it opens the door to disaster
[11:14] fine.Dreadlow @come.2fine.de:8002: backend modifications are always risky in any case
[11:14] Justin Clark-Casey: still, putting in all the estate commands and making sure they work will take some time
[11:14] Nebadon Izumi: ya
[11:14] Nebadon Izumi: just one of those things
[11:15] Nebadon Izumi: the only Estate thing that is a real pain is changing owners
[11:15] Nebadon Izumi: which shouldnt be that difficult
[11:15] Justin Clark-Casey: Yes - now I think about it, for most prameter changing we rely on the client interface
[11:16] Justin Clark-Casey: but the viewer doesn't have any way to manipulate estate details, afaik
[11:16] Nebadon Izumi: another "Bug" that keeps popping up that people constantly ask me about is deleted prims coming back after region restart
[11:16] Nebadon Izumi: i noticed today when i logged in here for meeting, stuff i rezzed last week at meeting had come back
[11:16] Richardus Raymaker: that bug still exist ?
[11:16] Nebadon Izumi: stuff i erased last week before logging out
[11:16] Master Dubrovna: That has become a big issue for me too.
[11:16] Justin Clark-Casey: nebadon: yeah, I know there's a mantis about that but simply haven't had time to investigate. And when I last looked at the coode,there were no obvious race conditions
[11:16] fine.Dreadlow @come.2fine.de:8002: the better way is to modify the estate owner through robust themself
[11:16] Justin Clark-Casey: but clearly there is something somewhere
[11:16] Freakys.Clone @freakytech.spdns.de:8002: regions tend to lock the inventory change
[11:16] Mata Hari: it's prim karma....they were very good in their previous life so they get reincarnated
[11:16] Freakys.Clone @freakytech.spdns.de:8002: and they seem to vanish
[11:16] Pathfinder.Lester @pathlandia.dlinkddns.com:9000: lol
[11:16] Nebadon Izumi: ya, no worries, i just wanted to bring this stuff up today
[11:16] Freakys.Clone @freakytech.spdns.de:8002: but only after restart you will be able to delete them
[11:16] Nebadon Izumi: i was not expecting any quick fixes or easy answers
[11:17] Nebadon Izumi: that bug has lingered for years
[11:17] Pathfinder.Lester @pathlandia.dlinkddns.com:9000: and if they're very good, cubes come back as toruses
[11:17] Justin Clark-Casey: nebadon: I fyou or anyone else has more details about reappearing prims, would be good to add to the mantis so it comes back to the top of the recently changed list
[11:17] Simulator Version v0.5 shouts: OpenSim 0.8.0 Dev 873eee5: 2014-03-13 00:04:20 +0000 (Unix/Mono)
[11:17] Nebadon Izumi: only reason i even really mentioned it was because when i logged in today stuff i erased was sitting here again
[11:17] Richardus Raymaker: whats the experience here with terrain loading. on my own 0.8dev its still much slower then with 0.7.6
[11:17] Justin Clark-Casey: and if it has been a while since the last entry, even a "this is still happening" is helpful
[11:17] Freakys.Clone @freakytech.spdns.de:8002: Justin try to use another account on that region rezz a bunch of things
[11:18] Nebadon Izumi: ok Justin I will see what I can find
[11:18] Freakys.Clone @freakytech.spdns.de:8002: and then try to delete them with an estate account
[11:18] Andrew Hellershanks: I've also been getting reports with 076PF that objects are re-appearing after grid restarts.
[11:18] Nebadon Izumi: I can say that Sandbox Plaza gets hit hard with that problem
[11:18] Andrew Hellershanks: s/grid/region/
[11:18] fine.Dreadlow @come.2fine.de:8002: yepp its slower a little bit and asset server as well on my side
[11:18] Nebadon Izumi: my guess is when multiple people are building simultaneously you increase the chances of it happening greatly
[11:18] Andrew Hellershanks: Sometimes the items that reappear were deleted days before.
[11:18] Nebadon Izumi: its almost like things are not queuing properly
[11:18] Justin Clark-Casey: nebadon: is this when the regions is halted very shortly after deletion?
[11:18] Freakys.Clone @freakytech.spdns.de:8002: only one at a time suffices
[11:18] Freakys.Clone @freakytech.spdns.de:8002: just needs to be a different account
[11:18] Justin Clark-Casey: hmmm
[11:19] Freakys.Clone @freakytech.spdns.de:8002: pretty reproducable with HG account
[11:19] Nebadon Izumi: Justin, in the case of my prims being back here this week I think not
[11:19] Richardus Raymaker: hi freaky
[11:19] Master Dubrovna: I see stuff that was deleted days ago reappear after restart
[11:19] Freakys.Clone @freakytech.spdns.de:8002: hi@all
[11:19] Justin Clark-Casey: yeah, there is an async mecnahism for immediate deletion but you would have to stop the region very quickly for that to be a problem
[11:19] Nebadon Izumi: there have even been cases, where i forcibly type "backup" and do a proper shutdown
[11:19] Nebadon Izumi: and my prims still came back
[11:19] Andrew Hellershanks: I have "backup" in the shutdown commands file and still things come back.
[11:19] Freakys.Clone @freakytech.spdns.de:8002: I always restart the region after dropping items on a region with an HG account
[11:19] Freakys.Clone @freakytech.spdns.de:8002: to circumvent that problem
[11:20] Justin Clark-Casey: yeah, backup will not affect deletion in any wya
[11:20] Justin Clark-Casey: sounds like it's a fairly widespread issue
[11:20] Nebadon Izumi: however I have not ever once experienced this problem on a region where I am the only builder
[11:20] UUID Speaker: Starr Haven: 47e08731-87f0-45a5-9cf9-5e933bf1a47a
[11:20] Justin Clark-Casey: I've never encountered it myself
[11:20] UUID Speaker: Starr Haven: 47e08731-87f0-45a5-9cf9-5e933bf1a47a
[11:20] Mata Hari: I have
[11:20] Nebadon Izumi: this is what leads me to beleive it only happens when you have 2 or more builders
[11:20] Freakys.Clone @freakytech.spdns.de:8002: does not happen with a single builder
[11:20] Freakys.Clone @freakytech.spdns.de:8002: for me either
[11:20] Justin Clark-Casey: hmm
[11:20] Freakys.Clone @freakytech.spdns.de:8002: only when building with another account
[11:20] Mata Hari: I've had cases where I'm deleted stuff I built only to have it come back again
[11:20] Richardus Raymaker: query pileup ?
[11:20] Justin Clark-Casey: are these builders manipulating the same prims?
[11:21] Freakys.Clone @freakytech.spdns.de:8002: and then cleaning up
[11:21] Nebadon Izumi: no
[11:21] Freakys.Clone @freakytech.spdns.de:8002: afterwards
[11:21] Master Dubrovna: It happens on a single builder region that I support
[11:21] Freakys.Clone @freakytech.spdns.de:8002: for pick up
[11:21] Nebadon Izumi: the other thing that could be problematic too is if you are building to fast
[11:21] Nebadon Izumi: say you unlink a large linkset and then delete stuff too fast
[11:21] Freakys.Clone @freakytech.spdns.de:8002: I needed only two accounts for reproduction that are allowed to build
[11:21] Master Dubrovna: Strange thing is it only seems to happen to one particular person
[11:21] Master Dubrovna: in my group
[11:22] Nebadon Izumi: if you delete stuff maybe before the linkset updates complete things get missed or something
[11:22] Freakys.Clone @freakytech.spdns.de:8002: worked with root prims to be reproduced for me too
[11:22] fine.Dreadlow @come.2fine.de:8002: hmm how do you know it? 1 prim per minute ? :D
[11:22] Mata Hari: I think it might be related to deleting multiple prims at once....it might get "stuck/lost" coalescing them to trash
[11:22] Justin Clark-Casey: freaky: If you have a case where you can reproduce the problem every time then that would be very helpful
[11:22] Justin Clark-Casey: freaky:if you can add that to the mantis... if it's just here the details will get lost
[11:22] Nebadon Izumi: ya that would be good, because I have never been able to force it to happen myself
[11:22] Freakys.Clone @freakytech.spdns.de:8002: I can reproduce it all time after some larger set of items build with an HG account
[11:22] Nebadon Izumi: not sure what the recipe is, I suspect though its when too many things happen simultaneously
[11:23] Nebadon Izumi: like unlinking a large linkset, and then quickly erasing part of it
[11:23] Freakys.Clone @freakytech.spdns.de:8002: it locks the whole admin set owner to me as well
[11:23] Richardus Raymaker: maby good to report also wich viewer
[11:23] Freakys.Clone @freakytech.spdns.de:8002: independent of viewer
[11:23] Andrew Hellershanks: yup.
[11:23] Nebadon Izumi: fast builders are likely going to have a better chance of triggering it
[11:23] Justin Clark-Casey: I can imagine there might be an issue with large linksets. But it sounds like it also happens just with small linkets or individual prims?
[11:23] Nebadon Izumi: which leads me to believe its some kind of queue issue
[11:23] Andrew Hellershanks: I have people using three different viewers at least and they see the problem.
[11:23] Richardus Raymaker: ok
[11:23] Justin Clark-Casey: there is a deletion queue
[11:23] Dev Random: all using mysql?
[11:23] Freakys.Clone @freakytech.spdns.de:8002: seems the inventory module gets locked up
[11:24] Justin Clark-Casey: when you firsst delete something, all viewers are notified that it has gone, but the actual deletion only happens a bit later
[11:24] Freakys.Clone @freakytech.spdns.de:8002: because it does not solve without restart
[11:24] Nebadon Izumi: the viewer is obviously getting the signal, but it seems like datastore isnt
[11:24] Justin Clark-Casey: but it should happen very quickly afterwards. Still, that's probably the area to investigate
[11:24] Justin Clark-Casey nods
[11:24] Justin Clark-Casey: freaky: I plan to put in some stats so we can see what is happening on invetnroy fetch
[11:24] Justin Clark-Casey: whether the module isn't deal with a bunch of requests sometimes for some reason
[11:25] Justin Clark-Casey: that will give a clue as to where the problem is
[11:25] Mata Hari: I've also noticed a new problem at a number of parties recently...after some period of time everyone in the region will lose the ability to move and any new avi that arrive show to the existing people as untextured Ruths (no attachments, etc and no textures even on clothing) but appear fine to each other. The only way to "fix" is to relog
[11:25] Nebadon Izumi: maybe just some kind of threading issue
[11:25] Justin Clark-Casey: I expect there is a race condition
[11:25] Nebadon Izumi: where if just too much stuff in general is going on, datastore isnt safe
[11:25] Nebadon Izumi: ya probably
[11:25] Justin Clark-Casey: yeah... when loading increases the change of race conditions increases because timings deviate from normal
[11:26] Nebadon Izumi: like here at Wright Plaza for instance when i erased those prims last week
[11:26] Nebadon Izumi: everyone was logging out
[11:26] Nebadon Izumi: probably a really horrible time to be building or doing stuff like deleting
[11:26] Nebadon Izumi: we know how harsh logins and logoouts can be
[11:26] fine.Dreadlow @come.2fine.de:8002: please tell me why isn't the datastore unsafe ? from your point? i can't see anything unsecure
[11:26] Nebadon Izumi: they still freeze the scene with lots of avatars present
[11:27] Nebadon Izumi: by unsafe i mean its prone to missing part of its queue
[11:27] Nebadon Izumi: other threads are making it not function properly
[11:27] Nebadon Izumi: not security unsafe
[11:27] Justin Clark-Casey: the current default mechanism of aborting scirpts is also unsafe and has the potential to make the sim unstable, particularly under loading
[11:27] fine.Dreadlow @come.2fine.de:8002: you need a better integration of it thats all look on your design you will see much things about it
[11:27] Nebadon Izumi: ah yes maybe scripted objects being delete
[11:27] Freakys.Clone @freakytech.spdns.de:8002: you remember me talking about mutex issues within opensim's design
[11:27] Justin Clark-Casey: really, the default has to become co-operative script stop... but I need to work out the migratio napproach for that
[11:27] Nebadon Izumi: the items that returned here were scripted
[11:28] Justin Clark-Casey: I would be surprised if that's a factor but anything is possible
[11:28] Nebadon Izumi: ya
[11:28] Justin Clark-Casey: it's certianly not impossible
[11:28] Justin Clark-Casey: I assume those regions are not doing co-operative script stop?
[11:28] Freakys.Clone @freakytech.spdns.de:8002: the scene module locks pretty commonly the whole scene with mutexes
[11:28] Nebadon Izumi: i remember while back when you deleted objects that had tons of scripts you would totally freeze up simulator, sometimes crashing it
[11:28] Nebadon Izumi: we did fix that eventually
[11:28] Freakys.Clone @freakytech.spdns.de:8002: that means it is like a big global lock in it
[11:29] Justin Clark-Casey: freaky: this is not the issue
[11:29] fine.Dreadlow @come.2fine.de:8002: i think some bad things are caused arround the http implementation of rubust
[11:30] Freakys.Clone @freakytech.spdns.de:8002: but things like that definitely increase he possibility
[11:30] Freakys.Clone @freakytech.spdns.de:8002: he=>the
[11:31] Justin Clark-Casey: without changing the design radically, you need mutexes. Without them, the race conditions would run untramelled and we would be in a 100X worse situation
[11:31] Justin Clark-Casey: you need exclusion between threads
[11:31] Freakys.Clone @freakytech.spdns.de:8002: you need a clean concept with message passing and very limited amount of critical sections
[11:32] Freakys.Clone @freakytech.spdns.de:8002: not those nasty global locks
[11:32] Freakys.Clone @freakytech.spdns.de:8002: like in the asset cache, scene, inventory fetcher and a lot more
[11:32] Justin Clark-Casey: that's a huge amount of work which will not happen overnight
[11:32] Freakys.Clone @freakytech.spdns.de:8002: my rough estimate with the rate of change: 2 years at least
[11:33] fine.Dreadlow @come.2fine.de:8002: i tried the folling .. the xengine loaded 2k of scripts and in the meanwhile i've deleted another sim inside the same instance .. the server wasn't reachable anymore ..
[11:33] Andrew Hellershanks: Just takes one person with the motivation and skills to make the changes in a much shorter time frame.
[11:33] Justin Clark-Casey: You would need more than one person, frankly
[11:33] Justin Clark-Casey: anyway
[11:33] Mata Hari: speaking of assets....still getting the very, very, very long initial inventory fetching times when logging into a non-plaza region but at least they generally now eventually finish
[11:33] Justin Clark-Casey: mata: yes, I'm actually starting to look at that
[11:34] Freakys.Clone @freakytech.spdns.de:8002: that is why expecting that kind of change to take probably another decade
[11:34] Justin Clark-Casey: mata: very soon I will be adding some stats numbers which I will need feedback on when the inventory fetch halts or is very slow
[11:34] Mata Hari: kk
[11:34] fine.Dreadlow @come.2fine.de:8002: the robust service crashed as well .. the inventory service had to be restarted and all was broke
[11:34] Mata Hari: pots of coffee consumed while waiting for inventory fetch today: 2
[11:34] Andrew Hellershanks: hehe
[11:34] Dahlia Trimble is Online
[11:34] Justin Clark-Casey: mata: you control the region where it is slow?
[11:34] Mata Hari: yes
[11:34] Justin Clark-Casey: ok, god
[11:35] Justin Clark-Casey: good
[11:35] Mata Hari: no console messages at all
[11:35] Justin Clark-Casey: yes, right now there is nothing
[11:35] Mata Hari: it only happens on my region here....on my standalone it never happens
[11:36] Mata Hari: I'm *guessing* it's a ROBUST thing only
[11:36] Justin Clark-Casey: there's many possible interactions
[11:36] Nebadon Izumi: if its on OSgrid it could be more complicated than that
[11:36] Nebadon Izumi: we do a lot of throtteling with nginx
[11:36] fine.Dreadlow @come.2fine.de:8002: so my 2nd question to you justin why is there now way to handle a multicore system .. xengine and some other things can't handy more then one core ..
[11:36] Justin Clark-Casey: at the moment, my guess would actually be that a single request gets lost but the inventory fetch never times out
[11:36] Freakys.Clone @freakytech.spdns.de:8002: osgrid's robust reacts slow compared to other grids
[11:36] Richardus Raymaker: i think more in the direction oif a flooded connection with to many data ? (not correct explained)
[11:37] Justin Clark-Casey: fine: what ar eyou talking about? opensim is massively threaded
[11:37] Nebadon Izumi: however my invnetory is absolutely gigantic and I dont have much trouble here
[11:37] Freakys.Clone @freakytech.spdns.de:8002: justin the massively threaded is counteracted by a bunch of critical sections
[11:37] Richardus Raymaker: hehe, your inventory is skyscraper hugh nebadon
[11:37] Arielle.Popstar @bound2.dyndns.org:9005: you login to other then plaza regions?
[11:38] fine.Dreadlow @come.2fine.de:8002: my tests said other things .. there is a smartthreadpool inside but no parts of the engine can use it .. xengine can only use one cores of 4 ..
[11:38] Nebadon Izumi: me? generally I log in either plazas or my own regions
[11:38] Justin Clark-Casey: fine: you're wrong
[11:38] Dahlia Trimble: hi
[11:38] Justin Clark-Casey: hello dhalia
[11:38] Arielle.Popstar @bound2.dyndns.org:9005: it is non plaza regions where the problem is
[11:38] Nebadon Izumi: but they are all configured pretty much the same and run on same platforms and hardware
[11:38] Mata Hari: it might have to do with how many fetches it needs to do....I can't fathom why it needs to get 1000+ items when I log in, do almost nothing, then log out again
[11:38] Arielle.Popstar @bound2.dyndns.org:9005: has been for a long time
[11:38] Richardus Raymaker: i never have seen xengine, spiking script on 1 core
[11:38] Arielle.Popstar @bound2.dyndns.org:9005: like plaza's get priority
[11:38] Nebadon Izumi: Arielle we dont give any priority to plazas
[11:38] Justin Clark-Casey: mata: I know firestorm used to fetch all inventory because of the way it handled viewer side AOs (?)
[11:38] Nebadon Izumi: its likely they are just optimized better than most
[11:38] Justin Clark-Casey: that may still be the case
[11:38] Richardus Raymaker: unless the operating system you use is bad in diveding threads
[11:39] Nebadon Izumi: we know what we are doing :)
[11:39] Justin Clark-Casey: mata: but maybe you'll tell me you aren't using fs :)
[11:39] Nebadon Izumi: we also have crazy bandwidth
[11:39] Mata Hari: I'm using FS 99.999% of the time
[11:39] Nebadon Izumi: people running from home regions over off the shelf consumer routers are going to run considerably slower
[11:39] Dahlia Trimble: plazas are in same datacenter as services atent they?
[11:39] Nebadon Izumi: there is nothing osgrid can do about that
[11:39] Arielle.Popstar @bound2.dyndns.org:9005: well isnt the default osgrid release the best optimization?
[11:39] Justin Clark-Casey: mata: yeah, I think it's something fs does - at least that certainly used tobe the case
[11:39] Nebadon Izumi: Dahlia, yes
[11:40] Nebadon Izumi: but my home regions are about 2000 miles away
[11:40] Mata Hari: it's still doing it with the new one (4.6.1)
[11:40] Nebadon Izumi: and also dont have any trouble
[11:40] Nebadon Izumi: I have a very high end router and server hardware though
[11:40] Dahlia Trimble: I dont have a problem on my regions or plazas but some regions on osgrid are horrible
[11:40] Nebadon Izumi: ya
[11:40] Nebadon Izumi: home routers!
[11:40] Nebadon Izumi: lol
[11:40] Mata Hari: I've never had it on a plaza
[11:41] Dahlia Trimble: like they are connected via 56k modem or something
[11:41] Nebadon Izumi: some routers are absolutely horrible at handling UDP
[11:41] Arielle.Popstar @bound2.dyndns.org:9005: think it is more then just that
[11:41] Arielle.Popstar @bound2.dyndns.org:9005: cause the prob is only apparent on osgrid
[11:41] Nebadon Izumi: that could just mean you have a bad connection to our data server
[11:41] Nebadon Izumi: grids like metropolis are in France on completely different networks
[11:41] Arielle.Popstar @bound2.dyndns.org:9005: but no matter, i just mention it cause i have also seen it like others
[11:42] Nebadon Izumi: the internet tends to suck like that
[11:42] Nebadon Izumi: some people have good routes, others dont
[11:42] Richardus Raymaker: No i think the are in germany ?
[11:42] Nebadon Izumi: also ISPs are heavy into throtteling their customers now
[11:42] Nebadon Izumi: you have to realize that if your ISP finds out your running servers, you are probablyn going to get throttled
[11:43] Nebadon Izumi: there is a reason they block port 80,443,21 etc..
[11:43] Freakys.Clone @freakytech.spdns.de:8002: Neb I tested American grids from my place and European grids
[11:43] Nebadon Izumi: its because technically your not allowed to run servers
[11:43] Freakys.Clone @freakytech.spdns.de:8002: and especially OSG was rather bad in reaction
[11:43] Mata Hari: well barring shifting to "all region servers must be hosted on top end pro servers" something needs to be tweaked to catch the problem and resolve it rather than sitting there locked forever
[11:43] Dahlia Trimble: you've always been throttled, thats why upload speed has always been less than download
[11:43] Nebadon Izumi: thats not true for everyone though Freakys
[11:44] Nebadon Izumi: if it was true for everyone then I could say yes, maybe there is something we can do
[11:44] Freakys.Clone @freakytech.spdns.de:8002: How comes that a grid close by OSG reacts better?
[11:44] Freakys.Clone @freakytech.spdns.de:8002: by means of datacenter location
[11:44] Nebadon Izumi: location has nothing to do with routes
[11:44] Arielle.Popstar @bound2.dyndns.org:9005: still the question is why plazas load inventory faster and completely then even hosted regions connected to osgrid
[11:45] Dahlia Trimble: plazas are on higher powered servers
[11:45] Nebadon Izumi: Arielle I can not say, but i can say for certain its nothing we are doing intentionally
[11:45] Nebadon Izumi: the PLaza servers are very high end hardware
[11:45] Nebadon Izumi: the latest Xeon CPUs
[11:45] Nebadon Izumi: 24gb ram
[11:45] Richardus Raymaker: PLaza's are closer to thge grid server ?
[11:45] Master Dubrovna: Server power and bandwidth make a difference
[11:45] Nebadon Izumi: 100mb gauranteed networks
[11:45] Richardus Raymaker: and good connection
[11:45] Arielle.Popstar @bound2.dyndns.org:9005: i realize Neb but i think it could be something that could be looked at a bit
[11:46] Arielle.Popstar @bound2.dyndns.org:9005: it doesnt make for a good user experience
[11:46] Nebadon Izumi: it could be the fact that the plaza servers run in the same data center as the asset and robust servers
[11:46] Mata Hari: needless to say it's also a huge issue when teleporting due to the relatively short window of time to complete the trasfer
[11:46] Nebadon Izumi: youn really cant compare any other grid to OSgrid
[11:46] Nebadon Izumi: even the next largest grid is small by comparison
[11:47] Nebadon Izumi: smaller less used grids are likely to respond faster no matter what
[11:47] Nebadon Izumi: there are so many factors
[11:47] Nebadon Izumi: there is no smoking gun
[11:47] Master Dubrovna: Based on size of the asset database?
[11:48] Nebadon Izumi: thats probably part of it
[11:48] Nebadon Izumi: asset server has nothing to do with inventory though
[11:48] Justin Clark-Casey: I think we do need yet more stats
[11:48] Dahlia Trimble: Arielle, how big is your inventory?
[11:48] Richardus Raymaker: most other grid use pure robust for database access, sofar i remeber osgrid use different way
[11:48] Justin Clark-Casey: hopefully, some work on higher concurrency issues will happen this year
[11:48] Arielle.Popstar @bound2.dyndns.org:9005: in this case it is not about grid vs grid but region vs region on the same grid
[11:48] Nebadon Izumi: technically OSgrids asset server should in theory be much faster than Robust
[11:49] Justin Clark-Casey: afaik nobody has done any formal testing?
[11:49] Nebadon Izumi: not really, but the way SRAS works it should be able to handle a much larger load
[11:49] Nebadon Izumi: than a single robust instance can
[11:50] Richardus Raymaker: how do you want to test it justin ?
[11:50] Andrew Hellershanks: I've often wondered why a viewer has to download the inventory list everytime you login. If it could save the inventory information it would be right there the next time you login.
[11:50] Justin Clark-Casey: yes, I think just being hosted on Apache would be a huge difference
[11:50] Richardus Raymaker: yes nebadon.
[11:50] Arielle.Popstar @bound2.dyndns.org:9005: me too Andrew
[11:50] Nebadon Izumi: no apache Justin
[11:50] Nebadon Izumi: its pure Ruby
[11:50] Arielle.Popstar @bound2.dyndns.org:9005: it is a PITA
[11:50] Dahlia Trimble: Andrew I think it does cache, or it used to
[11:50] Nebadon Izumi: Rails sucks
[11:50] Justin Clark-Casey: andrew: there is supposed to be some caching. But I think firestorm, at least, is downloading all inventory details at startup
[11:50] Mata Hari: it does but it seems to cache in a weird way
[11:50] Andrew Hellershanks: Dahlia, I can't say I've seen anything to support signs of inventory being cached.
[11:51] Nebadon Izumi: the way SRAS handles threading though is way more efficient than Robust
[11:51] Justin Clark-Casey: nebadon: really? That's unxpected
[11:51] acryline erin is Online
[11:51] Mata Hari: I think it only has hashes of bottom level folders and any change forces the entire folder to re-fetch
[11:51] Nebadon Izumi: well i guess that also depends on if your using mono or not
[11:51] Andrew Hellershanks: Justin, I often see my viewer downloading inventory on login.
[11:51] Nebadon Izumi: Robust is very dependant on mono
[11:51] Dahlia Trimble: Andrew I dont know if inventory caching works in opensim but it used to work in SL
[11:51] Justin Clark-Casey: tbh, I think the weak link is the C# HTTP webserver
[11:51] Dev Random: Andrew: also the possibiltiy you logged in with another viewer, and inventory changed
[11:51] Dahlia Trimble: maybe still does
[11:51] Nebadon Izumi: which we all know mono is not the most efficeient thing on earth
[11:51] Nebadon Izumi: also another thing with SRAS on OSgrid
[11:52] Nebadon Izumi: its cached by NGINX
[11:52] Arielle.Popstar @bound2.dyndns.org:9005: S/L loads my nventory before i even can see in world
[11:52] Nebadon Izumi: it caches like the last 10gb of assets
[11:52] Richardus Raymaker: does sras make region terrain loading faster to ?
[11:52] Arielle.Popstar @bound2.dyndns.org:9005: in opensim i can have a coffee
[11:52] Nebadon Izumi: so if your constantly hitting same assets over and over again it never even reaches SRAS
[11:52] Dahlia Trimble: terrain is via UDP
[11:52] Justin Clark-Casey nods
[11:52] Nebadon Izumi: nginx serves the asset from its cache
[11:52] Justin Clark-Casey: that's probably a non-trival performance improvement
[11:52] Andrew Hellershanks: Dev, yeah. that could be a problem. If it had a way to know the inventory count it could decide if it has change but even simple count won't help if you deleted an item and created a new one.
[11:52] Nebadon Izumi: well there is no database interaction
[11:53] Nebadon Izumi: if it hits cache
[11:53] Nebadon Izumi: so ti shoudl be considerably faster
[11:53] Justin Clark-Casey: depends - I'm not convinced a file cache is faster than db
[11:53] Justin Clark-Casey: of course, in memory will be faster
[11:53] Nebadon Izumi: ya in a fair head to head test
[11:53] Justin Clark-Casey: though that should be swamped by network times anyway
[11:53] Nebadon Izumi: on a small dataset I think there would be no difference with SRAS and Robust
[11:54] Nebadon Izumi: its not until you get into the 10's of millions of assets and TB of data that sras/nginx is going to stomp robust
[11:54] Richardus Raymaker: whats small <1TB ?
[11:54] Nebadon Izumi: less
[11:54] Nebadon Izumi: im talking gb's
[11:54] Andrew Hellershanks: Dev, perhaps if there was a date of last update to inventory. It would allow a quick check if the inventory list need to be (partially?) reloaded.
[11:54] Nebadon Izumi: under 100gb
[11:54] Nebadon Izumi: however you could do the same nginx cache with robust
[11:54] Justin Clark-Casey: andrew: there is a system of inventory versioning
[11:55] Nebadon Izumi: thats how SL works
[11:55] Nebadon Izumi: they have 100s of asset cache nodes for thier simulators
[11:55] Nebadon Izumi: nothing really hits asset server directly
[11:55] Justin Clark-Casey: in theory, the viewre should be receiving version numbers and be able to compare those with the ones it has cached and not download a lot of data
[11:55] Dahlia Trimble: opensim caches assets at the region level also
[11:55] Nebadon Izumi: other than the proxies
[11:56] fine.Dreadlow @come.2fine.de:8002: and you mean they have no clustered backend for it? :D
[11:56] Nebadon Izumi: oh SLs asset cluster is absurd
[11:56] Nebadon Izumi: its like 200 servers or something ridiculous
[11:56] Mata Hari: trouble is when you log out it "saves" the current worn asset data and then considers the folder that your clothing/attachments are in as having changed since last login so it then needs to completely re-fetch the entire folder even though its contents haven't actually changed (or that's my best guess)
[11:56] Richardus Raymaker: the opensim and region cache is running on ssd. but the regions are empty so cant say if its fater, because still not started with building
[11:57] Andrew Hellershanks: Justin, yeah. I seem to remember hearing about inventory versioning ages ago.
[11:57] Nebadon Izumi: but ya asset server has no bearing on inventory load times
[11:57] Justin Clark-Casey: yes, I fixed bugs in that area which made it inaccurate and observed that the viewer no longer made so many inventory requests
[11:57] Justin Clark-Casey: but that was some time ago now
[11:57] Justin Clark-Casey: in theory, nothing wiill have changed but that's not guaranteed
[11:58] Nebadon Izumi: inventory issues are likely bottlenecks in the simulator itself
[11:58] fine.Dreadlow @come.2fine.de:8002: Nebadon but some kind of problems could also be caused by the mono runtime .. i never tested the installation with a .net environment..
[11:58] acryline erin is Offline
[11:58] Justin Clark-Casey: nebadon: have you considered serving inventory directly from robust on osgrid?
[11:58] Nebadon Izumi: it could be that the throtteling OSgrid does makes the simulators act differently
[11:58] Mata Hari: and unwearing a mesh sends the *entire* mesh data rather than just the attachment point and any texture changes....it sends the entire mesh geometry, etc
[11:58] Nebadon Izumi: Justin, not really no
[11:59] Justin Clark-Casey: mata: it would not surprise me if there are big inefficiences with this stuff currently
[11:59] Mata Hari: yeah, that's a huge one
[11:59] Dahlia Trimble: Mata, the viewer should cache the geometry
[11:59] Mata Hari: no, it sends it Dahlia
[11:59] Dahlia Trimble: if it doesnt then maybe your viewer has a problem
[11:59] Justin Clark-Casey: nebadon: why not? potentially the cpu time required?
[11:59] Dahlia Trimble: the viewer should not re-request the geometry if its cached
[11:59] Mata Hari: if I unwear my body and wear something else it will sit there and send each piece in its entirety before replacing what I'm wearing
[12:00] Dahlia Trimble: not the mesh geometry
[12:00] Dahlia Trimble: unless your cache is failing
[12:00] Mata Hari: yes, it sends everything
[12:00] Nebadon Izumi: I am not really sure what the impat would be, and that scares me
[12:00] Dahlia Trimble: then you have a viewer problem
[12:00] Dahlia Trimble: the server only sends geometry if the viewer asks for it
[12:00] Nebadon Izumi: we have such limited resources, I am afraid to put to much load onto our backend servers
[12:00] Justin Clark-Casey nods
[12:00] Mata Hari: no, when I wear something it pulls it from the viewer inventory
[12:01] Justin Clark-Casey: nebadon: well, over the next year maybe we'll get a chance to look at that in other contexts
[12:01] Nebadon Izumi: ya
[12:01] Mata Hari: it's when I unwear something
[12:01] Dahlia Trimble: Mata, the server will NOT send mesh geometry unless the viewer asks for it
[12:01] Mata Hari: read what I am saying
[12:01] Dahlia Trimble: I did read it
[12:01] Mata Hari: I have it in cache in my viewer.... if I wear something that is in viewer cache it pulls it instantly
[12:02] Dahlia Trimble: yes
[12:02] Mata Hari: but when I then unwear it the entire mesh data is sent to the server
[12:02] Dahlia Trimble: not the mesh
[12:02] Mata Hari: even if no changes at all were made to it
[12:02] Mata Hari: okay....so what else could it be sending that is 5-10MB of data
[12:02] Justin Clark-Casey: you see this on a network monitor?
[12:02] Richardus Raymaker: i think its just sending the information to unwear and show/load the items under the mesh
[12:03] Mata Hari: I see this on network monitor and on slow server messages in the console
[12:03] dan banner: how slow?
[12:03] Justin Clark-Casey: and viewer and simulator not running on same machine?
[12:03] Dahlia Trimble: I dont know what its sending but if its sending the mesh then the viewer has serious bugs in its implementation of the protocol
[12:03] Justin Clark-Casey: yeah, it will never send up the mesh again
[12:03] Mata Hari: often around 4k-5k ms depending on the mesh size
[12:03] Justin Clark-Casey: that would only happen on an upload..... maybe it's something else
[12:04] Arielle.Popstar @bound2.dyndns.org:9005: what viewer/?
[12:04] Mata Hari: FS
[12:04] Justin Clark-Casey: it wouldn't surprise me if the server is adding new assets to the service
[12:04] Mata Hari: and happened on 4.5.x and also now on 4.6.1
[12:04] Justin Clark-Casey: but of course, that wouldn't show up on a viewer to simulator connection
[12:05] Mata Hari: it is the region sending data to OSG
[12:05] Justin Clark-Casey: oh, well that's very different
[12:05] Fearghus McMahon: hi all
[12:05] Dahlia Trimble: the region may be sending data to the inventory servers
[12:05] Mata Hari: there is a delay and then eventually OSG responds and the console shows "updated asset"
[12:05] Justin Clark-Casey: yes, I think in attachment manipulation we create far too much new asset data because of the way position informatio nis stored in thre asset itself
[12:06] Dahlia Trimble: assets dont update, thats probably a misworded message
[12:06] Mata Hari: so when that involves mesh the entire mesh deometry is being sent
[12:06] Justin Clark-Casey: yes, this is a simulator issue -I would be incredibly surprised if anything is going from viewer to simulator
[12:06] Dahlia Trimble: inventory does update but inventory is not assets
[12:06] Justin Clark-Casey: apart from bakes, I guess
[12:06] Mata Hari: yeah, you're probably right Justin....it's a communication between the region and something at OSG
[12:07] Justin Clark-Casey: a new asset does get written on detach
[12:07] Justin Clark-Casey: in various circumstances, because information is different embedded in there
[12:07] Justin Clark-Casey: though I'm suprised if that triggers a seperate mesh update
[12:08] Dahlia Trimble: it shouldnt write a new asset, it should store state data in inventory
[12:08] Justin Clark-Casey: yes
[12:08] Mata Hari: this is just for regular stuff though...if I unwear everything I'm wearing right now (when in my own region) it will have to send all the geometry of the meshes to OSG and wait for whatever response, even though I haven't changed anything
[12:09] Mata Hari: but since mesh geometry can't change after upload it really should never need to send that
[12:10] Justin Clark-Casey: ok, I need to pop off and do some stuff now
[12:10] Andrew Hellershanks: I have some documentation to finish. I'm going to head out.
[12:10] Justin Clark-Casey: I think next meeting is still an hour earlier for eu but then dst changes
[12:10] Dahlia Trimble: me too, bye all :)
[12:11] Fearghus McMahon: doh...totally forgot about that
[12:11] Andrew Hellershanks: See everyone next week.
[12:11] Fearghus McMahon: tc andrew dahlia and justin
[12:11] Mata Hari: bye all
[12:11] Andrew Hellershanks is Offline
[12:11] fine.Dreadlow @come.2fine.de:8002: bye
[12:11] Master Dubrovna: Gotta get back to work....Good day all
[12:11] Nebadon Izumi: bye everyone leaving, thanks for coming
[12:11] Richardus Raymaker: bye andrew dahlia justin
[12:11] Nebadon Izumi: see you all next week, or in 5 minutes on IRC
[12:11] Nebadon Izumi: lol
[12:12] Fearghus McMahon: c ya Neb
[12:12] Freakys.Clone @freakytech.spdns.de:8002: byeall
[12:12] Justin Clark-Casey waves
[12:12] dan banner: bye justin
