Chat log from the meeting on 2015-01-20

From OpenSimulator

Jump to: navigation, search
[10:59]  Arielle.Popstar heyas
[11:00]  Justin Clark-Casey: hi folks
[11:00]  BlueWall.Slade Hi Justin
[11:00]  Shez.Oyen Mornin' Justin
[11:00]  Seth.Nygard Hello Justin
[11:00]  James Stallings: howdy Justin ;)
[11:00]  Justin Clark-Casey: hey James, long time no see
[11:00]  Arielle.Popstar Hi Justin
[11:00]  Justin Clark-Casey: Not sure if neb will be around today
[11:00]  James Stallings: heehee, yeah, I been layin low :)
[11:00]  Aine.Caoimhe so glad to finally get a really good data capture on the latency's astounding just how badly a single latent viewer can torpedo a region and how it can snowball into a complete melt-down
[11:01]  Justin Clark-Casey: aine: yeah, it might be some effect from the number of resends required
[11:01]  Aine.Caoimhe yes, and it seems like the resends aren't throttled
[11:01]  Justin Clark-Casey: I still need to insert some debugging stuff to isolate the area - as usual all my other priorities jostling for attention
[11:01]  Justin Clark-Casey: but hope to that soon
[11:02]  Aine.Caoimhe so with that one user he managed to rack up 60k resends in less than 10 minutes
[11:02]  Aine.Caoimhe and that was backlogging the queue and putting everyone else into pseudo latency
[11:02]  Justin Clark-Casey: don't know for sure if that's the real issue yet - these things can be deceptive
[11:02]  Seth.Nygard I am very happy to do a build with whatever test changes you want to do so we can help track down what is happening and ideally find a workable fix.
[11:02]  Aine.Caoimhe nods
[11:02]  Justin Clark-Casey: but the pile up of resends is a candidate
[11:03]  Justin Clark-Casey: or possibly the process of queueing a resend in the first place
[11:03]  Richardus.Raymaker hi
[11:03]  Aine.Caoimhe we also have a really good "lab rat" with a reliably horrid connection :p
[11:03]  Richardus.Raymaker dont pile up the piles
[11:03]  BlueWall.Slade I remember the map working in a what that could pack up the queue if something went wrong - was a while back
[11:03]  Aine.Caoimhe and he's willing to be used
[11:04]  Arielle.Popstar how bad is bad?
[11:04]  Aine.Caoimhe he's on a sat connection
[11:04]  Andrew.Hellershanks Not good to arrive in time to see a question "how bad is bad" :)
[11:04]  Aine.Caoimhe and then towards the end of each month he's over his limit so they throttle him even more
[11:04]  BlueWall.Slade It's like, when something goes wrong we just keep blindly putting things in the queue
[11:04]  Aine.Caoimhe so it's baaaaaaaaaaad
[11:04]  Arielle.Popstar :)
[11:04]  Aine.Caoimhe it's compelely expected that his inworld experience would be horrible
[11:05]  Aine.Caoimhe but it's a surprise that his connection can single-handedly bring a region to its knees
[11:06]  Aine.Caoimhe I also know of a few other users that I encounter frequently while HGing around who I suspect have significant latency in their connections for whatever reason since any time one of them enters a region it grinds to a halt
[11:06]  Justin Clark-Casey: I'm surprised we didn't experience this effect during the conference and prior testing
[11:06]  Richardus.Raymaker just get a switch with variable traffic shaping and you can simulate it i think also nice
[11:06]  Arielle.Popstar not just from a high arc?
[11:07]  Aine.Caoimhe I'm a bit surprised too, although maybe it was just blind luck
[11:07]  Justin Clark-Casey: hundreds of people though, from all kinds of connections
[11:07]  Justin Clark-Casey: but perhaps you're right
[11:08]  Aine.Caoimhe there aren't all that many with really horrible connections, and the OSCC is waaaaaaaaaaaaay above the norm for hardware
[11:08]  James Stallings: it could be soething specific to his sat connection
[11:08]  Justin Clark-Casey: yeah, could be a lot of things in play
[11:09]  Seth.Nygard His sat connection is perhaps a worst case, but is certainly not the only connection that can cause these issues
[11:09]  Aine.Caoimhe another user is Shawn from Craft....I always experience extreme sim issues when he enters a region (though not as bad as Danger can produce and usually the sim is able to catch up)
[11:09]  Seth.Nygard I see it with users that are tethered with cellular connections as well as some WiFi connected users
[11:09]  Aine.Caoimhe would be a case where if you get enough somewhat-latent viewers in a region it can snowball
[11:10]  Justin Clark-Casey: I'm also surprised it didn't show up with pCampbot, since that produces massive numbers of resends
[11:10]  Justin Clark-Casey: maybe one factor is the oscc regions being fairly low prim
[11:10]  Richardus.Raymaker hi robert
[11:10]  James Stallings: o/
[11:10]  Justin Clark-Casey: and with people sitting for the most part
[11:10]  Richardus.Raymaker hmm sat connections. :\
[11:11]  Robert.Adams hello all
[11:11]  Aine.Caoimhe perhaps something unique about the different between pCampbot and a "real" viewer?
[11:11]  Justin Clark-Casey: yeah, that's possible too
[11:11]  Justin Clark-Casey: basically, it's complicated :)
[11:11]  Seth.Nygard I had tried to simulate things but was unable to come up with a set of conditions to trigger the issue. However when he, or others with bad connections come into a region the effect is very noticable.
[11:11]  Aine.Caoimhe maybe his viewer is expecting responses in "x" time and when it doesn't receive them it starts to send out even more?
[11:11]  Justin Clark-Casey: yeah, which is why live testing will now be valuable. I'll try and put the debug stuff in soon
[11:12]  BlueWall.Slade could they have a modified viewer that is doing spmething deliberately?
[11:12]  Seth.Nygard I'll be ready whenever you have that for me
[11:12]  Aine.Caoimhe not that particular user
[11:12]  Arielle.Popstar shaWNS backpack is too heavy
[11:12]  Aine.Caoimhe he's actively happy to try to help us find and solve the issue
[11:12]  Marcus.Llewellyn also vaguely wonders about badly scripted attachments.
[11:12]  Seth.Nygard No, I have had him do a complete re-install and tried a few other things
[11:12]  Aine.Caoimhe not in his case, Marcus....he can do it while Ruthed
[11:13]  Marcus.Llewellyn nods amiably.
[11:13]  Aine.Caoimhe though that doesn't mean that scripted attachments aren't a contributing factor in whether a sim can recover
[11:13]  Arielle.Popstar any idea what their ping sim is?
[11:14]  Dahlia.Trimble is Online
[11:14]  Aine.Caoimhe his ping on a 50-ping test was averaging 1000ms with 10% drop iirc
[11:14]  Seth.Nygard His ping times can vary from 500mS to as much at 1600mS
[11:14]  Aine.Caoimhe or aws it 1500ms
[11:14]  Richardus.Raymaker 1000ms is very high
[11:14]  Aine.Caoimhe yeah, his connection is really, really horrid
[11:14]  BlueWall.Slade Colony on Mars?
[11:14]  Justin Clark-Casey: that's wha tI would expect from a lossy and overloaded sat connection
[11:14]  Aine.Caoimhe but it makes him a good lab rat
[11:14]  Richardus.Raymaker Nah, mas must be 2500ms
[11:15]  Arielle.Popstar explains the probs i used to have on my old interleaved dsl connection
[11:15]  Dahlia.Trimble is Online
[11:15]  Richardus.Raymaker hi sarah
[11:15]  Aine.Caoimhe he knows his inworlld performance is going to be attrocious and will live with it because he doesn't ahve any alternative
[11:15]  Aine.Caoimhe but what he doesn't want to do is kill the experience for eveyrone else
[11:16]  Richardus.Raymaker justin, maby you need to remove the dust from your dailup to test it :)
[11:16]  Aine.Caoimhe but he would also prefer not to have to become a hermit
[11:16]  Seth.Nygard Regardless of his own experience, he doesn't expect that to be great. But even he had no idea he had a significant negative impact on the whole region until we started digging into see what was happening.
[11:16]  Dahlia.Trimble hi
[11:16]  Marcus.Llewellyn I'm curious what might happen if he used a client like radegast.
[11:16]  Richardus.Raymaker hi
[11:16]  Sarah Kline: hi Richardous
[11:16]  Shez.Oyen Hi Dahlia :)
[11:16]  Justin Clark-Casey: hi dahlia
[11:17]  Aine.Caoimhe nods
[11:17]  James Stallings: so, Justin, what have you been up to lately? ;)
[11:17]  Aine.Caoimhe or that new remote thing that FS was promoting
[11:17]  Dahlia.Trimble remote thing?
[11:18]  Shez.Oyen SLGo?
[11:18]  Justin Clark-Casey: james: doing my taxes, trying to fix bugs, etc.
[11:18]  Robert.Adams the OnLive-like thing
[11:18]  Aine.Caoimhe
[11:18]  Marcus.Llewellyn The one that uses the same service as Sl Go.
[11:18]  Robert.Adams client is in the cloud and just video goes to the user's computer
[11:18]  Marcus.Llewellyn Taxes are a bug. ;)
[11:18]  Dahlia.Trimble people willing to pay for that?
[11:19]  Aine.Caoimhe I guess so
[11:19]  Marcus.Llewellyn Seems so.
[11:19]  Shez.Oyen it is quite limited
[11:19]  Dahlia.Trimble interesting
[11:19]  Robert.Adams you either pay for that or pay for a graphics card and connectivitiy
[11:19]  Justin Clark-Casey: I haven't heard of anybody using it much with opensim yet
[11:19]  Aine.Caoimhe nor have I
[11:19]  Richardus.Raymaker yes, people paying for that
[11:19]  Dahlia.Trimble you need connectivity either way I'd think
[11:19]  Aine.Caoimhe I can see a niche for it though
[11:20]  Richardus.Raymaker because a pc you cannot drag to other homes
[11:20]  Marcus.Llewellyn I've not tried that service. I was in the Playstation Now beta, where they streamed games. I was very suprised at the quality. Latency was barely detectable.
[11:20]  Richardus.Raymaker you cannot upgrade the graphios on ipad. i know users that run it on tablet when the are from home
[11:20]  Robert.Adams does anyone build libraries on macs? I need to build BulletSim on Windows, Linux but don't have anywhere to build on Macs... what are people using?
[11:20]  Allen.Kerensky HTML5/webgl viewer is a better future path IMO
[11:21]  Richardus.Raymaker No
[11:21]  BlueWall.Slade I wish all the viewer had devent webgl in them for MOAP, etc.
[11:21]  BlueWall.Slade decent.
[11:21]  Marcus.Llewellyn The last time I built OpenSim for mac I used a Hackinstosh in a VM. But that was ages ago.
[11:21]  BlueWall.Slade I have a beta singularity that rocks
[11:21]  Dahlia.Trimble I had my webgl viewer working in moap on recent linux firestorm but I couldnt get the controls to work
[11:22]  BlueWall.Slade But, Windows not so much I think.
[11:22]  Robert.Adams current problems include that I have to build BulletSim for a 32bit mac on the 64bit systems since Mono is only available in 32 bit :-(
[11:22]  BlueWall.Slade maybe they use a different webkit lib on windows?
[11:22]  Jak Daniels: I run OSX Snow Leopard VMware for building/testing things Robert
[11:22]  Dahlia.Trimble Bluewall, seems likely
[11:22]  Jak Daniels: *on VMware
[11:22]  Justin Clark-Casey: robert: how complex is the build process?
[11:22]  BlueWall.Slade Robert, have yuou looked at the mono-project page lately?
[11:23]  BlueWall.Slade They're stepping up a little more with releases.
[11:23]  Robert.Adams a week or two ago... I was hoping things had changed...
[11:23]  Robert.Adams last I tried, it is not easy building 32bit on the 64bit systems
[11:23]  BlueWall.Slade I'm running 3.12 on a debian box atm
[11:23]  Justin Clark-Casey: yeah, cross-compiling is a pita
[11:24]  Robert.Adams ;me wishes AWS would just make osx vm's available
[11:24]  Dahlia.Trimble I have 3.99 working on debian but it took some coaxing to get it to build
[11:24]  Marcus.Llewellyn I was just about to ask if anyone tried 3.99 :)
[11:24]  Marcus.Llewellyn Which took coaxing? Compiling mono, or building opensim under 3.99?
[11:25]  Dahlia.Trimble I had memory problems with anything 3.x but 3.99 seems to have fixed them
[11:25]  Marcus.Llewellyn The proboems you had with mono not freeing memory?
[11:25]  Robert.Adams someone making a Mono-For-OpenSim available for Macs would make some people happy
[11:25]  James Stallings: Justin, I was wondering if we miht talk a bit about xassets
[11:25]  Dahlia.Trimble Marcus, it would leak, dont know why
[11:26]  Justin Clark-Casey: robert: If we can't get a mac build for bulletsim we may just have to drop support....
[11:26]  Justin Clark-Casey: james: sure
[11:26]  James Stallings: this past weekend we got into a bit of light testing of a restart of osgrid on our replicating mysql system
[11:26]  James Stallings: mel warned us off throwing the switch, saying we were pretty much oomed at around 80 to 100 gb
[11:27]  James Stallings: experience pretty well supports that
[11:27]  James Stallings: (using blobs in the tables)
[11:27]  Justin Clark-Casey: oomed? out of memory?
[11:27]  James Stallings: she also suggested she had slutions in many xassets commits she's not made
[11:27]  Marcus.Llewellyn I translated that as doomed. :)
[11:27]  James Stallings: doomed, sorry
[11:27]  Arielle.Popstar lol
[11:27]  Dahlia.Trimble I thought you were using SRAS2?
[11:27]  James Stallings: so in any case
[11:27]  Justin Clark-Casey: hello nebadon
[11:27]  Nebadon Izumi: hey
[11:27]  Nebadon Izumi: sorry I am late
[11:28]  Marcus.Llewellyn Neb is late. 40 demterits.
[11:28]  Nebadon Izumi: my internet is on the fritz
[11:28]  James Stallings: we note that you have some changes you were going to make, ad were wondering about the timing of that, etc
[11:28]  Justin Clark-Casey: james: I'm not sure what that would be. The only person who has written any major xassets code so far is me
[11:28]  Andrew.Hellershanks Dahlia, there is an SRAS 2?
[11:28]  Justin Clark-Casey: apart from some of the pgsql stuff
[11:28]  James Stallings: she lists a number of features sh has in reserve; filesystem blobs, dedup, others...
[11:29]  James Stallings: *she has
[11:29]  Marcus.Llewellyn I promise I'm not trying to be snarky, but mel has a history of promising commits that take months or more to materialize. :/
[11:29]  Justin Clark-Casey: james: I'm sure it's very helpful for you to hear that
[11:29]  James Stallings: yeah that is a concern
[11:29]  Dahlia.Trimble Andrew, thought there was, maybe my memory is going the way of mono 3.x
[11:29]  James Stallings: it would be more helpful for her to commit those changes lol
[11:30]  Justin Clark-Casey: james: there are two major things I need to do. 1) Make sure I'm recording the data I want to with xassets, particularly access times. 2) to work out some way of completing a migration from an older set of asstes tables
[11:30]  Marcus.Llewellyn The big question is whether they're true show stoppers. If not, then OSgrid is, by design, meant to upgrade to add improvements, fix things, etc.
[11:30]  Justin Clark-Casey: james: At the moment, it auto-migrates as you go along but there's no mechanism for finally taking everything out of old asset tables and putting it into xasset ones
[11:30]  James Stallings: nods
[11:30]  Andrew.Hellershanks Dahlia, I haven't looked at SRAS for a while. I'm not aware of the current version numbers on it. I'll have to take a closer look. Still trying to decide if I should go that way, or xassets, or something else to get blobs out of the db.
[11:30]  Justin Clark-Casey: james: so in principle it's usable... but I realy wanted the opportunity to do a final checkover without having to do a migration. Hence the massive experimetnal thing plastered on it
[11:31]  James Stallings: right
[11:31]  Justin Clark-Casey: and of course, it'#s never been used at scale so I don't know what problems might exist
[11:31]  Justin Clark-Casey: though I would be reasonably confident it would be fine
[11:31]  Nebadon Izumi: I think Melanie is right though, that its unwise for OSgrid to move forward with blobs in the DB
[11:31]  Justin Clark-Casey: how so?
[11:31]  Seth.Nygard Last I looked at SRAS V2 it was being officially declared as abandonware by the developer
[11:31]  Nebadon Izumi: cause in 6 months we will probably have to bring the grid down again
[11:32]  Nebadon Izumi: in the past it didnt go well with a massive DB
[11:32]  Justin Clark-Casey: in what way?
[11:32]  Dahlia.Trimble I remember it failing
[11:32]  Nebadon Izumi: it was failing constantly
[11:32]  Nebadon Izumi: around 100gb
[11:32]  Justin Clark-Casey: what kind of failure?
[11:32]  Nebadon Izumi: thats why we switched
[11:32]  Nebadon Izumi: the DB kept corrupting
[11:32]  Nebadon Izumi: we would have to rebuild the inde
[11:32]  Justin Clark-Casey: ok
[11:32]  Nebadon Izumi: probably like 10 times we had to do that
[11:32]  Nebadon Izumi: thats when adam wrote new asset server
[11:33]  Nebadon Izumi: then eventually coyled rewrote it twice
[11:33]  Dahlia.Trimble then Adam was trying to use voldemort, then SRAS came about which finally worked
[11:33]  Andrew.Hellershanks Nebadon, it kept corrupting even without the blobs in the db tables?
[11:33]  Nebadon Izumi: with xassets we can do blobs on the disk
[11:33]  Nebadon Izumi: according to melanie
[11:33]  James Stallings: I think that is functionality she hasnt comitted
[11:33]  Justin Clark-Casey: not really, xassets is all about the db
[11:33]  Nebadon Izumi: using Sras isnt really realistic
[11:33]  Nebadon Izumi: since no one is maintaining it
[11:33]  Nebadon Izumi: according to melanie
[11:34]  Nebadon Izumi: xassets can do blobs on disk
[11:34]  Justin Clark-Casey: well, tjhat's news to me
[11:34]  Justin Clark-Casey: the code I wrote is certainly not set up to do that
[11:34]  Nebadon Izumi: that might be what she needs to push in
[11:34]  Andrew.Hellershanks My blobs on disk experimental patch can be looked at to see how blobs can be stored on disk if that would be helpful for someone workingon xassets.
[11:34]  Nebadon Izumi: this is really the big holdup for OSgrid
[11:34]  James Stallings: ^^^
[11:34]  Justin Clark-Casey: I don't support having a file system xassets system in core anyway, core isn't meant for hugely scalable service stuff
[11:35]  Nebadon Izumi: we cant really bring the grid backup until we figure out what to do
[11:35]  Nebadon Izumi: we dont want to bring the grid up only to have it immediately fail again
[11:35]  Justin Clark-Casey: one major point of having it in the db was not to complicate things
[11:35]  Nebadon Izumi: and be down for another few months cause we have to convert a gigantic database
[11:35]  Dahlia.Trimble isnt meant for?
[11:35]  Marcus.Llewellyn The conversion from Robust to SRAS took about 3 days, if I recall.
[11:35]  Dahlia.Trimble I wasnt aware that was the policy
[11:35]  James Stallings: if it can be done simply enough that it works, why not have it scale
[11:35]  Nebadon Izumi: what?
[11:35]  Nebadon Izumi: try 3 months
[11:36]  Nebadon Izumi: it took forever
[11:36]  Justin Clark-Casey: because it's not simple
[11:36]  Marcus.Llewellyn We were not down for three months! Were we??
[11:36]  Nebadon Izumi: no we kind of did it in the background
[11:36]  Dahlia.Trimble nothing in opensim is simple
[11:36]  Nebadon Izumi: but I dont want to do that again
[11:36]  Nebadon Izumi: it was a complete nightmare
[11:36]  Andrew.Hellershanks 3 months to migrate data? Seems excessive.
[11:36]  Nebadon Izumi: took me literally watching the console for weeks
[11:36]  Nebadon Izumi: sleepless nights
[11:37]  Nebadon Izumi: i dont think anyone here actually knows how stressful that was
[11:37]  Justin Clark-Casey: perhaps osgrid shouldn't come back as an open grid, it's a big problem to run one of thoese anyway
[11:37]  Justin Clark-Casey: maybe a smaller set of hypergrid regions providing public facilities
[11:37]  Marcus.Llewellyn Users are usually not told about that sort of thing. You guys like to keep it close the hip.
[11:37]  Nebadon Izumi: there would be no point in bringing it back at all then honestly
[11:37]  Justin Clark-Casey: why?
[11:37]  James Stallings: that's what osgrid is - an open grid
[11:37]  James Stallings: it always has been
[11:37]  Nebadon Izumi: for what 6 regions?
[11:38]  Nebadon Izumi: not worth it
[11:38]  Justin Clark-Casey: I didn't say 6 regions. But nto something where you allow anybody to connect
[11:38]  Justin Clark-Casey: that model has always been fraught with problems
[11:38]  Nebadon Izumi: i guess id rather not do it at all myself
[11:38]  Justin Clark-Casey: like a huge asset db for instance
[11:38]  Nebadon Izumi: just run my own grid
[11:38]  James Stallings: we've always managed
[11:38]  Justin Clark-Casey: except you haven't!
[11:39]  Dahlia.Trimble well had without a backup strategy
[11:39]  Nebadon Izumi: ok well I really don't know
[11:39]  James Stallings: we've always managed to the limits of the software
[11:40]  Robert.Adams OSGrid is a place to connect ones own servers.... that's what makes it work
[11:40]  James Stallings: and thats pretty well where we are now
[11:40]  Dahlia.Trimble Robert was looking into CMS systems, maybe they are const cost competitive?
[11:40]  Justin Clark-Casey: and hence anybody can come along and access internal services and screw the entire thing
[11:40]  Nebadon Izumi: but thats never really happened but maybe once
[11:40]  Robert.Adams there are a lot of good cloud services for storing blobs but they cost money
[11:40]  Nebadon Izumi: it sounds like your just not interested Justin
[11:41]  Arielle.Popstar cant trash older assets to get it to a reasonable size?\
[11:41]  Nebadon Izumi: if thats the case we will find someone who is
[11:41]  James Stallings: maybe we just need to accept mel's work and move on
[11:41]  Justin Clark-Casey: nebadon: interesting in what way? In trying to help you guys?
[11:41]  Nebadon Izumi: you dont sound interested in OSgrid at all
[11:41]  Justin Clark-Casey: I would love to but I'm trying to do a huge number of other things simultaneously
[11:41]  Justin Clark-Casey: I think the open grid approach has massive architectural problems.
[11:41]  Nebadon Izumi: ok we will see what Mel can do, she is willing to help
[11:42]  Justin Clark-Casey: ok
[11:42]  Nebadon Izumi: she said she would push her current version of xassets in
[11:42]  Dahlia.Trimble she probably has the most experience
[11:42]  Nebadon Izumi: I think that is really what we need
[11:42]  Nebadon Izumi: avinations asset services is nearly as large as OSgrid was
[11:43]  Marcus.Llewellyn OSgrid isn't the only entity interested in xassets.
[11:43]  Justin Clark-Casey: you still shouldn't use it until it's not experimental
[11:43]  Billy.Bradshaw Hypergrid has changed Opensim dynamics. Not perfect, maybe incomplete but offers alternative data distribution models
[11:43]  Nebadon Izumi: they are all experimental
[11:43]  Justin Clark-Casey: I have every intention of doing a sweep through where things may change without migrations
[11:43]  Andrew.Hellershanks I'd like to see some built-in(?) solution to asset blob storage. I've got a rather large DB and want to do something about it before it blows up (if that might happen).
[11:43]  Dahlia.Trimble osgrid has always been experimental
[11:43]  Nebadon Izumi: the core asset service just isnt good enough
[11:44]  BlueWall.Slade MySQL advertises that they can do files as large as the OS can handle ...
[11:44]  Andrew.Hellershanks Nebadon, what is the stated mission/purpose for osgrid?
[11:44]  BlueWall.Slade So, what blows up with it?
[11:44]  Nebadon Izumi: we could use SRAS again, but no one maintains it
[11:44]  Nebadon Izumi: if something changes osgrid would be screwed again
[11:44]  Andrew.Hellershanks BlueWall, if that is the case, why do people keep saying that the DB can go bad at certain sizes, or under certain combination of conditions?
[11:45]  Dahlia.Trimble would xassets be maintained? Melanie doesn't seem to have all that much time lately
[11:45]  Nebadon Izumi: if its in core anyone really can
[11:45]  BlueWall.Slade Andrew, that is what I'm saying.
[11:45]  Arielle.Popstar but how much of that assets will never be called up? Assets from people who were around years ago but now gone
[11:45]  Andrew.Hellershanks Dahlia, not as likely to be maintained if it is a separate module outside core
[11:45]  Dahlia.Trimble anyone can if they have a need to
[11:45]  Nebadon Izumi: the blob on disk is optional thing
[11:45]  Nebadon Izumi: its not mandatory
[11:45]  Nebadon Izumi: i dont see why that should be excluded from core
[11:45]  Richardus.Raymaker but make things maby more easy
[11:46]  Andrew.Hellershanks Haing a way to dedupe would be very useful.
[11:46]  Richardus.Raymaker if blobs are on disk
[11:46]  Dahlia.Trimble lots of stuff in core is not maintained either
[11:46]  Justin Clark-Casey: like what?
[11:46]  Dahlia.Trimble it eventually gets trimmed out
[11:46]  Nebadon Izumi: ya thats another reason core asset server is not a good fit right now
[11:46]  Nebadon Izumi: no dedupe
[11:46]  Nebadon Izumi: we really want to bring the grid back up
[11:47]  Nebadon Izumi: but this is whats holding us back
[11:47]  Nebadon Izumi: techincally the asset servers are ready to go
[11:47]  Andrew.Hellershanks Assets table also gets filled with lots of "one of" things, AFAIK. Editing a script or NC can create a new asset instead of updating the existing one, IIRC
[11:47]  Dahlia.Trimble is there reason to believe SRAS is incapabie?
[11:48]  Nebadon Izumi: its capable, but again no one really knows the code
[11:48]  Nebadon Izumi: coyled isnt really interested anymore
[11:48]  Nebadon Izumi: i don't know ruby
[11:48]  Nebadon Izumi: i can barely even get it setup without his help
[11:48]  Allen.Kerensky not even ruby knows ruby - its a language without a spec
[11:48]  Andrew.Hellershanks I did a proof of concept of blobs on disk. The changes to core were minimal.
[11:49]  Andrew.Hellershanks Could that not be the basis of some changes to asset storage in core?
[11:49]  Richardus.Raymaker yes andrew, thats one of the bad things with scripts
[11:49]  Allen.Kerensky if one of OSgrid purposes is to provide an open testbed for OpenSim - using Core solutions to problems is better testing for a wider audience
[11:49]  Dahlia.Trimble using blobs on disk also assumes you format the disk correctly (number of inodes, etc)
[11:49]  Nebadon Izumi: ext4 shouldnt really have any problems
[11:50]  Allen.Kerensky when you stop using core, you stop helping test some of opensim and invalidate the results for a larger segment of the users
[11:50]  Allen.Kerensky ext4 or xfs yeah - scale away
[11:50]  Dahlia.Trimble ext4 still has internal tables that need to be configured for massive # of files
[11:50]  Dahlia.Trimble defaults will probably fail
[11:50]  Justin Clark-Casey: If I were to run a grid I would not use the core service. I would write something with a better scaling approach, like maybe something using cassandra
[11:51]  Nebadon Izumi: i agree that core isnt suitable currently
[11:51]  Robert.Adams scaling always needs config changes
[11:51]  Nebadon Izumi: but we cant wait months again either really
[11:51]  James Stallings: if it were in core we would test it to the breaking point
[11:51]  Seth.Nygard although not part of any standard linux distro, I have had good luck with ZFS on linux for very large datastores. It is also capable to doing snapshots which can be leveraged for backups.
[11:51]  Dahlia.Trimble ya a nosql replicating approach would be a better choice I'd think
[11:51]  Andrew.Hellershanks James, we do have a way of breaking things. :)
[11:51]  Dahlia.Trimble like cassandra
[11:52]  Allen.Kerensky asset clusters with shared storage etc - cost prohibitive however
[11:52]  Dahlia.Trimble I had tried riak a couple years ago but didnt get far
[11:52]  Nebadon Izumi: if we have to wait months to get osgrid back up while something gets developed and tested, i just don't see it coming back
[11:52]  Robert.Adams someone needs to make an S3 based asset store :)
[11:52]  Justin Clark-Casey: You wouldn't expect to run a huge website on Apache alone. There have to be a number of components in play
[11:52]  Nebadon Izumi: we have limited funds that are quickly being depleted paying for servers that are not being used
[11:52]  Robert.Adams put it in core and get OSGrid up!
[11:52]  Marcus.Llewellyn Has using the MOSES/Simian asset service been explored?
[11:52]  James Stallings: you have to remember, osgrid uses a few servers. The asset servers especially are not cheap. People wont keep paying for them to sit idly for long.
[11:53]  Nebadon Izumi: nah no freaking way
[11:53]  Justin Clark-Casey: which is why I don't think an open grid model works. But clearly you guys don't agree
[11:53]  Justin Clark-Casey: so I will shut up about that
[11:53]  Nebadon Izumi: MOSES uses simian grid
[11:53]  Nebadon Izumi: and its horrible
[11:53]  James Stallings: they've already sat idly for months.
[11:53]  Robert.Adams just do it.... OpenSim will always be one the boundry of 'clean core' and 'full of fun stuff'
[11:53]  Nebadon Izumi: who would decide who can and can't be on OSgrid
[11:54]  Dahlia.Trimble I asked cmickyb about using simian assets with a robust grid, he thought it would work but there are some hypergrid issues that need to be addressed first
[11:54]  Arielle.Popstar why so adamant on restoring osgrid to its former state  vs starting empty?
[11:54]  Nebadon Izumi: i dont think you realize what your proposing justin
[11:54]  Justin Clark-Casey: and hwo about the endless bug rpoerts of 'fun stuff' that was never written properly and was long abandoned?
[11:54]  Justin Clark-Casey: nebadon: how do I not?
[11:54]  Marcus.Llewellyn I guess it's a balance. I've always known OSgrid could crash and burn. I'm suprised it didn't earlier. But I completely appreciate the admins wanting to future-proof things too.
[11:55]  Robert.Adams justin, those bug reports will do what that type of bug report does now.... waits for a volunteer who cares
[11:55]  Robert.Adams important things get fixed
[11:55]  Justin Clark-Casey: which is never in many cases
[11:55]  Dahlia.Trimble I wish you would have brought these issues up a few months ago when I had some time to look at them
[11:55]  Andrew.Hellershanks I was only surprised there weren't other backups of the osgrid dbs
[11:55]  Justin Clark-Casey: and when you have a project full of bugs it's low quality and people abandon it
[11:55]  Robert.Adams an strange interaction with a particular terrain painting tool will not get fixed right away
[11:56]  Nebadon Izumi: well honestly I havnt had the time either, but now we have people willing to do the work
[11:56]  Nebadon Izumi: but we want whatever we decide to do to benefit opensim
[11:56]  Nebadon Izumi: going with SRAS again will not
[11:56]  Nebadon Izumi: and will likely lead to repeat of this whole thing
[11:56]  Justin Clark-Casey: getting a bunchj of code inside core that people may well not maintain will not help either
[11:56]  Robert.Adams 'full of bugs' is not binary.... it's a gray area.... if the important bugs are fixed, people will use it
[11:57]  Marcus.Llewellyn Is Melanie's xasset code in production use, maybe at Avination?
[11:57]  Nebadon Izumi: not getting in in core doesnt insure it gets maintained either
[11:57]  Nebadon Izumi: lol
[11:57]  James Stallings: yes
[11:57]  Nebadon Izumi: so who cares
[11:57]  Justin Clark-Casey: of course it does, you think I do nothing all the time
[11:57]  Robert.Adams and I would agree that everything shouldn't be thrown into core
[11:57]  Justin Clark-Casey: I spend practically all my time trying to maintain things
[11:57]  Andrew.Hellershanks How is xassets different from core assets in terms of what it does?
[11:57]  BlueWall.Slade dedup
[11:57]  James Stallings: blobs on disk
[11:57]  Andrew.Hellershanks that's it?
[11:57]  James Stallings: those are the two biggies
[11:57]  BlueWall.Slade yes
[11:58]  Justin Clark-Casey: xassets does not do blobs on disk as I wrote
[11:58]  Marcus.Llewellyn I very much doubt anyone thinks that, JCC. Anyone who sees the channel bots on IRC knows you're doing a great deal.
[11:58]  Nebadon Izumi: its basicall a better version of whats in core now
[11:58]  Justin Clark-Casey: if melanie has private changes which do that then that's something else
[11:58]  Dahlia.Trimble I dont see a problem with having highly scalable services in core but I agree that they probably wont be maintained as most of the people who use them will probably have closed large grids and wont share
[11:58]  Nebadon Izumi: I don't really see how that is bad
[11:58]  BlueWall.Slade xassets in core stores blobs in database
[11:58]  James Stallings: I dont think you do nothing Justin, really; I just know you arent superhuman
[11:58]  Robert.Adams we're talking around a gray area.... can't put everyone's abandable pet project in core.... but can't exclude everything that is not military hardened
[11:58]  Nebadon Izumi: let me see if i can get melanie
[11:58]  Justin Clark-Casey: I have to go and do other things now
[11:59]  Justin Clark-Casey: it's the top of the hour
[11:59]  Dahlia.Trimble bb jcc
[11:59]  James Stallings: we should distinguish between core xassets and avination xassets in this discussion I think
[11:59]  Marcus.Llewellyn Take care, Justin.
[11:59]  BlueWall.Slade ++
[11:59]  Richardus.Raymaker bye justin
[11:59]  James Stallings: thanks Justin
[11:59]  BlueWall.Slade See ya Justin, take care
[11:59]  Nebadon Izumi: ok melanie responded
[11:59]  Aine.Caoimhe bye Justin
[11:59]  Seth.Nygard Have a great night Justin, take care and thanks for the help
[11:59]  Nebadon Izumi: oh your leaving?
[11:59]  Nebadon Izumi: sigh
[11:59]  Justin Clark-Casey: I'm sure there will be more discussion on this but I think something majro should be on the mailing list and not just put into core as is
[12:00]  James Stallings: what did Mel respond?
[12:00]  Nebadon Izumi: i was going to have her come here
[12:00]  James Stallings: ah
[12:00]  Nebadon Izumi: but if justin is leaving nevermind
[12:00]  Justin Clark-Casey: alright, let me stick around for 15 mins
[12:00]  BlueWall.Slade it would behave like coupling the asset cache to the backend?
[12:00]  James Stallings: /|\ thanks Justin
[12:01]  Nebadon Izumi: nice
[12:01]  Nebadon Izumi: hello Melanie
[12:01]  Melanie Milland: hi
[12:01]  James Stallings: o/
[12:01]  Andrew.Hellershanks Hey, Melanie
[12:01]  BlueWall.Slade hi Melanie
[12:01]  Arielle.Popstar Hi Mel
[12:02]  Nebadon Izumi: ok so, we are trying to get OSgrid resolved
[12:02]  Nebadon Izumi: and we are wondering about your version os xassets
[12:02]  Nebadon Izumi: some are not sure it is suitable for core inclusion
[12:02]  Nebadon Izumi: hoping you could maybe fill us in on what its all about
[12:02]  Nebadon Izumi: whats different than what is already in core
[12:03]  Melanie Milland: i don't know why anyone whould think it's not suitable for core without having seen it
[12:03]  Justin Clark-Casey: you have added code to the core xasstes for storing blobs on disk rather than in the db?
[12:04]  Melanie Milland: i thought xassets in core was already blobs. if not, yes, ours is
[12:04]  Melanie Milland: i mean, on-disk blobs
[12:04]  Melanie Milland: we also gzip them where it saves space
[12:04]  Dahlia.Trimble multiple file systems? replicatable?
[12:04]  Justin Clark-Casey: xassets in core stores blobs in db. One reason was to not create a more complicated backup situations wher eboth file system stuff and db has to be backed up
[12:04]  Justin Clark-Casey: to make core xassets usable out of the box without making things more complicated for the user
[12:04]  Andrew.Hellershanks The gzip/gunzip process doesn't add excessive overhead to handing out the blobs to viewers?
[12:05]  BlueWall.Slade Melanie, is it sort-of like placing the asset cache in the backend?
[12:05]  BlueWall.Slade similar?
[12:05]  Melanie Milland: we find that gunzip is very lightweight and gzip isn't too frequent to be an issue
[12:05]  Dahlia.Trimble large assets like meshes and textures are already compressed
[12:05]  Melanie Milland: correct - as i said, it compresses only where there are saving afair
[12:06]  Melanie Milland: so that hits mainly notecards and scripts
[12:06]  Dahlia.Trimble and linksets
[12:06]  Melanie Milland: indeed
[12:06]  BlueWall.Slade We already have the file-based asset cache in OpenSim, so I guess it wouldn't be a huge stretch
[12:06]  Melanie Milland: justin, what would be the reason to not include a disk based asset service in core? we're all about choices, or used to be anyway
[12:07]  Justin Clark-Casey: melanie: one concern is greater complexity for moderate users. If they use the option, the backup approach becomes more complex
[12:07]  Andrew.Hellershanks There was some concern it would get put in core and then be unmaintained
[12:07]  Justin Clark-Casey: you can't just say "backup the database" any more
[12:07]  Nebadon Izumi: ya but its optional and not by default
[12:07]  BlueWall.Slade that is on the user Justin
[12:07]  Nebadon Izumi: why would that really matter
[12:08]  Justin Clark-Casey: if something is optional it does get used and should be supported
[12:08]  BlueWall.Slade many are in over their head already
[12:08]  James Stallings: lol
[12:08]  Shez.Oyen :o
[12:08]  James Stallings: arent we all heh
[12:08]  Dahlia.Trimble I like the current core asset server for my small grid, it's very easy to back up. I wouldnt want to use it for anything much larger tho
[12:08]  Nebadon Izumi: why should we dictate how people do their backups?
[12:08]  BlueWall.Slade I would +1 inclusion of it
[12:08]  Justin Clark-Casey: melanie: actually, just to clear things up, if the xassets service you are talking about one that builds on the code in core or is it entirely separate?
[12:08]  Robert.Adams just like turning on HG... there is a lot of stuff a user has to know and account for..... turning on blob storage would be another of those
[12:09]  Marcus.Llewellyn Would this replace current robust assets? Or just be another option? If the former, people wanting a simpler setup can choose the old asset server.
[12:09]  BlueWall.Slade That is as long as the current default remains the same.
[12:09]  Marcus.Llewellyn Novices will usually use SQLite anyway.
[12:09]  Nebadon Izumi: its not like we teach people how to do database backups in the first place
[12:09]  James Stallings: I wouldn't want it on by default
[12:09]  Nebadon Izumi: something they need to learn on their own
[12:09]  Melanie Milland: justin: ours is called FSAssets and descends from core assets, but it's seen a lot of revisions since then so may be significantly different in places
[12:09]  BlueWall.Slade We should keep assets + xassets
[12:09]  James Stallings: I just want to be able to use it
[12:09]  Justin Clark-Casey: ok, so we are talking about two differnet things
[12:09]  Andrew.Hellershanks I've got a 400+G asset table and wondering if I need to migrate it to something else than stock core asset system.
[12:10]  Justin Clark-Casey: melanie: if you put fsassets into core would you be willing to spend time to maintain it?
[12:10]  Nebadon Izumi: I am surrpised you havent ran into trouble Andrew
[12:10]  Nebadon Izumi: how many users?
[12:10]  James Stallings: yeah that kinda had me wondering too
[12:10]  Nebadon Izumi: and how do you back that up?
[12:10]  BlueWall.Slade ++ - at least until is sees some use and bugs are reported.
[12:11]  Andrew.Hellershanks Nebadon, not sure of the numbers off hand. May be couple thousand but in the hundreds, re: active users.
[12:11]  Melanie Milland: justin: we maintain it anyway, it's what runs avination
[12:11]  Dahlia.Trimble can it be easily backed up and/or replicated? if not, all this is moot anyway
[12:11]  Justin Clark-Casey: melanle: does it have an optino to store in db or is it filesystem only?
[12:11]  Melanie Milland: whether or not we copy that to core is a nonissue in terms of work
[12:11]  Andrew.Hellershanks Dahlia, it would need to have both DB bacup and file system backup.
[12:11]  Melanie Milland: as long as no one goes and does refactoring on it to make it diverge too much
[12:12]  Andrew.Hellershanks It might complicate backing up both in terms of making sure they are both in sync.
[12:12]  Justin Clark-Casey: of course, any code that goes into core can be touched by any core developer
[12:12]  Melanie Milland: it stores assets in the filesystem, always. blobs are BAD for mysql so they're not supported - what would be the point?
[12:12]  Marcus.Llewellyn Does Avination's code contain a migration tool/strategy for going from current assets to the new one?
[12:12]  Justin Clark-Casey: I've never read anything that definitely says blobs are terrible in mysql
[12:13]  Allen.Kerensky percona, maria 10, and mysql 5.6 are supposed to handle blobs betterer
[12:13]  Dahlia.Trimble Melanie, how about backups and/or replication?
[12:13]  BlueWall.Slade I think that's a religious argument
[12:13]  BlueWall.Slade hehe
[12:13]  Andrew.Hellershanks Justin, neither have I but I keep hearing people say its bad.
[12:13]  Melanie Milland: marcus: we made a copy-on-read connector at some point but i couldn't say where it currently is
[12:13]  James Stallings: BlueWall: we just dodgin what done bit us before
[12:13]  BlueWall.Slade ++
[12:14]  Melanie Milland: justin: bug fixes and enhancements are one thing, wholesale refactoring is another. after all, this is tried and tested code, it should not need anyone's hands on it
[12:14]  BlueWall.Slade but I have heard both arguments about it.
[12:14]  James Stallings: I hadnt really heard either, just workin off direct experience ;)
[12:15]  Justin Clark-Casey: melanie: one can't impose conditions on code that comes into core. If changes are necessary then anybody can make them.
[12:15]  Melanie Milland: dahlia: fsassets doesn't handle either. there are tools out there that can do that job much better. avination is fully replicated using mysql replication and drbd
[12:15]  Justin Clark-Casey: a service can always be provided outside of core - by going into core it becomes something else
[12:15]  James Stallings: lets ot be tedious; that's pretty fundamental to the nature of the project
[12:15]  Nebadon Izumi: past experience for OSgrid blobs in the database was not good, but I can't really say ive seen that in writing anyway, only my own very horrible experiences
[12:15]  Andrew.Hellershanks just had the thought that gunzip on assets on disk isn't such a big deal as the regions would cache some of the assets. Biggest hit is always on startup with empty caches.
[12:15]  BlueWall.Slade Conditions can't be imposed - but I really doubt that it would see huge changes
[12:15]  Melanie Milland: justin: no one is talking about conditions
[12:16]  BlueWall.Slade many eyes on it may find some improvements though.
[12:16]  BlueWall.Slade those are usually small
[12:16]  Billy.Bradshaw Would have been advantageous for this issue to have been raised on the dev mailing list, at least people would be better prepared.
[12:16]  Nebadon Izumi: it wont be a forced change
[12:16]  Nebadon Izumi: there is no real reason to prepare anyone for anything
[12:16]  Melanie Milland: justin: but if someone refactors stuff just because they don't like names or ordr of methods in a file, then that would be something i'd dislike
[12:16]  Melanie Milland: because it makes work for me
[12:16]  Marcus.Llewellyn Presumably, Avination would welcome fixes or improvements, so long as they were reasonably easy to merge.
[12:16]  James Stallings: we just need the codez :3
[12:17]  Justin Clark-Casey: those things can always change. But as BlueWall says, if it already makes sense there's no reason to touch it.
[12:17]  BlueWall.Slade Melanie, is it compatible with the code in core now - names, format, etc.?
[12:17]  Robert.Adams JCC, Mel is saying smaller changes allows cross-merging and is a Good Thing.... major re-arrangement of code makes cross-merging nearly impossible
[12:17]  Dahlia.Trimble naming or code ordering is subjective anyway, there is no "right or wrong"
[12:18]  Justin Clark-Casey: if something is put into opensim then it can get changed. OpenSim should not be held static by proprietary code
[12:18]  Justin Clark-Casey: if someone wants proprietary code then that is fine, but it cannot be allowed to hold back change in an open-source project
[12:18]  BlueWall.Slade I think it would be a plus
[12:19]  Marcus.Llewellyn I don't think anyone suggested that the contribution would be propietary. I only saw a request for consideration in merging changes.
[12:19]  BlueWall.Slade we should keep current assets + xassets and add this one as whatever Melanie want's to anme it.
[12:19]  Robert.Adams not suggesting static... why take things to extreme... just know that if major refactoring is done then changes to and from avination won't be able to happen
[12:19]  Melanie Milland: justin, you're flogging the shadow of the stain that used to be in the place where a dead horse used to lie
[12:19]  Robert.Adams it's not the end of the world... but it is not optimal
[12:19]  BlueWall.Slade lol, Melanie
[12:19]  Melanie Milland: the point i'm making is that changes only designed to make work for me would not be welcome
[12:19]  Melanie Milland: note, ONLY DESIGNED
[12:20]  Justin Clark-Casey: Changes are not made just to make work for you.
[12:20]  Melanie Milland: functional enhancements, fixes, etc are always welcome in core
[12:20]  BlueWall.Slade Melanie, if you bring it in, will you make it a 3rd option?
[12:20]  BlueWall.Slade or replace xassets
[12:21]  Melanie Milland: yes. i see no point in removing any of the existing services
[12:21]  Marcus.Llewellyn Good.
[12:21]  Melanie Milland: they do appear to be in use
[12:21]  BlueWall.Slade I like having a deduping option w/o writing blobs to disk too.
[12:21]  Marcus.Llewellyn My grid is young and tiny. I'd like to have dedup sooner rather than later.
[12:21]  BlueWall.Slade I think xassets, not so much. But it would be useful when stable.
[12:21]  Nebadon Izumi: do we currently have that now BlueWall?
[12:21]  Nebadon Izumi: I didnt think we did
[12:22]  BlueWall.Slade Nebadon, we do.
[12:22]  Melanie Milland: avn started without dedup. took months to process the data later
[12:22]  BlueWall.Slade xassets dedupe, but keep blobs in the database.
[12:22]  Nebadon Izumi: ok cool i didnt think dedupe was working
[12:22]  Nebadon Izumi: thats good
[12:22]  Nebadon Izumi: not sure why I thought that, i thought it was incomplete
[12:22]  BlueWall.Slade It is not done - per warnings
[12:22]  Nebadon Izumi: ah ok
[12:22]  Justin Clark-Casey: nebadon: It is incomplete, because there are two things I want to do to it
[12:22]  BlueWall.Slade could change out from under you
[12:22]  Nebadon Izumi: I didnt realize it was functional at all
[12:22]  Justin Clark-Casey: as I said earlier
[12:22]  Nebadon Izumi: yea
[12:23]  Justin Clark-Casey: it is completely functional
[12:23]  Nebadon Izumi: i guess i misunderstood
[12:23]  Andrew.Hellershanks I have access to an older 100G or so DB and found it had around 40% duplication of asset data.
[12:23]  James Stallings: the mig tool shouldnt be THAT heinous
[12:23]  Melanie Milland: justin, what are you planning?
[12:23]  James Stallings: if I had good before and after schemata I could do that easily
[12:23]  Nebadon Izumi: actually we have one already
[12:23]  Nebadon Izumi: probably could be modified slightly
[12:23]  Nebadon Izumi: andrew wrote it
[12:23]  Nebadon Izumi: its php-cli
[12:23]  Nebadon Izumi: worked very well
[12:23]  James Stallings: nice
[12:23]  James Stallings: that woould be my precise approach
[12:23]  Justin Clark-Casey: melanie: I want to make sure the core xassets isn't missing any fields it might be good to have. Then I wanted to work out a completely migratio nstrategy from existing assets
[12:23]  Andrew.Hellershanks Yeah, I was about to mention my migration script. It can be modified as needed.
[12:24]  BlueWall.Slade ++ on the migration part.
[12:24]  James Stallings: the tl;dr is a mig tool in this crowd is not problematic
[12:24]  BlueWall.Slade that could probably work on both the xassets and melanies
[12:24]  Nebadon Izumi: i have it packaged in my sras kit
[12:24]  Nebadon Izumi: if you want it
[12:25]  Justin Clark-Casey: james: alrigjht. So let's say I can try to complete what I wanted to do on core xassets
[12:25]  James Stallings: we may need it
[12:25]  Nebadon Izumi:
[12:25]  Nebadon Izumi: its in the zip
[12:25]  Justin Clark-Casey: james: which is relatively simple. The step of allowing complete migration can be added later. In fact, it doesn't matter if you are starting afresh
[12:25]  Andrew.Hellershanks I was also thinking that the migration script can be run on a live system to migrate the assets up to a certain date and then a grid can be taken down to migrate the assets after that date to minimize grid down times during migration
[12:25]  James Stallings: nods
[12:25]  Justin Clark-Casey: james: so then you have the option of either using the existing core assets or melanie's new fsassets when she addes it
[12:25]  Justin Clark-Casey: core xassets
[12:26]  James Stallings: they'd both be core at that point, but yeah...
[12:26]  Nebadon Izumi: on of the reasons we are keen on using melanies
[12:26]  Justin Clark-Casey: I'm not trying to undercut melanie, I just feel bad now
[12:26]  Nebadon Izumi: is because there is a small chance we might still be able to recover sras assets
[12:26]  Nebadon Izumi: and push them into xassets
[12:26]  Melanie Milland: why do you feel bad?
[12:26]  Justin Clark-Casey: but yeah, if you want assets on fs you will want melanie's fsasset
[12:26]  James Stallings: you shouldn't
[12:26]  Andrew.Hellershanks Are we now talking about having 3 asset storage systems? core, xassets, and fsassets?
[12:26]  Justin Clark-Casey: because this is the thing stopping osgrid from coming back up
[12:26]  James Stallings: you arent superman bub
[12:26]  BlueWall.Slade With both in core, we would have a small, medium and large solution
[12:26]  Justin Clark-Casey: some kind of sustainable asset strategy
[12:26]  James Stallings: and you did not know that
[12:26]  Justin Clark-Casey: although I think an open tgird is fundamentaly flawed in this respect
[12:26]  James Stallings: hell, we didnt know your code was diff from mels
[12:26]  Nebadon Izumi: ya beleive me justin
[12:27]  Nebadon Izumi: my goal here is to minimize work load
[12:27]  Nebadon Izumi: and not repeat this whole scenario 6 months from now
[12:27]  James Stallings: mine is to get our admittedly suicidal grid back up
[12:27]  Justin Clark-Casey: neb: ok, let's be clear. core xassets is what I wrote and it is in core but experimental. Melanie's fsaset s is a separate service which stores blobs on fs instead of db, as far as I'm hearing
[12:27]  Nebadon Izumi: sure
[12:27]  Nebadon Izumi: that sounds great to me
[12:27]  James Stallings: I think we've been working miracles with it for years and I'm not finished yet
[12:27]  James Stallings: we're not finished yet.
[12:27]  Nebadon Izumi: and I am sorry if I didnt explain things as good as possible
[12:28]  Nebadon Izumi: I also am juggling many projects
[12:28]  BlueWall.Slade Long term, it would be good to have migration for straight assets ---> xassets ----> fassets
[12:28]  Melanie Milland: well, as i outlined in chat the other day, we have tried and proven replication strategies and also growths paths for fsassets
[12:28]  Nebadon Izumi: currently in sweden, so im a bit scrambled
[12:28]  Andrew.Hellershanks Aren't many of us jugglingtoo many projects?
[12:28]  Justin Clark-Casey: so perhaps you guys could go away and decide if you want to use xassets or fsassets. Then let one of us know?
[12:28]  Justin Clark-Casey: unless Melanie was planning on putting fsassets in core anyway
[12:28]  Justin Clark-Casey: imminently
[12:28]  Marcus.Llewellyn I'm still confused on one point. Does Mel's service do *only* the filesystem, or can it do both fs and db?
[12:28]  Nebadon Izumi: personally id prefer to start with fsassets
[12:28]  Nebadon Izumi: if melanie is game
[12:29]  James Stallings: lol I would hope you both contribute your work
[12:29]  Nebadon Izumi: but honestly im not doing the work
[12:29]  Nebadon Izumi: Allen and James?
[12:29]  Nebadon Izumi: what say thee
[12:29]  Melanie Milland: marcus: fsassets does a db that is used as an index to the blobs on the fs, which are deduped
[12:29]  James Stallings: which is the quicker path to grid upness?
[12:29]  Melanie Milland: it doesn't ever store assets as blobs
[12:29]  Marcus.Llewellyn Melanie: So id does not and will not store asset blobs in the DB.
[12:29]  Nebadon Izumi: its basically SRAS
[12:29]  Nebadon Izumi: but in C#
[12:30]  Melanie Milland: marcus: correct. use xassets if that is what you want
[12:30]  BlueWall.Slade ++
[12:30]  Justin Clark-Casey: marcus: expect not yet
[12:30]  Marcus.Llewellyn Thanks. All clear now. :)
[12:30]  Dahlia.Trimble OSGrid has always been somewhat experimental, and that's been very valuable to OpenSimulator. I say go ahead with Melanie's service if she's game
[12:30]  James Stallings: It would help to know what kind of timeframe Mel can afford in terms of pushing FSAssets
[12:30]  Nebadon Izumi: Melanie what kind of turn around
[12:30]  James Stallings: "no pressure" LOL
[12:30]  Nebadon Izumi: hate to pressure you, but ball is kind of in your court now
[12:30]  Nebadon Izumi: lol
[12:30]  Melanie Milland: i can likely manage it this week
[12:30]  Marcus.Llewellyn Yay! (insert happy dance)
[12:31]  Melanie Milland: if neb pokes me a couple of times - mind like a sieve ;)
[12:31]  BlueWall.Slade ++
[12:31]  Nebadon Izumi: if there is one thing I am expert at its poking
[12:31]  Melanie Milland: k
[12:31]  James Stallings: Justin, I would still love to see your polish to Xassets
[12:31]  Dahlia.Trimble how about restoring the recovered data? can that work with fsassets?
[12:31]  James Stallings: I would like for us to come out with a mig tool that can mig data all around
[12:31]  Marcus.Llewellyn I think that's the whole point, Dahlia.
[12:31]  Andrew.Hellershanks That's why I use Tomboy for keeping an electronic ToDo list.
[12:31]  BlueWall.Slade Justin, the xassets is a really good mid-level soulution
[12:31]  Nebadon Izumi: if the data is good, possibly dahlia
[12:31]  Melanie Milland: so i guess that is settled
[12:31]  Nebadon Izumi: but it kind of gets tricky
[12:31]  Melanie Milland: Welcome to the Dark Side. Are you surprised we lied about the cookies?
[12:31]  Marcus.Llewellyn Well, not the whole point.
[12:31]  Nebadon Izumi: we may need help with that
[12:32]  Dahlia.Trimble does an importing procedure exist?
[12:32]  Allen.Kerensky well there you go - I am game for whatever folks want - and yes, Xassets is what I'd like for my private grid - just one db to wrangle and not a lot of files I already have millions
[12:32]  James Stallings: I thought theCAKE was a lie. No one said there would be cookies.
[12:32]  Nebadon Izumi: importing assets is easy
[12:32]  Melanie Milland: if i get access to the data i can tell you
[12:32]  Nebadon Izumi: merging in old inventory
[12:32]  Nebadon Izumi: not so
[12:32]  Nebadon Izumi: im sure it can be done
[12:32]  Marcus.Llewellyn What I want most is dedup. DB or FS isn't as relevant to me.
[12:32]  Nebadon Izumi: but its not as simple as the assets
[12:32]  Dahlia.Trimble easy, assuming the procedure exists
[12:32]  Melanie Milland: in avn, we have transparently and without downtime handled several catastrophic hardware failures
[12:32]  Melanie Milland: i'm pretty good with mangling big data by now
[12:32]  James Stallings: +100
[12:33]  Allen.Kerensky does FSassets also support HGAssetService?
[12:33]  Nebadon Izumi: and Melanie you said you might be able to lay hands on osgrid hardware?
[12:33]  Nebadon Izumi: ]\\
[12:33]  James Stallings: excellent question Allen
[12:33]  Melanie Milland: of course. it supports the asset protocol, HG builds on it
[12:33]  James Stallings: :)
[12:33]  BlueWall.Slade plug&pray
[12:34]  BlueWall.Slade :)
[12:34]  Arielle.Popstar lol
[12:34]  Andrew.Hellershanks How assets are saved shouldn't have any real impact on anything else. Assets are requeste and passed around as they always have been. Just the source of them has changed.
[12:34]  Melanie Milland: well, i have a soft spot for osgrid
[12:34]  Melanie Milland: so i'll be there when it counts
[12:34]  James Stallings: dont we all :)
[12:34]  Nebadon Izumi: great, thanks Melanie
[12:34]  Melanie Milland: after all, i'm past my health issues somewhat
[12:34]  Nebadon Izumi: that means a lot to me personally
[12:34]  James Stallings: Thanks Mel
[12:34]  Nebadon Izumi: I wish I had more time myself
[12:34]  Allen.Kerensky yeah everyone has been really great during this mess - it is much appreciated
[12:34]  Shez.Oyen I'm greatful Melanie
[12:34]  James Stallings: especially the users
[12:34]  Nebadon Izumi: I really appreciate what James and Allen and Dan have been doing
[12:34]  Andrew.Hellershanks Melanie, Good to hear you are (mostly) better
[12:34]  Nebadon Izumi: in my absence
[12:35]  Allen.Kerensky I've bothered Diva and Justin with questions too lol everyone has been supportive
[12:35]  Justin Clark-Casey: I honestly tihnk you guys should serious think about the nature of the grid though
[12:35]  Nebadon Izumi: we are
[12:35]  Nebadon Izumi: things are not going to be the same
[12:35]  Nebadon Izumi: its a whole new ball game
[12:35]  James Stallings: we want the cieling to be high when we crash into it for good
[12:35]  Allen.Kerensky that's what the "restructuring" headline in the updates is sort of about
[12:35]  Justin Clark-Casey: the architecture of an open grid is really difficult - sl just wasn't designed for this kind of thing
[12:36]  Justin Clark-Casey: can you say anything about those things or would you rather wait?
[12:36]  Allen.Kerensky mostly me asking stupid questions about what needs to be done with each service OSG had configured lol
[12:36]  James Stallings: well we're looking into certain backend changes
[12:37]  James Stallings: replication and HA filesystems are front and center
[12:37]  Nebadon Izumi: most of the initial changes will be about having solid backups
[12:37]  Marcus.Llewellyn I've always regarded OSgrid's open grid model as an aberration. It isn't how OpenSim is meant to run. But I bet, at least for core services, it has been good testbed and source of reports from the admins. It has other less tangible benefits for users.
[12:37]  James Stallings: yes
[12:37]  Nebadon Izumi: we have a replication asset server in place now
[12:37]  Dahlia.Trimble an abberation thats being copied a lot
[12:37]  James Stallings: the ability to perform hot incremental backups is a key need we are attempting to address
[12:37]  Allen.Kerensky for example, do we need jump4000/jump8000 regions anymore or can we recover the resources for those
[12:37]  Nebadon Izumi: mysql replication and possibly some other methods that melanie will guide us with for replicating blobs
[12:38]  Nebadon Izumi: nothings 100% decided there yet
[12:38]  Nebadon Izumi: still exploring best solution that fits our servers
[12:38]  Marcus.Llewellyn The only time I've run into 4000/800 scenarios is when trying to HG to someone running 0.7.6. I'm looking at you, Caltech.
[12:38]  Andrew.Hellershanks Allen, I thought the TP limit is gone now so jump regions should be a thing of the past
[12:38]  Allen.Kerensky yeah just me questioning stuff to see how much I've gotten myself into
[12:38]  Nebadon Izumi: we dont need those anymore
[12:38]  Allen.Kerensky Andrew exactly why I was bothering them about it hehehe
[12:38]  Nebadon Izumi: they should go away for sure
[12:38]  James Stallings: lol caltech
[12:39]  Melanie Milland: OSG would be better off with 4 servers @3tb rather than 2 servers @6tb
[12:39]  Justin Clark-Casey: melanie: I think notice about a new assets esrvice should be posted to the dev mailing list first so other devs will know what's going on
[12:39]  Nebadon Izumi: honestly though other than this asset mess
[12:39]  Marcus.Llewellyn +1 mailing list
[12:39]  Nebadon Izumi: all of the other services have held up amazingly well
[12:40]  Nebadon Izumi: having opensimulator behind nginx and balancing things works very well
[12:40]  James Stallings: Mel: financial constraints
[12:40]  Nebadon Izumi: osgrid is not just stock opensiim exposed to the world
[12:40]  Nebadon Izumi: its very different than say how OSCC grid is setup
[12:40]  Nebadon Izumi: its very well protected by nginx
[12:40]  Justin Clark-Casey: but you still have the big vulnerability of having to expose the private servivce port/s to region operators
[12:41]  Melanie Milland: well, that is mostly a content risk
[12:41]  Melanie Milland: which is why there isn't commercial content
[12:41]  Nebadon Izumi: yea content on OSgrid cant be protected
[12:41]  Marcus.Llewellyn Many things on OSgrid are vulnerable. Content permission are, for all practical purposes, useless there. It's a price one pays for joining that grid.
[12:42]  Justin Clark-Casey: inventory operations are also a problem
[12:42]  Nebadon Izumi: even if those were not exposed, anyone hosting a region can snag any content
[12:42]  Nebadon Izumi: well we dont allow deletes
[12:42]  Nebadon Izumi: nothing can ever really be permantly deleted
[12:42]  Justin Clark-Casey: as are various other service operations
[12:42]  James Stallings: it's a job of mitigation and mediation
[12:42]  James Stallings: we learn a lot from it
[12:42]  Justin Clark-Casey: have you looked into storing assets in the cloud?
[12:42]  Marcus.Llewellyn And users accept those risks too. Or should.
[12:42]  James Stallings: it's pricey
[12:42]  Justin Clark-Casey: let someone else take care of the baclup problem
[12:43]  Justin Clark-Casey: ok
[12:43]  Nebadon Izumi: we have looked into cold storage
[12:43]  Nebadon Izumi: via amazon
[12:43]  James Stallings: yeah I was about to mention that
[12:43]  Nebadon Izumi: its cheap
[12:43]  James Stallings: slow but viable as a backup solution
[12:43]  Nebadon Izumi:
[12:43]  Melanie Milland: justin: "the cloud" just means "servers you don't control". for the huge space and bandwidth assets need, that isn't workable financially and even "cloud" servers can lose data
[12:44]  Melanie Milland: and you'll have NO recovery from that
[12:44]  Melanie Milland: because they'll never send you the HD
[12:44]  Nebadon Izumi: yea its long process to get data back unfortunately
[12:44]  Nebadon Izumi: thats what happened with osgrid originally
[12:44]  Melanie Milland: i don't know why it took this long, tbh
[12:44]  Nebadon Izumi: for a long time i was runningn backups to servers at my house
[12:45]  Nebadon Izumi: but the hardware eventually died
[12:45]  Andrew.Hellershanks Some cloud storage solutions claim you don't lose data as they copy the data to multiple places.
[12:45]  Nebadon Izumi: honestly I am a bit perplexed at why it took so long myself
[12:45]  Seth.Nygard ... until they actually do lose it ...
[12:45]  Andrew.Hellershanks Seth: yea, :)
[12:46]  Nebadon Izumi: I am not all that impressed with the recovery service that the data center uses
[12:46]  Nebadon Izumi: but we can't change that
[12:46]  Aine.Caoimhe htey don't lose it...they just can't find it
[12:46]  Marcus.Llewellyn I doubt anyone is. :/
[12:46]  Nebadon Izumi: what is done is done
[12:46]  James Stallings: this took so long because there were acually two full recovery operations.
[12:46]  Arielle.Popstar lol Aine
[12:46]  James Stallings: the first failed when delivered to us.
[12:46]  Melanie Milland: well, let me know when i can take a peek at the data
[12:46]  James Stallings: you need to see the old sras data mel?
[12:46]  Nebadon Izumi: ok we will for sure, step one is getting new fassets functional
[12:46]  Melanie Milland: yes
[12:47]  James Stallings: Ok I will get it 'hooked up'
[12:47]  Melanie Milland: i want to see how hard a reimport is
[12:47]  James Stallings: nods
[12:47]  Melanie Milland: it may be trivial
[12:47]  Melanie Milland: we did make a tool for it and i may even find it
[12:47]  James Stallings: I'll ping you on IRC when its avail
[12:47]  Marcus.Llewellyn From your fingers to fate's ears. :)
[12:47]  Nebadon Izumi: i suspect sras doesnt have the embedded data your server does
[12:47]  Nebadon Izumi: im like 98% sure it doesnt
[12:47]  Melanie Milland: we shall see
[12:47]  Andrew.Hellershanks Melanie, there is always my migration script that might be helpful.
[12:47]  Nebadon Izumi: but if we are luckly there should be 5 sql dumps
[12:47]  Nebadon Izumi: and if were really lucky the actual mysql structure is still valid
[12:48]  Melanie Milland: if the data were naked, that would be pretty indecent....
[12:48]  Nebadon Izumi: i give us 50/50 chance right now
[12:48]  Nebadon Izumi: either way grids coming back up
[12:48]  Melanie Milland: neb, you ain't seen nuthin' yet
[12:48]  Nebadon Izumi: hehe if anyone can do it its you Melanie
[12:48]  Melanie Milland: giving up just isn't in my book
[12:48]  Nebadon Izumi: i have no doubt of that
[12:49]  Andrew.Hellershanks Rebuilding the database from blobs on disk is a problem if there isn't extra data in the blobs that would assist in recreating the assets db.
[12:49]  BlueWall.Slade yeah, I want my "No Pants Weekend Barrel"
[12:49]  Shez.Oyen Melanie, if I can wash your car or walk your dog to give your more time for OSG, let me know!
[12:49]  Nebadon Izumi: lol
[12:49]  Marcus.Llewellyn I know it's not as valuable, but if you guys need additional crash test dummies in testing, from people who're willing to lose stuff, don't hesitate to put out that call.
[12:49]  Melanie Milland: i need to run
[12:49]  Nebadon Izumi: yea dont worry when it comes testing time, you all will know
[12:49]  BlueWall.Slade Thanks Melanie
[12:49]  Marcus.Llewellyn Thank you Mel. :)
[12:49]  Justin Clark-Casey: ok, I need to run too
[12:49]  Melanie Milland: you're welcome
[12:49]  BlueWall.Slade Thanks Justin
[12:49]  Melanie Milland: bye for now
[12:49]  Nebadon Izumi: ya thanks guys, this ran longer than i expected
[12:49]  Justin Clark-Casey: best of luck to you guys, I hope you get osgrid back up soon
[12:49]  Aine.Caoimhe ty all
[12:49]  Nebadon Izumi: but i am super happy we all had time finally
[12:49]  Aine.Caoimhe have a great week
[12:49]  Andrew.Hellershanks I'd also like to get going.
[12:50]  Jak Daniels: thank you everyone :)
[12:50]  Seth.Nygard have a great week everyone, take care
[12:50]  Nebadon Izumi: talk soon everyone we will keep everyone updated as we humanly can
[12:50]  Dahlia.Trimble sorry I stepped away. Does this mean Melanie will push the Avination asset server to core?
[12:50]  Arielle.Popstar good meeting
[12:50]  Andrew.Hellershanks Let me know if my scripts might be helpful in doing migrations and if they need mods.
[12:50]  Marcus.Llewellyn Dahlia, hopefully this week.
[12:50]  Nebadon Izumi: thanks Andrew we will
[12:50]  BlueWall.Slade I have to take off too - Thanks everyone. See you all next week.
[12:50]  Dahlia.Trimble cool
[12:50]  BlueWall.Slade is Offline
Personal tools
About This Wiki