Chat log from the meeting on 2015-01-20

 [10:59] Arielle.Popstar @login.greatcanadiangrid.ca:8002: heyas [11:00] Justin Clark-Casey: hi folks [11:00] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Hi Justin [11:00] Shez.Oyen @mirage.duckdns.org:9038: Mornin' Justin [11:00] Seth.Nygard @refugegrid.com:8002: Hello Justin [11:00] James Stallings: howdy Justin ;) [11:00]  Justin Clark-Casey: hey James, long time no see [11:00]  Arielle.Popstar @login.greatcanadiangrid.ca:8002: Hi Justin [11:00]  Justin Clark-Casey: Not sure if neb will be around today [11:00]  James Stallings: heehee, yeah, I been layin low :) [11:00] Aine.Caoimhe @refugegrid.com:8002: so glad to finally get a really good data capture on the latency issue...it's astounding just how badly a single latent viewer can torpedo a region and how it can snowball into a complete melt-down [11:01] Justin Clark-Casey: aine: yeah, it might be some effect from the number of resends required [11:01] Aine.Caoimhe @refugegrid.com:8002: yes, and it seems like the resends aren't throttled [11:01] Justin Clark-Casey: I still need to insert some debugging stuff to isolate the area - as usual all my other priorities jostling for attention [11:01] Justin Clark-Casey: but hope to that soon [11:02] Aine.Caoimhe @refugegrid.com:8002: so with that one user he managed to rack up 60k resends in less than 10 minutes [11:02] Aine.Caoimhe @refugegrid.com:8002: and that was backlogging the queue and putting everyone else into pseudo latency [11:02] Justin Clark-Casey: don't know for sure if that's the real issue yet - these things can be deceptive [11:02] Seth.Nygard @refugegrid.com:8002: I am very happy to do a build with whatever test changes you want to do so we can help track down what is happening and ideally find a workable fix. [11:02] Aine.Caoimhe @refugegrid.com:8002 nods [11:02] Justin Clark-Casey: but the pile up of resends is a candidate [11:03] Justin Clark-Casey: or possibly the process of queueing a resend in the first place [11:03] Richardus.Raymaker @hypergrid.org:8002: hi [11:03]  Aine.Caoimhe @refugegrid.com:8002: we also have a really good "lab rat" with a reliably horrid connection :p [11:03] Richardus.Raymaker @hypergrid.org:8002: dont pile up the piles [11:03] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I remember the map working in a what that could pack up the queue if something went wrong - was a while back [11:03] Aine.Caoimhe @refugegrid.com:8002: and he's willing to be used [11:04] Arielle.Popstar @login.greatcanadiangrid.ca:8002: how bad is bad? [11:04] Aine.Caoimhe @refugegrid.com:8002: he's on a sat connection [11:04] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Not good to arrive in time to see a question "how bad is bad" :) [11:04]  Aine.Caoimhe @refugegrid.com:8002: and then towards the end of each month he's over his limit so they throttle him even more [11:04]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: It's like, when something goes wrong we just keep blindly putting things in the queue [11:04]  Aine.Caoimhe @refugegrid.com:8002: so it's baaaaaaaaaaad [11:04]  Arielle.Popstar @login.greatcanadiangrid.ca:8002: :) [11:04] Aine.Caoimhe @refugegrid.com:8002: it's compelely expected that his inworld experience would be horrible [11:05] Aine.Caoimhe @refugegrid.com:8002: but it's a surprise that his connection can single-handedly bring a region to its knees [11:06] Aine.Caoimhe @refugegrid.com:8002: I also know of a few other users that I encounter frequently while HGing around who I suspect have significant latency in their connections for whatever reason since any time one of them enters a region it grinds to a halt [11:06] Justin Clark-Casey: I'm surprised we didn't experience this effect during the conference and prior testing [11:06] Richardus.Raymaker @hypergrid.org:8002: just get a switch with variable traffic shaping and you can simulate it i think also nice [11:06] Arielle.Popstar @login.greatcanadiangrid.ca:8002: not just from a high arc? [11:07]  Aine.Caoimhe @refugegrid.com:8002: I'm a bit surprised too, although maybe it was just blind luck [11:07] Justin Clark-Casey: hundreds of people though, from all kinds of connections [11:07] Justin Clark-Casey: but perhaps you're right [11:08] Aine.Caoimhe @refugegrid.com:8002: there aren't all that many with really horrible connections, and the OSCC is waaaaaaaaaaaaay above the norm for hardware [11:08] James Stallings: it could be soething specific to his sat connection [11:08] Justin Clark-Casey: yeah, could be a lot of things in play [11:09] Seth.Nygard @refugegrid.com:8002: His sat connection is perhaps a worst case, but is certainly not the only connection that can cause these issues [11:09] Aine.Caoimhe @refugegrid.com:8002: another user is Shawn from Craft....I always experience extreme sim issues when he enters a region (though not as bad as Danger can produce and usually the sim is able to catch up) [11:09] Seth.Nygard @refugegrid.com:8002: I see it with users that are tethered with cellular connections as well as some WiFi connected users [11:09] Aine.Caoimhe @refugegrid.com:8002: would be a case where if you get enough somewhat-latent viewers in a region it can snowball [11:10] Justin Clark-Casey: I'm also surprised it didn't show up with pCampbot, since that produces massive numbers of resends [11:10] Justin Clark-Casey: maybe one factor is the oscc regions being fairly low prim [11:10] Richardus.Raymaker @hypergrid.org:8002: hi robert [11:10] James Stallings: o/ [11:10]  Justin Clark-Casey: and with people sitting for the most part [11:10] Richardus.Raymaker @hypergrid.org:8002: hmm sat connections. :\ [11:11] Robert.Adams @home.livingroomcam.us:14559: hello all [11:11] Aine.Caoimhe @refugegrid.com:8002: perhaps something unique about the different between pCampbot and a "real" viewer? [11:11] Justin Clark-Casey: yeah, that's possible too [11:11] Justin Clark-Casey: basically, it's complicated :) [11:11]  Seth.Nygard @refugegrid.com:8002: I had tried to simulate things but was unable to come up with a set of conditions to trigger the issue. However when he, or others with bad connections come into a region the effect is very noticable. [11:11]  Aine.Caoimhe @refugegrid.com:8002: maybe his viewer is expecting responses in "x" time and when it doesn't receive them it starts to send out even more? [11:11]  Justin Clark-Casey: yeah, which is why live testing will now be valuable. I'll try and put the debug stuff in soon [11:12]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: could they have a modified viewer that is doing spmething deliberately? [11:12]  Seth.Nygard @refugegrid.com:8002: I'll be ready whenever you have that for me [11:12]  Aine.Caoimhe @refugegrid.com:8002: not that particular user [11:12]  Arielle.Popstar @login.greatcanadiangrid.ca:8002: shaWNS backpack is too heavy [11:12] Aine.Caoimhe @refugegrid.com:8002: he's actively happy to try to help us find and solve the issue [11:12] Marcus.Llewellyn @grid.verymad.net:8002 also vaguely wonders about badly scripted attachments. [11:12] Seth.Nygard @refugegrid.com:8002: No, I have had him do a complete re-install and tried a few other things [11:12] Aine.Caoimhe @refugegrid.com:8002: not in his case, Marcus....he can do it while Ruthed [11:13] Marcus.Llewellyn @grid.verymad.net:8002 nods amiably. [11:13] Aine.Caoimhe @refugegrid.com:8002: though that doesn't mean that scripted attachments aren't a contributing factor in whether a sim can recover [11:13] Arielle.Popstar @login.greatcanadiangrid.ca:8002: any idea what their ping sim is? [11:14] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online [11:14] Aine.Caoimhe @refugegrid.com:8002: his ping on a 50-ping test was averaging 1000ms with 10% drop iirc [11:14] Seth.Nygard @refugegrid.com:8002: His ping times can vary from 500mS to as much at 1600mS [11:14] Aine.Caoimhe @refugegrid.com:8002: or aws it 1500ms [11:14] Richardus.Raymaker @hypergrid.org:8002: 1000ms is very high [11:14] Aine.Caoimhe @refugegrid.com:8002: yeah, his connection is really, really horrid [11:14] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Colony on Mars? [11:14] Justin Clark-Casey: that's wha tI would expect from a lossy and overloaded sat connection [11:14] Aine.Caoimhe @refugegrid.com:8002: but it makes him a good lab rat [11:14] Richardus.Raymaker @hypergrid.org:8002: Nah, mas must be 2500ms [11:15] Arielle.Popstar @login.greatcanadiangrid.ca:8002: explains the probs i used to have on my old interleaved dsl connection [11:15] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online [11:15] Richardus.Raymaker @hypergrid.org:8002: hi sarah [11:15] Aine.Caoimhe @refugegrid.com:8002: he knows his inworlld performance is going to be attrocious and will live with it because he doesn't ahve any alternative [11:15] Aine.Caoimhe @refugegrid.com:8002: but what he doesn't want to do is kill the experience for eveyrone else [11:16] Richardus.Raymaker @hypergrid.org:8002: justin, maby you need to remove the dust from your dailup to test it :) [11:16]  Aine.Caoimhe @refugegrid.com:8002: but he would also prefer not to have to become a hermit [11:16]  Seth.Nygard @refugegrid.com:8002: Regardless of his own experience, he doesn't expect that to be great. But even he had no idea he had a significant negative impact on the whole region until we started digging into see what was happening. [11:16]  Dahlia.Trimble @grid.wwweb3d.net:8002: hi [11:16]  Marcus.Llewellyn @grid.verymad.net:8002: I'm curious what might happen if he used a client like radegast. [11:16]  Richardus.Raymaker @hypergrid.org:8002: hi [11:16]  Sarah Kline: hi Richardous [11:16]  Shez.Oyen @mirage.duckdns.org:9038: Hi Dahlia :) [11:16] Justin Clark-Casey: hi dahlia [11:17] Aine.Caoimhe @refugegrid.com:8002 nods [11:17] James Stallings: so, Justin, what have you been up to lately? ;) [11:17] Aine.Caoimhe @refugegrid.com:8002: or that new remote thing that FS was promoting [11:17]  Dahlia.Trimble @grid.wwweb3d.net:8002: remote thing? [11:18]  Shez.Oyen @mirage.duckdns.org:9038: SLGo? [11:18]  Justin Clark-Casey: james: doing my taxes, trying to fix bugs, etc. [11:18]  Robert.Adams @home.livingroomcam.us:14559: the OnLive-like thing [11:18]  Aine.Caoimhe @refugegrid.com:8002: http://www.firestormviewer.org/firestorm-on-sl-go/ [11:18]  Marcus.Llewellyn @grid.verymad.net:8002: The one that uses the same service as Sl Go. [11:18]  Robert.Adams @home.livingroomcam.us:14559: client is in the cloud and just video goes to the user's computer [11:18]  Marcus.Llewellyn @grid.verymad.net:8002: Taxes are a bug. ;) [11:18] Dahlia.Trimble @grid.wwweb3d.net:8002: people willing to pay for that? [11:19] Aine.Caoimhe @refugegrid.com:8002: I guess so [11:19]  Marcus.Llewellyn @grid.verymad.net:8002: Seems so. [11:19] Shez.Oyen @mirage.duckdns.org:9038: it is quite limited [11:19] Dahlia.Trimble @grid.wwweb3d.net:8002: interesting [11:19] Robert.Adams @home.livingroomcam.us:14559: you either pay for that or pay for a graphics card and connectivitiy [11:19] Justin Clark-Casey: I haven't heard of anybody using it much with opensim yet [11:19] Aine.Caoimhe @refugegrid.com:8002: nor have I [11:19]  Richardus.Raymaker @hypergrid.org:8002: yes, people paying for that [11:19] Dahlia.Trimble @grid.wwweb3d.net:8002: you need connectivity either way I'd think [11:19] Aine.Caoimhe @refugegrid.com:8002: I can see a niche for it though [11:20] Richardus.Raymaker @hypergrid.org:8002: because a pc you cannot drag to other homes [11:20] Marcus.Llewellyn @grid.verymad.net:8002: I've not tried that service. I was in the Playstation Now beta, where they streamed games. I was very suprised at the quality. Latency was barely detectable. [11:20] Richardus.Raymaker @hypergrid.org:8002: you cannot upgrade the graphios on ipad. i know users that run it on tablet when the are from home [11:20] Robert.Adams @home.livingroomcam.us:14559: does anyone build libraries on macs? I need to build BulletSim on Windows, Linux but don't have anywhere to build on Macs... what are people using? [11:20] Allen.Kerensky @icemoon.no-ip.org:8002: HTML5/webgl viewer is a better future path IMO [11:21] Richardus.Raymaker @hypergrid.org:8002: No [11:21]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: I wish all the viewer had devent webgl in them for MOAP, etc. [11:21]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: decent. [11:21] Marcus.Llewellyn @grid.verymad.net:8002: The last time I built OpenSim for mac I used a Hackinstosh in a VM. But that was ages ago. [11:21] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I have a beta singularity that rocks [11:21] Dahlia.Trimble @grid.wwweb3d.net:8002: I had my webgl viewer working in moap on recent linux firestorm but I couldnt get the controls to work [11:22] BlueWall.Slade @gateway.bluewallvirtual.net:8002: But, Windows not so much I think. [11:22] Robert.Adams @home.livingroomcam.us:14559: current problems include that I have to build BulletSim for a 32bit mac on the 64bit systems since Mono is only available in 32 bit :-( [11:22]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: maybe they use a different webkit lib on windows? [11:22]  Jak Daniels: I run OSX Snow Leopard VMware for building/testing things Robert [11:22]  Dahlia.Trimble @grid.wwweb3d.net:8002: Bluewall, seems likely [11:22]  Jak Daniels: *on VMware [11:22]  Justin Clark-Casey: robert: how complex is the build process? [11:22]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: Robert, have yuou looked at the mono-project page lately? [11:23]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: They're stepping up a little more with releases. [11:23]  Robert.Adams @home.livingroomcam.us:14559: a week or two ago... I was hoping things had changed... [11:23]  Robert.Adams @home.livingroomcam.us:14559: last I tried, it is not easy building 32bit on the 64bit systems [11:23] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I'm running 3.12 on a debian box atm [11:23] Justin Clark-Casey: yeah, cross-compiling is a pita [11:24] Robert.Adams @home.livingroomcam.us:14559: ;me wishes AWS would just make osx vm's available [11:24] Dahlia.Trimble @grid.wwweb3d.net:8002: I have 3.99 working on debian but it took some coaxing to get it to build [11:24] Marcus.Llewellyn @grid.verymad.net:8002: I was just about to ask if anyone tried 3.99 :) [11:24]  Marcus.Llewellyn @grid.verymad.net:8002: Which took coaxing? Compiling mono, or building opensim under 3.99? [11:25]  Dahlia.Trimble @grid.wwweb3d.net:8002: I had memory problems with anything 3.x but 3.99 seems to have fixed them [11:25]  Marcus.Llewellyn @grid.verymad.net:8002: The proboems you had with mono not freeing memory? [11:25]  Robert.Adams @home.livingroomcam.us:14559: someone making a Mono-For-OpenSim available for Macs would make some people happy [11:25]  James Stallings: Justin, I was wondering if we miht talk a bit about xassets [11:25]  Dahlia.Trimble @grid.wwweb3d.net:8002: Marcus, it would leak, dont know why [11:26]  Justin Clark-Casey: robert: If we can't get a mac build for bulletsim we may just have to drop support.... [11:26]  Justin Clark-Casey: james: sure [11:26] James Stallings: this past weekend we got into a bit of light testing of a restart of osgrid on our replicating mysql system [11:26] James Stallings: mel warned us off throwing the switch, saying we were pretty much oomed at around 80 to 100 gb [11:27]  James Stallings: experience pretty well supports that [11:27] James Stallings: (using blobs in the tables) [11:27] Justin Clark-Casey: oomed? out of memory? [11:27] James Stallings: she also suggested she had slutions in many xassets commits she's not made [11:27] Marcus.Llewellyn @grid.verymad.net:8002: I translated that as doomed. :) [11:27] James Stallings: doomed, sorry [11:27]  Arielle.Popstar @login.greatcanadiangrid.ca:8002: lol [11:27]  Dahlia.Trimble @grid.wwweb3d.net:8002: I thought you were using SRAS2? [11:27]  James Stallings: so in any case [11:27]  Justin Clark-Casey: hello nebadon [11:27]  Nebadon Izumi: hey [11:27]  Nebadon Izumi: sorry I am late [11:28]  Marcus.Llewellyn @grid.verymad.net:8002: Neb is late. 40 demterits. [11:28]  Nebadon Izumi: my internet is on the fritz [11:28]  James Stallings: we note that you have some changes you were going to make, ad were wondering about the timing of that, etc [11:28]  Justin Clark-Casey: james: I'm not sure what that would be. The only person who has written any major xassets code so far is me [11:28]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Dahlia, there is an SRAS 2? [11:28]  Justin Clark-Casey: apart from some of the pgsql stuff [11:28]  James Stallings: she lists a number of features sh has in reserve; filesystem blobs, dedup, others... [11:29] James Stallings: *she has [11:29] Marcus.Llewellyn @grid.verymad.net:8002: I promise I'm not trying to be snarky, but mel has a history of promising commits that take months or more to materialize. :/ [11:29] Justin Clark-Casey: james: I'm sure it's very helpful for you to hear that [11:29] James Stallings: yeah that is a concern [11:29] Dahlia.Trimble @grid.wwweb3d.net:8002: Andrew, thought there was, maybe my memory is going the way of mono 3.x [11:29]  James Stallings: it would be more helpful for her to commit those changes lol [11:30] Justin Clark-Casey: james: there are two major things I need to do. 1) Make sure I'm recording the data I want to with xassets, particularly access times. 2) to work out some way of completing a migration from an older set of asstes tables [11:30] Marcus.Llewellyn @grid.verymad.net:8002: The big question is whether they're true show stoppers. If not, then OSgrid is, by design, meant to upgrade to add improvements, fix things, etc. [11:30] Justin Clark-Casey: james: At the moment, it auto-migrates as you go along but there's no mechanism for finally taking everything out of old asset tables and putting it into xasset ones [11:30] James Stallings: nods [11:30] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Dahlia, I haven't looked at SRAS for a while. I'm not aware of the current version numbers on it. I'll have to take a closer look. Still trying to decide if I should go that way, or xassets, or something else to get blobs out of the db. [11:30] Justin Clark-Casey: james: so in principle it's usable... but I realy wanted the opportunity to do a final checkover without having to do a migration. Hence the massive experimetnal thing plastered on it [11:31] James Stallings: right [11:31] Justin Clark-Casey: and of course, it'#s never been used at scale so I don't know what problems might exist [11:31] Justin Clark-Casey: though I would be reasonably confident it would be fine [11:31] Nebadon Izumi: I think Melanie is right though, that its unwise for OSgrid to move forward with blobs in the DB [11:31]  Justin Clark-Casey: how so? [11:31] Seth.Nygard @refugegrid.com:8002: Last I looked at SRAS V2 it was being officially declared as abandonware by the developer [11:31] Nebadon Izumi: cause in 6 months we will probably have to bring the grid down again [11:32] Nebadon Izumi: in the past it didnt go well with a massive DB [11:32]  Justin Clark-Casey: in what way? [11:32] Dahlia.Trimble @grid.wwweb3d.net:8002: I remember it failing [11:32] Nebadon Izumi: it was failing constantly [11:32] Nebadon Izumi: around 100gb [11:32] Justin Clark-Casey: what kind of failure? [11:32] Nebadon Izumi: thats why we switched [11:32] Nebadon Izumi: the DB kept corrupting [11:32] Nebadon Izumi: we would have to rebuild the inde [11:32] Justin Clark-Casey: ok [11:32]  Nebadon Izumi: probably like 10 times we had to do that [11:32] Nebadon Izumi: thats when adam wrote new asset server [11:33] Nebadon Izumi: then eventually coyled rewrote it twice [11:33] Dahlia.Trimble @grid.wwweb3d.net:8002: then Adam was trying to use voldemort, then SRAS came about which finally worked [11:33] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, it kept corrupting even without the blobs in the db tables? [11:33] Nebadon Izumi: with xassets we can do blobs on the disk [11:33] Nebadon Izumi: according to melanie [11:33] James Stallings: I think that is functionality she hasnt comitted [11:33] Justin Clark-Casey: not really, xassets is all about the db [11:33]  Nebadon Izumi: using Sras isnt really realistic [11:33] Nebadon Izumi: since no one is maintaining it [11:33]  Nebadon Izumi: according to melanie [11:34] Nebadon Izumi: xassets can do blobs on disk [11:34] Justin Clark-Casey: well, tjhat's news to me [11:34]  Justin Clark-Casey: the code I wrote is certainly not set up to do that [11:34] Nebadon Izumi: that might be what she needs to push in [11:34]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: My blobs on disk experimental patch can be looked at to see how blobs can be stored on disk if that would be helpful for someone workingon xassets. [11:34] Nebadon Izumi: this is really the big holdup for OSgrid [11:34] James Stallings: ^^^ [11:34] Justin Clark-Casey: I don't support having a file system xassets system in core anyway, core isn't meant for hugely scalable service stuff [11:35] Nebadon Izumi: we cant really bring the grid backup until we figure out what to do [11:35]  Nebadon Izumi: we dont want to bring the grid up only to have it immediately fail again [11:35] Justin Clark-Casey: one major point of having it in the db was not to complicate things [11:35] Nebadon Izumi: and be down for another few months cause we have to convert a gigantic database [11:35] Dahlia.Trimble @grid.wwweb3d.net:8002: isnt meant for? [11:35] Marcus.Llewellyn @grid.verymad.net:8002: The conversion from Robust to SRAS took about 3 days, if I recall. [11:35] Dahlia.Trimble @grid.wwweb3d.net:8002: I wasnt aware that was the policy [11:35] James Stallings: if it can be done simply enough that it works, why not have it scale [11:35] Nebadon Izumi: what? [11:35] Nebadon Izumi: try 3 months [11:36] Nebadon Izumi: it took forever [11:36] Justin Clark-Casey: because it's not simple [11:36] Marcus.Llewellyn @grid.verymad.net:8002: We were not down for three months! Were we?? [11:36] Nebadon Izumi: no we kind of did it in the background [11:36] Dahlia.Trimble @grid.wwweb3d.net:8002: nothing in opensim is simple [11:36] Nebadon Izumi: but I dont want to do that again [11:36] Nebadon Izumi: it was a complete nightmare [11:36] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: 3 months to migrate data? Seems excessive. [11:36] Nebadon Izumi: took me literally watching the console for weeks [11:36] Nebadon Izumi: sleepless nights [11:37] Nebadon Izumi: i dont think anyone here actually knows how stressful that was [11:37] Justin Clark-Casey: perhaps osgrid shouldn't come back as an open grid, it's a big problem to run one of thoese anyway [11:37] Justin Clark-Casey: maybe a smaller set of hypergrid regions providing public facilities [11:37] Marcus.Llewellyn @grid.verymad.net:8002: Users are usually not told about that sort of thing. You guys like to keep it close the hip. [11:37] Nebadon Izumi: there would be no point in bringing it back at all then honestly [11:37] Justin Clark-Casey: why? [11:37] James Stallings: that's what osgrid is - an open grid [11:37] James Stallings: it always has been [11:37] Nebadon Izumi: for what 6 regions? [11:38] Nebadon Izumi: not worth it [11:38]  Justin Clark-Casey: I didn't say 6 regions. But nto something where you allow anybody to connect [11:38] Justin Clark-Casey: that model has always been fraught with problems [11:38] Nebadon Izumi: i guess id rather not do it at all myself [11:38] Justin Clark-Casey: like a huge asset db for instance [11:38] Nebadon Izumi: just run my own grid [11:38] James Stallings: we've always managed [11:38] Justin Clark-Casey: except you haven't! [11:39]  Dahlia.Trimble @grid.wwweb3d.net:8002: well had without a backup strategy [11:39] Nebadon Izumi: ok well I really don't know [11:39] James Stallings: we've always managed to the limits of the software [11:40] Robert.Adams @home.livingroomcam.us:14559: OSGrid is a place to connect ones own servers.... that's what makes it work [11:40] James Stallings: and thats pretty well where we are now [11:40] Dahlia.Trimble @grid.wwweb3d.net:8002: Robert was looking into CMS systems, maybe they are const cost competitive? [11:40] Justin Clark-Casey: and hence anybody can come along and access internal services and screw the entire thing [11:40] Nebadon Izumi: but thats never really happened but maybe once [11:40] Robert.Adams @home.livingroomcam.us:14559: there are a lot of good cloud services for storing blobs but they cost money [11:40] Nebadon Izumi: it sounds like your just not interested Justin [11:41] Arielle.Popstar @login.greatcanadiangrid.ca:8002: cant trash older assets to get it to a reasonable size?\ [11:41] Nebadon Izumi: if thats the case we will find someone who is [11:41]  James Stallings: maybe we just need to accept mel's work and move on [11:41]  Justin Clark-Casey: nebadon: interesting in what way? In trying to help you guys? [11:41] Nebadon Izumi: you dont sound interested in OSgrid at all [11:41] Justin Clark-Casey: I would love to but I'm trying to do a huge number of other things simultaneously [11:41] Justin Clark-Casey: I think the open grid approach has massive architectural problems. [11:41] Nebadon Izumi: ok we will see what Mel can do, she is willing to help [11:42] Justin Clark-Casey: ok [11:42]  Nebadon Izumi: she said she would push her current version of xassets in [11:42]  Dahlia.Trimble @grid.wwweb3d.net:8002: she probably has the most experience [11:42] Nebadon Izumi: I think that is really what we need [11:42] Nebadon Izumi: avinations asset services is nearly as large as OSgrid was [11:43] Marcus.Llewellyn @grid.verymad.net:8002: OSgrid isn't the only entity interested in xassets. [11:43] Justin Clark-Casey: you still shouldn't use it until it's not experimental [11:43] Billy.Bradshaw @hg.viewtwo.net:8600: Hypergrid has changed Opensim dynamics. Not perfect, maybe incomplete but offers alternative data distribution models [11:43] Nebadon Izumi: they are all experimental [11:43] Justin Clark-Casey: I have every intention of doing a sweep through where things may change without migrations [11:43] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I'd like to see some built-in(?) solution to asset blob storage. I've got a rather large DB and want to do something about it before it blows up (if that might happen). [11:43] Dahlia.Trimble @grid.wwweb3d.net:8002: osgrid has always been experimental [11:43] Nebadon Izumi: the core asset service just isnt good enough [11:44] BlueWall.Slade @gateway.bluewallvirtual.net:8002: MySQL advertises that they can do files as large as the OS can handle ... [11:44] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, what is the stated mission/purpose for osgrid? [11:44] BlueWall.Slade @gateway.bluewallvirtual.net:8002: So, what blows up with it? [11:44] Nebadon Izumi: we could use SRAS again, but no one maintains it [11:44]  Nebadon Izumi: if something changes osgrid would be screwed again [11:44] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: BlueWall, if that is the case, why do people keep saying that the DB can go bad at certain sizes, or under certain combination of conditions? [11:45] Dahlia.Trimble @grid.wwweb3d.net:8002: would xassets be maintained? Melanie doesn't seem to have all that much time lately [11:45] Nebadon Izumi: if its in core anyone really can [11:45] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Andrew, that is what I'm saying. [11:45] Arielle.Popstar @login.greatcanadiangrid.ca:8002: but how much of that assets will never be called up? Assets from people who were around years ago but now gone [11:45] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Dahlia, not as likely to be maintained if it is a separate module outside core [11:45] Dahlia.Trimble @grid.wwweb3d.net:8002: anyone can if they have a need to [11:45]  Nebadon Izumi: the blob on disk is optional thing [11:45] Nebadon Izumi: its not mandatory [11:45] Nebadon Izumi: i dont see why that should be excluded from core [11:45] Richardus.Raymaker @hypergrid.org:8002: but make things maby more easy [11:46] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Haing a way to dedupe would be very useful. [11:46] Richardus.Raymaker @hypergrid.org:8002: if blobs are on disk [11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: lots of stuff in core is not maintained either [11:46] Justin Clark-Casey: like what? [11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: it eventually gets trimmed out [11:46] Nebadon Izumi: ya thats another reason core asset server is not a good fit right now [11:46] Nebadon Izumi: no dedupe [11:46] Nebadon Izumi: we really want to bring the grid back up [11:47]  Nebadon Izumi: but this is whats holding us back [11:47] Nebadon Izumi: techincally the asset servers are ready to go [11:47]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Assets table also gets filled with lots of "one of" things, AFAIK. Editing a script or NC can create a new asset instead of updating the existing one, IIRC [11:47] Dahlia.Trimble @grid.wwweb3d.net:8002: is there reason to believe SRAS is incapabie? [11:48] Nebadon Izumi: its capable, but again no one really knows the code [11:48] Nebadon Izumi: coyled isnt really interested anymore [11:48] Nebadon Izumi: i don't know ruby [11:48] Nebadon Izumi: i can barely even get it setup without his help [11:48] Allen.Kerensky @icemoon.no-ip.org:8002: not even ruby knows ruby - its a language without a spec [11:48] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I did a proof of concept of blobs on disk. The changes to core were minimal. [11:49] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Could that not be the basis of some changes to asset storage in core? [11:49] Richardus.Raymaker @hypergrid.org:8002: yes andrew, thats one of the bad things with scripts [11:49] Allen.Kerensky @icemoon.no-ip.org:8002: if one of OSgrid purposes is to provide an open testbed for OpenSim - using Core solutions to problems is better testing for a wider audience [11:49] Dahlia.Trimble @grid.wwweb3d.net:8002: using blobs on disk also assumes you format the disk correctly (number of inodes, etc) [11:49] Nebadon Izumi: ext4 shouldnt really have any problems [11:50] Allen.Kerensky @icemoon.no-ip.org:8002: when you stop using core, you stop helping test some of opensim and invalidate the results for a larger segment of the users [11:50] Allen.Kerensky @icemoon.no-ip.org:8002: ext4 or xfs yeah - scale away [11:50] Dahlia.Trimble @grid.wwweb3d.net:8002: ext4 still has internal tables that need to be configured for massive # of files [11:50] Dahlia.Trimble @grid.wwweb3d.net:8002: defaults will probably fail [11:50] Justin Clark-Casey: If I were to run a grid I would not use the core service. I would write something with a better scaling approach, like maybe something using cassandra [11:51] Nebadon Izumi: i agree that core isnt suitable currently [11:51] Robert.Adams @home.livingroomcam.us:14559: scaling always needs config changes [11:51] Nebadon Izumi: but we cant wait months again either really [11:51] James Stallings: if it were in core we would test it to the breaking point [11:51] Seth.Nygard @refugegrid.com:8002: although not part of any standard linux distro, I have had good luck with ZFS on linux for very large datastores. It is also capable to doing snapshots which can be leveraged for backups. [11:51] Dahlia.Trimble @grid.wwweb3d.net:8002: ya a nosql replicating approach would be a better choice I'd think [11:51] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: James, we do have a way of breaking things. :) [11:51] Dahlia.Trimble @grid.wwweb3d.net:8002: like cassandra [11:52]  Allen.Kerensky @icemoon.no-ip.org:8002: asset clusters with shared storage etc - cost prohibitive however [11:52]  Dahlia.Trimble @grid.wwweb3d.net:8002: I had tried riak a couple years ago but didnt get far [11:52]  Nebadon Izumi: if we have to wait months to get osgrid back up while something gets developed and tested, i just don't see it coming back [11:52]  Robert.Adams @home.livingroomcam.us:14559: someone needs to make an S3 based asset store :) [11:52] Justin Clark-Casey: You wouldn't expect to run a huge website on Apache alone. There have to be a number of components in play [11:52] Nebadon Izumi: we have limited funds that are quickly being depleted paying for servers that are not being used [11:52] Robert.Adams @home.livingroomcam.us:14559: put it in core and get OSGrid up! [11:52] Marcus.Llewellyn @grid.verymad.net:8002: Has using the MOSES/Simian asset service been explored? [11:52] James Stallings: you have to remember, osgrid uses a few servers. The asset servers especially are not cheap. People wont keep paying for them to sit idly for long. [11:53] Nebadon Izumi: nah no freaking way [11:53] Justin Clark-Casey: which is why I don't think an open grid model works. But clearly you guys don't agree [11:53] Justin Clark-Casey: so I will shut up about that [11:53] Nebadon Izumi: MOSES uses simian grid [11:53] Nebadon Izumi: and its horrible [11:53] James Stallings: they've already sat idly for months. [11:53] Robert.Adams @home.livingroomcam.us:14559: just do it.... OpenSim will always be one the boundry of 'clean core' and 'full of fun stuff' [11:53] Nebadon Izumi: who would decide who can and can't be on OSgrid [11:54] Dahlia.Trimble @grid.wwweb3d.net:8002: I asked cmickyb about using simian assets with a robust grid, he thought it would work but there are some hypergrid issues that need to be addressed first [11:54] Arielle.Popstar @login.greatcanadiangrid.ca:8002: why so adamant on restoring osgrid to its former state  vs starting empty? [11:54] Nebadon Izumi: i dont think you realize what your proposing justin [11:54] Justin Clark-Casey: and hwo about the endless bug rpoerts of 'fun stuff' that was never written properly and was long abandoned? [11:54] Justin Clark-Casey: nebadon: how do I not? [11:54] Marcus.Llewellyn @grid.verymad.net:8002: I guess it's a balance. I've always known OSgrid could crash and burn. I'm suprised it didn't earlier. But I completely appreciate the admins wanting to future-proof things too. [11:55] Robert.Adams @home.livingroomcam.us:14559: justin, those bug reports will do what that type of bug report does now.... waits for a volunteer who cares [11:55] Robert.Adams @home.livingroomcam.us:14559: important things get fixed [11:55] Justin Clark-Casey: which is never in many cases [11:55] Dahlia.Trimble @grid.wwweb3d.net:8002: I wish you would have brought these issues up a few months ago when I had some time to look at them [11:55] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I was only surprised there weren't other backups of the osgrid dbs [11:55] Justin Clark-Casey: and when you have a project full of bugs it's low quality and people abandon it [11:55]  Robert.Adams @home.livingroomcam.us:14559: an strange interaction with a particular terrain painting tool will not get fixed right away [11:56] Nebadon Izumi: well honestly I havnt had the time either, but now we have people willing to do the work [11:56] Nebadon Izumi: but we want whatever we decide to do to benefit opensim [11:56] Nebadon Izumi: going with SRAS again will not [11:56] Nebadon Izumi: and will likely lead to repeat of this whole thing [11:56] Justin Clark-Casey: getting a bunchj of code inside core that people may well not maintain will not help either [11:56] Robert.Adams @home.livingroomcam.us:14559: 'full of bugs' is not binary.... it's a gray area.... if the important bugs are fixed, people will use it [11:57] Marcus.Llewellyn @grid.verymad.net:8002: Is Melanie's xasset code in production use, maybe at Avination? [11:57] Nebadon Izumi: not getting in in core doesnt insure it gets maintained either [11:57] Nebadon Izumi: lol [11:57] James Stallings: yes [11:57] Nebadon Izumi: so who cares [11:57] Justin Clark-Casey: of course it does, you think I do nothing all the time [11:57] Robert.Adams @home.livingroomcam.us:14559: and I would agree that everything shouldn't be thrown into core [11:57] Justin Clark-Casey: I spend practically all my time trying to maintain things [11:57] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: How is xassets different from core assets in terms of what it does? [11:57] BlueWall.Slade @gateway.bluewallvirtual.net:8002: dedup [11:57] James Stallings: blobs on disk [11:57] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: that's it? [11:57] James Stallings: those are the two biggies [11:57] BlueWall.Slade @gateway.bluewallvirtual.net:8002: yes [11:58] Justin Clark-Casey: xassets does not do blobs on disk as I wrote [11:58] Marcus.Llewellyn @grid.verymad.net:8002: I very much doubt anyone thinks that, JCC. Anyone who sees the channel bots on IRC knows you're doing a great deal. [11:58] Nebadon Izumi: its basicall a better version of whats in core now [11:58] Justin Clark-Casey: if melanie has private changes which do that then that's something else [11:58] Dahlia.Trimble @grid.wwweb3d.net:8002: I dont see a problem with having highly scalable services in core but I agree that they probably wont be maintained as most of the people who use them will probably have closed large grids and wont share [11:58] Nebadon Izumi: I don't really see how that is bad [11:58] BlueWall.Slade @gateway.bluewallvirtual.net:8002: xassets in core stores blobs in database [11:58] James Stallings: I dont think you do nothing Justin, really; I just know you arent superhuman [11:58] Robert.Adams @home.livingroomcam.us:14559: we're talking around a gray area.... can't put everyone's abandable pet project in core.... but can't exclude everything that is not military hardened [11:58] Nebadon Izumi: let me see if i can get melanie [11:58] Justin Clark-Casey: I have to go and do other things now [11:59] Justin Clark-Casey: it's the top of the hour [11:59] Dahlia.Trimble @grid.wwweb3d.net:8002: bb jcc [11:59] James Stallings: we should distinguish between core xassets and avination xassets in this discussion I think [11:59] Marcus.Llewellyn @grid.verymad.net:8002: Take care, Justin. [11:59] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++ [11:59] Richardus.Raymaker @hypergrid.org:8002: bye justin [11:59] James Stallings: thanks Justin [11:59] BlueWall.Slade @gateway.bluewallvirtual.net:8002: See ya Justin, take care [11:59] Nebadon Izumi: ok melanie responded [11:59] Aine.Caoimhe @refugegrid.com:8002: bye Justin [11:59] Seth.Nygard @refugegrid.com:8002: Have a great night Justin, take care and thanks for the help [11:59] Nebadon Izumi: oh your leaving? [11:59] Nebadon Izumi: sigh [11:59] Justin Clark-Casey: I'm sure there will be more discussion on this but I think something majro should be on the mailing list and not just put into core as is [12:00]  James Stallings: what did Mel respond? [12:00] Nebadon Izumi: i was going to have her come here [12:00] James Stallings: ah [12:00]  Nebadon Izumi: but if justin is leaving nevermind [12:00] Justin Clark-Casey: alright, let me stick around for 15 mins [12:00] BlueWall.Slade @gateway.bluewallvirtual.net:8002: it would behave like coupling the asset cache to the backend? [12:00] James Stallings: /|\ thanks Justin [12:01] Nebadon Izumi: nice [12:01] Nebadon Izumi: hello Melanie [12:01] Melanie Milland: hi [12:01]  James Stallings: o/ [12:01]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Hey, Melanie [12:01] BlueWall.Slade @gateway.bluewallvirtual.net:8002: hi Melanie [12:01] Arielle.Popstar @login.greatcanadiangrid.ca:8002: Hi Mel [12:02] Nebadon Izumi: ok so, we are trying to get OSgrid resolved [12:02] Nebadon Izumi: and we are wondering about your version os xassets [12:02] Nebadon Izumi: some are not sure it is suitable for core inclusion [12:02] Nebadon Izumi: hoping you could maybe fill us in on what its all about [12:02] Nebadon Izumi: whats different than what is already in core [12:03] Melanie Milland: i don't know why anyone whould think it's not suitable for core without having seen it [12:03]  Justin Clark-Casey: you have added code to the core xasstes for storing blobs on disk rather than in the db? [12:04] Melanie Milland: i thought xassets in core was already blobs. if not, yes, ours is [12:04] Melanie Milland: i mean, on-disk blobs [12:04] Melanie Milland: we also gzip them where it saves space [12:04] Dahlia.Trimble @grid.wwweb3d.net:8002: multiple file systems? replicatable? [12:04] Justin Clark-Casey: xassets in core stores blobs in db. One reason was to not create a more complicated backup situations wher eboth file system stuff and db has to be backed up [12:04] Justin Clark-Casey: to make core xassets usable out of the box without making things more complicated for the user [12:04] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: The gzip/gunzip process doesn't add excessive overhead to handing out the blobs to viewers? [12:05] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Melanie, is it sort-of like placing the asset cache in the backend? [12:05] BlueWall.Slade @gateway.bluewallvirtual.net:8002: similar? [12:05] Melanie Milland: we find that gunzip is very lightweight and gzip isn't too frequent to be an issue [12:05] Dahlia.Trimble @grid.wwweb3d.net:8002: large assets like meshes and textures are already compressed [12:05] Melanie Milland: correct - as i said, it compresses only where there are saving afair [12:06] Melanie Milland: so that hits mainly notecards and scripts [12:06] Dahlia.Trimble @grid.wwweb3d.net:8002: and linksets [12:06] Melanie Milland: indeed [12:06] BlueWall.Slade @gateway.bluewallvirtual.net:8002: We already have the file-based asset cache in OpenSim, so I guess it wouldn't be a huge stretch [12:06] Melanie Milland: justin, what would be the reason to not include a disk based asset service in core? we're all about choices, or used to be anyway [12:07] Justin Clark-Casey: melanie: one concern is greater complexity for moderate users. If they use the option, the backup approach becomes more complex [12:07] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: There was some concern it would get put in core and then be unmaintained [12:07] Justin Clark-Casey: you can't just say "backup the database" any more [12:07] Nebadon Izumi: ya but its optional and not by default [12:07] BlueWall.Slade @gateway.bluewallvirtual.net:8002: that is on the user Justin [12:07] Nebadon Izumi: why would that really matter [12:08] Justin Clark-Casey: if something is optional it does get used and should be supported [12:08] BlueWall.Slade @gateway.bluewallvirtual.net:8002: many are in over their head already [12:08] James Stallings: lol [12:08] Shez.Oyen @mirage.duckdns.org:9038: :o [12:08] James Stallings: arent we all heh [12:08] Dahlia.Trimble @grid.wwweb3d.net:8002: I like the current core asset server for my small grid, it's very easy to back up. I wouldnt want to use it for anything much larger tho [12:08] Nebadon Izumi: why should we dictate how people do their backups? [12:08] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I would +1 inclusion of it [12:08]  Justin Clark-Casey: melanie: actually, just to clear things up, if the xassets service you are talking about one that builds on the code in core or is it entirely separate? [12:08] Robert.Adams @home.livingroomcam.us:14559: just like turning on HG... there is a lot of stuff a user has to know and account for..... turning on blob storage would be another of those [12:09] Marcus.Llewellyn @grid.verymad.net:8002: Would this replace current robust assets? Or just be another option? If the former, people wanting a simpler setup can choose the old asset server. [12:09] BlueWall.Slade @gateway.bluewallvirtual.net:8002: That is as long as the current default remains the same. [12:09] Marcus.Llewellyn @grid.verymad.net:8002: Novices will usually use SQLite anyway. [12:09] Nebadon Izumi: its not like we teach people how to do database backups in the first place [12:09] James Stallings: I wouldn't want it on by default [12:09] Nebadon Izumi: something they need to learn on their own [12:09] Melanie Milland: justin: ours is called FSAssets and descends from core assets, but it's seen a lot of revisions since then so may be significantly different in places [12:09] BlueWall.Slade @gateway.bluewallvirtual.net:8002: We should keep assets + xassets [12:09] James Stallings: I just want to be able to use it [12:09]  Justin Clark-Casey: ok, so we are talking about two differnet things [12:09] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I've got a 400+G asset table and wondering if I need to migrate it to something else than stock core asset system. [12:10] Justin Clark-Casey: melanie: if you put fsassets into core would you be willing to spend time to maintain it? [12:10] Nebadon Izumi: I am surrpised you havent ran into trouble Andrew [12:10] Nebadon Izumi: how many users? [12:10] James Stallings: yeah that kinda had me wondering too [12:10] Nebadon Izumi: and how do you back that up? [12:10] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++ - at least until is sees some use and bugs are reported. [12:11] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, not sure of the numbers off hand. May be couple thousand but in the hundreds, re: active users. [12:11] Melanie Milland: justin: we maintain it anyway, it's what runs avination [12:11] Dahlia.Trimble @grid.wwweb3d.net:8002: can it be easily backed up and/or replicated? if not, all this is moot anyway [12:11] Justin Clark-Casey: melanle: does it have an optino to store in db or is it filesystem only? [12:11] Melanie Milland: whether or not we copy that to core is a nonissue in terms of work [12:11] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Dahlia, it would need to have both DB bacup and file system backup. [12:11] Melanie Milland: as long as no one goes and does refactoring on it to make it diverge too much [12:12] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: It might complicate backing up both in terms of making sure they are both in sync. [12:12] Justin Clark-Casey: of course, any code that goes into core can be touched by any core developer [12:12] Melanie Milland: it stores assets in the filesystem, always. blobs are BAD for mysql so they're not supported - what would be the point? [12:12] Marcus.Llewellyn @grid.verymad.net:8002: Does Avination's code contain a migration tool/strategy for going from current assets to the new one? [12:12] Justin Clark-Casey: I've never read anything that definitely says blobs are terrible in mysql [12:13] Allen.Kerensky @icemoon.no-ip.org:8002: percona, maria 10, and mysql 5.6 are supposed to handle blobs betterer [12:13] Dahlia.Trimble @grid.wwweb3d.net:8002: Melanie, how about backups and/or replication? [12:13] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I think that's a religious argument [12:13] BlueWall.Slade @gateway.bluewallvirtual.net:8002: hehe [12:13] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Justin, neither have I but I keep hearing people say its bad. [12:13] Melanie Milland: marcus: we made a copy-on-read connector at some point but i couldn't say where it currently is [12:13]  James Stallings: BlueWall: we just dodgin what done bit us before [12:13] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++ [12:14] Melanie Milland: justin: bug fixes and enhancements are one thing, wholesale refactoring is another. after all, this is tried and tested code, it should not need anyone's hands on it [12:14] BlueWall.Slade @gateway.bluewallvirtual.net:8002: but I have heard both arguments about it. [12:14] James Stallings: I hadnt really heard either, just workin off direct experience ;) [12:15]  Justin Clark-Casey: melanie: one can't impose conditions on code that comes into core. If changes are necessary then anybody can make them. [12:15]  Melanie Milland: dahlia: fsassets doesn't handle either. there are tools out there that can do that job much better. avination is fully replicated using mysql replication and drbd [12:15]  Justin Clark-Casey: a service can always be provided outside of core - by going into core it becomes something else [12:15]  James Stallings: lets ot be tedious; that's pretty fundamental to the nature of the project [12:15]  Nebadon Izumi: past experience for OSgrid blobs in the database was not good, but I can't really say ive seen that in writing anyway, only my own very horrible experiences [12:15]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002 just had the thought that gunzip on assets on disk isn't such a big deal as the regions would cache some of the assets. Biggest hit is always on startup with empty caches. [12:15] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Conditions can't be imposed - but I really doubt that it would see huge changes [12:15] Melanie Milland: justin: no one is talking about conditions [12:16] BlueWall.Slade @gateway.bluewallvirtual.net:8002: many eyes on it may find some improvements though. [12:16] BlueWall.Slade @gateway.bluewallvirtual.net:8002: those are usually small [12:16] Billy.Bradshaw @hg.viewtwo.net:8600: Would have been advantageous for this issue to have been raised on the dev mailing list, at least people would be better prepared. [12:16] Nebadon Izumi: it wont be a forced change [12:16] Nebadon Izumi: there is no real reason to prepare anyone for anything [12:16] Melanie Milland: justin: but if someone refactors stuff just because they don't like names or ordr of methods in a file, then that would be something i'd dislike [12:16] Melanie Milland: because it makes work for me [12:16]  Marcus.Llewellyn @grid.verymad.net:8002: Presumably, Avination would welcome fixes or improvements, so long as they were reasonably easy to merge. [12:16] James Stallings: we just need the codez :3 [12:17] Justin Clark-Casey: those things can always change. But as BlueWall says, if it already makes sense there's no reason to touch it. [12:17] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Melanie, is it compatible with the code in core now - names, format, etc.? [12:17] Robert.Adams @home.livingroomcam.us:14559: JCC, Mel is saying smaller changes allows cross-merging and is a Good Thing.... major re-arrangement of code makes cross-merging nearly impossible [12:17] Dahlia.Trimble @grid.wwweb3d.net:8002: naming or code ordering is subjective anyway, there is no "right or wrong" [12:18] Justin Clark-Casey: if something is put into opensim then it can get changed. OpenSim should not be held static by proprietary code [12:18] Justin Clark-Casey: if someone wants proprietary code then that is fine, but it cannot be allowed to hold back change in an open-source project [12:18] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I think it would be a plus [12:19] Marcus.Llewellyn @grid.verymad.net:8002: I don't think anyone suggested that the contribution would be propietary. I only saw a request for consideration in merging changes. [12:19] BlueWall.Slade @gateway.bluewallvirtual.net:8002: we should keep current assets + xassets and add this one as whatever Melanie want's to anme it. [12:19] Robert.Adams @home.livingroomcam.us:14559: not suggesting static... why take things to extreme... just know that if major refactoring is done then changes to and from avination won't be able to happen [12:19] Melanie Milland: justin, you're flogging the shadow of the stain that used to be in the place where a dead horse used to lie [12:19] Robert.Adams @home.livingroomcam.us:14559: it's not the end of the world... but it is not optimal [12:19] BlueWall.Slade @gateway.bluewallvirtual.net:8002: lol, Melanie [12:19] Melanie Milland: the point i'm making is that changes only designed to make work for me would not be welcome [12:19] Melanie Milland: note, ONLY DESIGNED [12:20] Justin Clark-Casey: Changes are not made just to make work for you. [12:20] Melanie Milland: functional enhancements, fixes, etc are always welcome in core [12:20] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Melanie, if you bring it in, will you make it a 3rd option? [12:20] BlueWall.Slade @gateway.bluewallvirtual.net:8002: or replace xassets [12:21] Melanie Milland: yes. i see no point in removing any of the existing services [12:21] Marcus.Llewellyn @grid.verymad.net:8002: Good. [12:21] Melanie Milland: they do appear to be in use [12:21] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I like having a deduping option w/o writing blobs to disk too. [12:21] Marcus.Llewellyn @grid.verymad.net:8002: My grid is young and tiny. I'd like to have dedup sooner rather than later. [12:21] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I think xassets, not so much. But it would be useful when stable. [12:21] Nebadon Izumi: do we currently have that now BlueWall? [12:21] Nebadon Izumi: I didnt think we did [12:22] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Nebadon, we do. [12:22] Melanie Milland: avn started without dedup. took months to process the data later [12:22] BlueWall.Slade @gateway.bluewallvirtual.net:8002: xassets dedupe, but keep blobs in the database. [12:22] Nebadon Izumi: ok cool i didnt think dedupe was working [12:22] Nebadon Izumi: thats good [12:22] Nebadon Izumi: not sure why I thought that, i thought it was incomplete [12:22] BlueWall.Slade @gateway.bluewallvirtual.net:8002: It is not done - per warnings [12:22] Nebadon Izumi: ah ok [12:22]  Justin Clark-Casey: nebadon: It is incomplete, because there are two things I want to do to it [12:22]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: could change out from under you [12:22] Nebadon Izumi: I didnt realize it was functional at all [12:22] Justin Clark-Casey: as I said earlier [12:22] Nebadon Izumi: yea [12:23] Justin Clark-Casey: it is completely functional [12:23] Nebadon Izumi: i guess i misunderstood [12:23] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I have access to an older 100G or so DB and found it had around 40% duplication of asset data. [12:23] James Stallings: the mig tool shouldnt be THAT heinous [12:23] Melanie Milland: justin, what are you planning? [12:23] James Stallings: if I had good before and after schemata I could do that easily [12:23] Nebadon Izumi: actually we have one already [12:23] Nebadon Izumi: probably could be modified slightly [12:23] Nebadon Izumi: andrew wrote it [12:23]  Nebadon Izumi: its php-cli [12:23] Nebadon Izumi: worked very well [12:23] James Stallings: nice [12:23] James Stallings: that woould be my precise approach [12:23] Justin Clark-Casey: melanie: I want to make sure the core xassets isn't missing any fields it might be good to have. Then I wanted to work out a completely migratio nstrategy from existing assets [12:23] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Yeah, I was about to mention my migration script. It can be modified as needed. [12:24] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++ on the migration part. [12:24] James Stallings: the tl;dr is a mig tool in this crowd is not problematic [12:24] BlueWall.Slade @gateway.bluewallvirtual.net:8002: that could probably work on both the xassets and melanies [12:24] Nebadon Izumi: i have it packaged in my sras kit [12:24] Nebadon Izumi: if you want it [12:25]  Justin Clark-Casey: james: alrigjht. So let's say I can try to complete what I wanted to do on core xassets [12:25] James Stallings: we may need it [12:25]  Nebadon Izumi: http://onikenkon.com/sras/ [12:25] Nebadon Izumi: its in the zip [12:25] Justin Clark-Casey: james: which is relatively simple. The step of allowing complete migration can be added later. In fact, it doesn't matter if you are starting afresh [12:25] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I was also thinking that the migration script can be run on a live system to migrate the assets up to a certain date and then a grid can be taken down to migrate the assets after that date to minimize grid down times during migration [12:25] James Stallings: nods [12:25] Justin Clark-Casey: james: so then you have the option of either using the existing core assets or melanie's new fsassets when she addes it [12:25]  Justin Clark-Casey: core xassets [12:26] James Stallings: they'd both be core at that point, but yeah... [12:26] Nebadon Izumi: on of the reasons we are keen on using melanies [12:26] Justin Clark-Casey: I'm not trying to undercut melanie, I just feel bad now [12:26] Nebadon Izumi: is because there is a small chance we might still be able to recover sras assets [12:26] Nebadon Izumi: and push them into xassets [12:26] Melanie Milland: why do you feel bad? [12:26] Justin Clark-Casey: but yeah, if you want assets on fs you will want melanie's fsasset [12:26] James Stallings: you shouldn't [12:26]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Are we now talking about having 3 asset storage systems? core, xassets, and fsassets? [12:26] Justin Clark-Casey: because this is the thing stopping osgrid from coming back up [12:26]  James Stallings: you arent superman bub [12:26] BlueWall.Slade @gateway.bluewallvirtual.net:8002: With both in core, we would have a small, medium and large solution [12:26] Justin Clark-Casey: some kind of sustainable asset strategy [12:26] James Stallings: and you did not know that [12:26] Justin Clark-Casey: although I think an open tgird is fundamentaly flawed in this respect [12:26] James Stallings: hell, we didnt know your code was diff from mels [12:26] Nebadon Izumi: ya beleive me justin [12:27] Nebadon Izumi: my goal here is to minimize work load [12:27] Nebadon Izumi: and not repeat this whole scenario 6 months from now [12:27] James Stallings: mine is to get our admittedly suicidal grid back up [12:27]  Justin Clark-Casey: neb: ok, let's be clear. core xassets is what I wrote and it is in core but experimental. Melanie's fsaset s is a separate service which stores blobs on fs instead of db, as far as I'm hearing [12:27] Nebadon Izumi: sure [12:27] Nebadon Izumi: that sounds great to me [12:27]  James Stallings: I think we've been working miracles with it for years and I'm not finished yet [12:27] James Stallings: we're not finished yet. [12:27] Nebadon Izumi: and I am sorry if I didnt explain things as good as possible [12:28] Nebadon Izumi: I also am juggling many projects [12:28] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Long term, it would be good to have migration for straight assets ---> xassets > fassets [12:28] Melanie Milland: well, as i outlined in chat the other day, we have tried and proven replication strategies and also growths paths for fsassets [12:28] Nebadon Izumi: currently in sweden, so im a bit scrambled [12:28] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Aren't many of us jugglingtoo many projects? [12:28] Justin Clark-Casey: so perhaps you guys could go away and decide if you want to use xassets or fsassets. Then let one of us know? [12:28] Justin Clark-Casey: unless Melanie was planning on putting fsassets in core anyway [12:28] Justin Clark-Casey: imminently [12:28] Marcus.Llewellyn @grid.verymad.net:8002: I'm still confused on one point. Does Mel's service do *only* the filesystem, or can it do both fs and db? [12:28] Nebadon Izumi: personally id prefer to start with fsassets [12:28] Nebadon Izumi: if melanie is game [12:29] James Stallings: lol I would hope you both contribute your work [12:29] Nebadon Izumi: but honestly im not doing the work [12:29] Nebadon Izumi: Allen and James? [12:29] Nebadon Izumi: what say thee [12:29] Melanie Milland: marcus: fsassets does a db that is used as an index to the blobs on the fs, which are deduped [12:29] James Stallings: which is the quicker path to grid upness? [12:29] Melanie Milland: it doesn't ever store assets as blobs [12:29] Marcus.Llewellyn @grid.verymad.net:8002: Melanie: So id does not and will not store asset blobs in the DB. [12:29] Nebadon Izumi: its basically SRAS [12:29] Nebadon Izumi: but in C# [12:30]  Melanie Milland: marcus: correct. use xassets if that is what you want [12:30] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++ [12:30] Justin Clark-Casey: marcus: expect not yet [12:30] Marcus.Llewellyn @grid.verymad.net:8002: Thanks. All clear now. :) [12:30] Dahlia.Trimble @grid.wwweb3d.net:8002: OSGrid has always been somewhat experimental, and that's been very valuable to OpenSimulator. I say go ahead with Melanie's service if she's game [12:30]  James Stallings: It would help to know what kind of timeframe Mel can afford in terms of pushing FSAssets [12:30]  Nebadon Izumi: Melanie what kind of turn around [12:30]  James Stallings: "no pressure" LOL [12:30]  Nebadon Izumi: hate to pressure you, but ball is kind of in your court now [12:30]  Nebadon Izumi: lol [12:30]  Melanie Milland: i can likely manage it this week [12:30]  Marcus.Llewellyn @grid.verymad.net:8002: Yay! (insert happy dance) [12:31]  Melanie Milland: if neb pokes me a couple of times - mind like a sieve ;) [12:31] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++ [12:31] Nebadon Izumi: if there is one thing I am expert at its poking [12:31] Melanie Milland: k [12:31]  James Stallings: Justin, I would still love to see your polish to Xassets [12:31] Dahlia.Trimble @grid.wwweb3d.net:8002: how about restoring the recovered data? can that work with fsassets? [12:31] James Stallings: I would like for us to come out with a mig tool that can mig data all around [12:31] Marcus.Llewellyn @grid.verymad.net:8002: I think that's the whole point, Dahlia. [12:31] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: That's why I use Tomboy for keeping an electronic ToDo list. [12:31] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Justin, the xassets is a really good mid-level soulution [12:31] Nebadon Izumi: if the data is good, possibly dahlia [12:31] Melanie Milland: so i guess that is settled [12:31] Nebadon Izumi: but it kind of gets tricky [12:31] Melanie Milland: Welcome to the Dark Side. Are you surprised we lied about the cookies? [12:31] Marcus.Llewellyn @grid.verymad.net:8002: Well, not the whole point. [12:31] Nebadon Izumi: we may need help with that [12:32] Dahlia.Trimble @grid.wwweb3d.net:8002: does an importing procedure exist? [12:32] Allen.Kerensky @icemoon.no-ip.org:8002: well there you go - I am game for whatever folks want - and yes, Xassets is what I'd like for my private grid - just one db to wrangle and not a lot of files I already have millions [12:32] James Stallings: I thought theCAKE was a lie. No one said there would be cookies. [12:32] Nebadon Izumi: importing assets is easy [12:32] Melanie Milland: if i get access to the data i can tell you [12:32] Nebadon Izumi: merging in old inventory [12:32] Nebadon Izumi: not so [12:32]  Nebadon Izumi: im sure it can be done [12:32] Marcus.Llewellyn @grid.verymad.net:8002: What I want most is dedup. DB or FS isn't as relevant to me. [12:32] Nebadon Izumi: but its not as simple as the assets [12:32] Dahlia.Trimble @grid.wwweb3d.net:8002: easy, assuming the procedure exists [12:32] Melanie Milland: in avn, we have transparently and without downtime handled several catastrophic hardware failures [12:32] Melanie Milland: i'm pretty good with mangling big data by now [12:32] James Stallings: +100 [12:33] Allen.Kerensky @icemoon.no-ip.org:8002: does FSassets also support HGAssetService? [12:33] Nebadon Izumi: and Melanie you said you might be able to lay hands on osgrid hardware? [12:33] Nebadon Izumi: ]\\ [12:33] James Stallings: excellent question Allen [12:33] Melanie Milland: of course. it supports the asset protocol, HG builds on it [12:33] James Stallings: :) [12:33]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: plug&pray [12:34]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: :) [12:34] Arielle.Popstar @login.greatcanadiangrid.ca:8002: lol [12:34] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: How assets are saved shouldn't have any real impact on anything else. Assets are requeste and passed around as they always have been. Just the source of them has changed. [12:34] Melanie Milland: well, i have a soft spot for osgrid [12:34] Melanie Milland: so i'll be there when it counts [12:34] James Stallings: dont we all :) [12:34]  Nebadon Izumi: great, thanks Melanie [12:34]  Melanie Milland: after all, i'm past my health issues somewhat [12:34]  Nebadon Izumi: that means a lot to me personally [12:34]  James Stallings: Thanks Mel [12:34]  Nebadon Izumi: I wish I had more time myself [12:34]  Allen.Kerensky @icemoon.no-ip.org:8002: yeah everyone has been really great during this mess - it is much appreciated [12:34]  Shez.Oyen @mirage.duckdns.org:9038: I'm greatful Melanie [12:34]  James Stallings: especially the users [12:34]  Nebadon Izumi: I really appreciate what James and Allen and Dan have been doing [12:34]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Melanie, Good to hear you are (mostly) better [12:34]  Nebadon Izumi: in my absence [12:35]  Allen.Kerensky @icemoon.no-ip.org:8002: I've bothered Diva and Justin with questions too lol everyone has been supportive [12:35] Justin Clark-Casey: I honestly tihnk you guys should serious think about the nature of the grid though [12:35] Nebadon Izumi: we are [12:35] Nebadon Izumi: things are not going to be the same [12:35] Nebadon Izumi: its a whole new ball game [12:35] James Stallings: we want the cieling to be high when we crash into it for good [12:35] Allen.Kerensky @icemoon.no-ip.org:8002: that's what the "restructuring" headline in the updates is sort of about [12:35] Justin Clark-Casey: the architecture of an open grid is really difficult - sl just wasn't designed for this kind of thing [12:36] Justin Clark-Casey: can you say anything about those things or would you rather wait? [12:36] Allen.Kerensky @icemoon.no-ip.org:8002: mostly me asking stupid questions about what needs to be done with each service OSG had configured lol [12:36] James Stallings: well we're looking into certain backend changes [12:37] James Stallings: replication and HA filesystems are front and center [12:37] Nebadon Izumi: most of the initial changes will be about having solid backups [12:37] Marcus.Llewellyn @grid.verymad.net:8002: I've always regarded OSgrid's open grid model as an aberration. It isn't how OpenSim is meant to run. But I bet, at least for core services, it has been good testbed and source of reports from the admins. It has other less tangible benefits for users. [12:37] James Stallings: yes [12:37] Nebadon Izumi: we have a replication asset server in place now [12:37] Dahlia.Trimble @grid.wwweb3d.net:8002: an abberation thats being copied a lot [12:37] James Stallings: the ability to perform hot incremental backups is a key need we are attempting to address [12:37] Allen.Kerensky @icemoon.no-ip.org:8002: for example, do we need jump4000/jump8000 regions anymore or can we recover the resources for those [12:37] Nebadon Izumi: mysql replication and possibly some other methods that melanie will guide us with for replicating blobs [12:38] Nebadon Izumi: nothings 100% decided there yet [12:38] Nebadon Izumi: still exploring best solution that fits our servers [12:38] Marcus.Llewellyn @grid.verymad.net:8002: The only time I've run into 4000/800 scenarios is when trying to HG to someone running 0.7.6. I'm looking at you, Caltech. [12:38] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Allen, I thought the TP limit is gone now so jump regions should be a thing of the past [12:38] Allen.Kerensky @icemoon.no-ip.org:8002: yeah just me questioning stuff to see how much I've gotten myself into [12:38] Nebadon Izumi: we dont need those anymore [12:38] Allen.Kerensky @icemoon.no-ip.org:8002: Andrew exactly why I was bothering them about it hehehe [12:38] Nebadon Izumi: they should go away for sure [12:38] James Stallings: lol caltech [12:39] Melanie Milland: OSG would be better off with 4 servers @3tb rather than 2 servers @6tb [12:39] Justin Clark-Casey: melanie: I think notice about a new assets esrvice should be posted to the dev mailing list first so other devs will know what's going on [12:39]  Nebadon Izumi: honestly though other than this asset mess [12:39] Marcus.Llewellyn @grid.verymad.net:8002: +1 mailing list [12:39] Nebadon Izumi: all of the other services have held up amazingly well [12:40] Nebadon Izumi: having opensimulator behind nginx and balancing things works very well [12:40] James Stallings: Mel: financial constraints [12:40] Nebadon Izumi: osgrid is not just stock opensiim exposed to the world [12:40] Nebadon Izumi: its very different than say how OSCC grid is setup [12:40] Nebadon Izumi: its very well protected by nginx [12:40] Justin Clark-Casey: but you still have the big vulnerability of having to expose the private servivce port/s to region operators [12:41] Melanie Milland: well, that is mostly a content risk [12:41] Melanie Milland: which is why there isn't commercial content [12:41] Nebadon Izumi: yea content on OSgrid cant be protected [12:41] Marcus.Llewellyn @grid.verymad.net:8002: Many things on OSgrid are vulnerable. Content permission are, for all practical purposes, useless there. It's a price one pays for joining that grid. [12:42] Justin Clark-Casey: inventory operations are also a problem [12:42] Nebadon Izumi: even if those were not exposed, anyone hosting a region can snag any content [12:42] Nebadon Izumi: well we dont allow deletes [12:42] Nebadon Izumi: nothing can ever really be permantly deleted [12:42] Justin Clark-Casey: as are various other service operations [12:42] James Stallings: it's a job of mitigation and mediation [12:42] James Stallings: we learn a lot from it [12:42]  Justin Clark-Casey: have you looked into storing assets in the cloud? [12:42] Marcus.Llewellyn @grid.verymad.net:8002: And users accept those risks too. Or should. [12:42] James Stallings: it's pricey [12:42] Justin Clark-Casey: let someone else take care of the baclup problem [12:43] Justin Clark-Casey: ok [12:43]  Nebadon Izumi: we have looked into cold storage [12:43] Nebadon Izumi: via amazon [12:43] James Stallings: yeah I was about to mention that [12:43] Nebadon Izumi: its cheap [12:43] James Stallings: slow but viable as a backup solution [12:43] Nebadon Izumi: http://aws.amazon.com/glacier/ [12:43] Melanie Milland: justin: "the cloud" just means "servers you don't control". for the huge space and bandwidth assets need, that isn't workable financially and even "cloud" servers can lose data [12:44] Melanie Milland: and you'll have NO recovery from that [12:44] Melanie Milland: because they'll never send you the HD [12:44]  Nebadon Izumi: yea its long process to get data back unfortunately [12:44] Nebadon Izumi: thats what happened with osgrid originally [12:44] Melanie Milland: i don't know why it took this long, tbh [12:44] Nebadon Izumi: for a long time i was runningn backups to servers at my house [12:45] Nebadon Izumi: but the hardware eventually died [12:45] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Some cloud storage solutions claim you don't lose data as they copy the data to multiple places. [12:45] Nebadon Izumi: honestly I am a bit perplexed at why it took so long myself [12:45] Seth.Nygard @refugegrid.com:8002: ... until they actually do lose it ... [12:45] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Seth: yea, :) [12:46]  Nebadon Izumi: I am not all that impressed with the recovery service that the data center uses [12:46]  Nebadon Izumi: but we can't change that [12:46]  Aine.Caoimhe @refugegrid.com:8002: htey don't lose it...they just can't find it [12:46]  Marcus.Llewellyn @grid.verymad.net:8002: I doubt anyone is. :/ [12:46]  Nebadon Izumi: what is done is done [12:46]  James Stallings: this took so long because there were acually two full recovery operations. [12:46]  Arielle.Popstar @login.greatcanadiangrid.ca:8002: lol Aine [12:46]  James Stallings: the first failed when delivered to us. [12:46]  Melanie Milland: well, let me know when i can take a peek at the data [12:46]  James Stallings: you need to see the old sras data mel? [12:46]  Nebadon Izumi: ok we will for sure, step one is getting new fassets functional [12:46] Melanie Milland: yes [12:47] James Stallings: Ok I will get it 'hooked up' [12:47] Melanie Milland: i want to see how hard a reimport is [12:47]  James Stallings: nods [12:47] Melanie Milland: it may be trivial [12:47] Melanie Milland: we did make a tool for it and i may even find it [12:47]  James Stallings: I'll ping you on IRC when its avail [12:47] Marcus.Llewellyn @grid.verymad.net:8002: From your fingers to fate's ears. :) [12:47] Nebadon Izumi: i suspect sras doesnt have the embedded data your server does [12:47]  Nebadon Izumi: im like 98% sure it doesnt [12:47]  Melanie Milland: we shall see [12:47]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Melanie, there is always my migration script that might be helpful. [12:47]  Nebadon Izumi: but if we are luckly there should be 5 sql dumps [12:47]  Nebadon Izumi: and if were really lucky the actual mysql structure is still valid [12:48]  Melanie Milland: if the data were naked, that would be pretty indecent.... [12:48]  Nebadon Izumi: i give us 50/50 chance right now [12:48]  Nebadon Izumi: either way grids coming back up [12:48]  Melanie Milland: neb, you ain't seen nuthin' yet [12:48]  Nebadon Izumi: hehe if anyone can do it its you Melanie [12:48]  Melanie Milland: giving up just isn't in my book [12:48]  Nebadon Izumi: i have no doubt of that [12:49] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Rebuilding the database from blobs on disk is a problem if there isn't extra data in the blobs that would assist in recreating the assets db. [12:49] BlueWall.Slade @gateway.bluewallvirtual.net:8002: yeah, I want my "No Pants Weekend Barrel" [12:49] Shez.Oyen @mirage.duckdns.org:9038: Melanie, if I can wash your car or walk your dog to give your more time for OSG, let me know! [12:49] Nebadon Izumi: lol [12:49] Marcus.Llewellyn @grid.verymad.net:8002: I know it's not as valuable, but if you guys need additional crash test dummies in testing, from people who're willing to lose stuff, don't hesitate to put out that call. [12:49] Melanie Milland: i need to run [12:49] Nebadon Izumi: yea dont worry when it comes testing time, you all will know [12:49] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Thanks Melanie [12:49] Marcus.Llewellyn @grid.verymad.net:8002: Thank you Mel. :) [12:49] Justin Clark-Casey: ok, I need to run too [12:49]  Melanie Milland: you're welcome [12:49]  BlueWall.Slade @gateway.bluewallvirtual.net:8002: Thanks Justin [12:49]  Melanie Milland: bye for now [12:49]  Nebadon Izumi: ya thanks guys, this ran longer than i expected [12:49]  Justin Clark-Casey: best of luck to you guys, I hope you get osgrid back up soon [12:49]  Aine.Caoimhe @refugegrid.com:8002: ty all [12:49]  Nebadon Izumi: but i am super happy we all had time finally [12:49]  Aine.Caoimhe @refugegrid.com:8002: have a great week [12:49]  Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I'd also like to get going. [12:50]  Jak Daniels: thank you everyone :) [12:50] Seth.Nygard @refugegrid.com:8002: have a great week everyone, take care [12:50] Nebadon Izumi: talk soon everyone we will keep everyone updated as we humanly can [12:50] Dahlia.Trimble @grid.wwweb3d.net:8002: sorry I stepped away. Does this mean Melanie will push the Avination asset server to core? [12:50] Arielle.Popstar @login.greatcanadiangrid.ca:8002: good meeting [12:50] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Let me know if my scripts might be helpful in doing migrations and if they need mods. [12:50] Marcus.Llewellyn @grid.verymad.net:8002: Dahlia, hopefully this week. [12:50] Nebadon Izumi: thanks Andrew we will [12:50] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I have to take off too - Thanks everyone. See you all next week. [12:50] Dahlia.Trimble @grid.wwweb3d.net:8002: cool [12:50] BlueWall.Slade @gateway.bluewallvirtual.net:8002 is Offline