[2014/11/18 11:04] Mal.Burns @grid.avacon.org:8002 is Online
[2014/11/18 11:04] Nebadon Izumi: which sucks
[2014/11/18 11:04] Nebadon Izumi: but im not sure we can do much about that
[2014/11/18 11:04] Connecting to in-world Voice Chat...
[2014/11/18 11:04] Richardus.Raymaker @hypergrid.org:8002: so, lots of noobs for a long time
[2014/11/18 11:04] Connected
[2014/11/18 11:04] Marcus.Llewellyn @grid.verymad.net:8002: We've defeated the vast cloud army before. ;)
[2014/11/18 11:04] Nebadon Izumi: other than have lots of free avatars for everyone tat the landing zones
[2014/11/18 11:04] Richardus.Raymaker @hypergrid.org:8002: some sims with clothing and skin are maby gone
[2014/11/18 11:04] Nebadon Izumi: ya but it sholdnt be too bad
[2014/11/18 11:05] Justin Clark-Casey: hi folks
[2014/11/18 11:05] Marcus.Llewellyn @grid.verymad.net:8002: If you guys need lab rats, just holler.
[2014/11/18 11:05] Aine.Caoimhe @refugegrid.com:8002: have a really big Ruth party on day 1
[2014/11/18 11:05] Alicia.Raven @grid.spellscape.co.uk: hi justin
[2014/11/18 11:05] Marcus.Llewellyn @grid.verymad.net:8002: heya Justin :)
[2014/11/18 11:05] Nebadon Izumi: ok will doo
[2014/11/18 11:05] Andrew Hellershanks is Online
[2014/11/18 11:05] Nebadon Izumi: we will try to keep everyone informed as much as we can
[2014/11/18 11:05] Jak Daniels: so do we need to reupload our regions from oars, or can we just reconnect them as is?
[2014/11/18 11:05] Nebadon Izumi: Allen Kerensky is now on board as a Grid Admin
[2014/11/18 11:05] Nebadon Izumi: he will be helping out a bit with this
[2014/11/18 11:05] Nebadon Izumi: I am kind of in the background
[2014/11/18 11:05] Aine.Caoimhe @refugegrid.com:8002: when do you leave for Sweden?
[2014/11/18 11:05] Nebadon Izumi: mostly the grid diplomat
[2014/11/18 11:06] Richardus.Raymaker @hypergrid.org:8002: hi justin. well i like to see smooth avatar flying sooner. its still not smooth in 0.8.0.3
[2014/11/18 11:06] Nebadon Izumi: sometime in January
[2014/11/18 11:06] Nebadon Izumi: hello Justin
[2014/11/18 11:06] Andrew Hellershanks: I missed something about osgrid?
[2014/11/18 11:06] Justin Clark-Casey: richardus: it may be smoother soon - I think one of my changes recently may have improved it
[2014/11/18 11:07] Nebadon Izumi: yes we have started the grid recovery process
[2014/11/18 11:07] Nebadon Izumi: clean database for assets and inventory
[2014/11/18 11:07] Richardus.Raymaker @hypergrid.org:8002: ok. cool justin
[2014/11/18 11:07] Marcus.Llewellyn @grid.verymad.net:8002: OSgrid has been sold to Rubert Murdoch. We'll be his newest cable channel. :)
[2014/11/18 11:07] Justin Clark-Casey: I had a few days offline holiday but will be preparing for the merge from ghosts now soon
[2014/11/18 11:07] Nebadon Izumi: which I think while it ultimately sucks to loose content
[2014/11/18 11:07] Nebadon Izumi: i think the grid is going to run much better
[2014/11/18 11:07] Nebadon Izumi: we can rebuild the content, with better stuff
[2014/11/18 11:07] Andrew Hellershanks: Good to hear that the grid is coming back up. Not so good if the data recovery process failed.
[2014/11/18 11:07] Nebadon Izumi: the good news is
[2014/11/18 11:08] Nebadon Izumi: we probably get our money back
[2014/11/18 11:08] Nebadon Izumi: for the recovery services
[2014/11/18 11:08] Aine.Caoimhe @refugegrid.com:8002: I'm sure purging 19222853093821309 pine cubes and spheres from inventory will help a little
[2014/11/18 11:08] Nebadon Izumi: which is substantial refund
[2014/11/18 11:08] Justin Clark-Casey: sucks for them, but that's life I guess
[2014/11/18 11:08] Andrew Hellershanks thinks of the opening of Six Million Dollar Man
[2014/11/18 11:08] Sarah Kline: hi
[2014/11/18 11:08] Justin Clark-Casey: hi sarah
[2014/11/18 11:08] Nebadon Izumi: the grid will be in a good financial position though
[2014/11/18 11:08] Andrew Hellershanks: "We can rebuild him. We have the technology". :)
[2014/11/18 11:08] Marcus.Llewellyn @grid.verymad.net:8002: I don't think anyone can say you guys did try your best to recover data. Anyone who does is getting smacked with my Stupid Bat.
[2014/11/18 11:08] Nebadon Izumi: donations have been amazing
[2014/11/18 11:08] Justin Clark-Casey: marcus: heh your bat might be busy then
[2014/11/18 11:09] Marcus.Llewellyn @grid.verymad.net:8002: I can make copies. ;)
[2014/11/18 11:09] Aine.Caoimhe @refugegrid.com:8002: I suspect there are a lot of people with IARs and OARs ready and waiting to go so in the end probably the vast majority of assets will be re-injected fairly quickly
[2014/11/18 11:09] Nebadon Izumi: ya its unfortunate the success of recovery was completely out of our hands
[2014/11/18 11:09] Richardus.Raymaker @hypergrid.org:8002: Well, it works faster to clean the databvase from old i unused data then justingfs version :)
[2014/11/18 11:09] Marcus.Llewellyn @grid.verymad.net:8002: I tested it on myself. It really works.
[2014/11/18 11:09] Nebadon Izumi: but the new asset serer will be 2 servers
[2014/11/18 11:09] Nebadon Izumi: a complete mirror of the live data set
[2014/11/18 11:09] Seth.Nygard @refugegrid.com:8002: Data recovery on that scale is not an easy thing to do
[2014/11/18 11:09] Nebadon Izumi: ready to go if the main server goes kaput
[2014/11/18 11:10] Nebadon Izumi: but like i said earlier incase you missed it
[2014/11/18 11:10] Nebadon Izumi: the grid costs are going to rise
[2014/11/18 11:10] Marcus.Llewellyn @grid.verymad.net:8002: Is this just a simple round robin things, or is this something that's built into SRAS?
[2014/11/18 11:10] Nebadon Izumi: by about 305$ a month
[2014/11/18 11:10] Nebadon Izumi: puts us around 1000$ a month in costs
[2014/11/18 11:10] Andrew Hellershanks: ouch
[2014/11/18 11:10] Justin Clark-Casey: you're going to use a different hw approach?
[2014/11/18 11:10] Nebadon Izumi: yes basically run 2 asset servers
[2014/11/18 11:10] Nebadon Izumi: and it will post to both
[2014/11/18 11:10] Nebadon Izumi: but only pull from one
[2014/11/18 11:10] Richardus.Raymaker @hypergrid.org:8002: If the hdd's are from the same bathc and run in 2 servers. uhmm. ok a breaking controller is rare
[2014/11/18 11:11] Aine.Caoimhe @refugegrid.com:8002: @Justin...after the ghost merge is there any chance you could take another look into inventory fetch? That's probably going to be the single biggest region-killer remaining
[2014/11/18 11:11] Nebadon Izumi: it will be new drives all around of course :)
[2014/11/18 11:11] Nebadon Izumi: 8 3tb drives
[2014/11/18 11:11] Nebadon Izumi: so 6tb of assets mirrored 3 times basically
[2014/11/18 11:11] Justin Clark-Casey: aine: yes, though there are a number of other tasks queued up as well. Also, I need good test examples - I've only ever replicated it once on osgrid and that may have been due to something else
[2014/11/18 11:12] Aine.Caoimhe @refugegrid.com:8002: Bluewall and I were able to reliably replicate it on his region
[2014/11/18 11:12] Andrew Hellershanks: RL is calling. bbiab
[2014/11/18 11:12] quaezar.Agnomen @sciencecircle.org:8002: Good day to you all
[2014/11/18 11:12] Justin Clark-Casey: ok, I didn't know that
[2014/11/18 11:12] Marcus.Llewellyn @grid.verymad.net:8002 smiles affably. :)
[2014/11/18 11:12] Justin Clark-Casey: hi quaezar
[2014/11/18 11:12] Aine.Caoimhe @refugegrid.com:8002: and in RefugeGrid I can reliably replicate it 100% of the time by logging into my own region
[2014/11/18 11:12] Justin Clark-Casey: or bye...
[2014/11/18 11:12] quaezar.Agnomen @sciencecircle.org:8002: :)
[2014/11/18 11:12] Jak Daniels: is there an easy way to get content from an orphaned region back into the osgrid assets?
[2014/11/18 11:13] Aine.Caoimhe @refugegrid.com:8002: and my ping to that server is 32ms
[2014/11/18 11:13] Seth.Nygard @refugegrid.com:8002: I think the inventory fetch issue happens to most places, so should not be hard to reproduce
[2014/11/18 11:13] Aine.Caoimhe @refugegrid.com:8002: (between my region and the grid server)
[2014/11/18 11:13] Marcus.Llewellyn @grid.verymad.net:8002: Is this all about slow inventory loading?
[2014/11/18 11:13] Justin Clark-Casey: I think more frozen than slow
[2014/11/18 11:13] Aine.Caoimhe @refugegrid.com:8002: it never happened on my standalone, and it never happens in a grid-hosted region
[2014/11/18 11:14] Aine.Caoimhe @refugegrid.com:8002: it's frozen
[2014/11/18 11:14] Justin Clark-Casey: seth: it's mujch more useful if at some point somebody has it reliably and they allow me direct access to their serfer to run test code
[2014/11/18 11:14] Aine.Caoimhe @refugegrid.com:8002: and it cycles through repeated fetch attempt for a dozen or so items
[2014/11/18 11:14] Justin Clark-Casey: server
[2014/11/18 11:14] Nebadon Izumi: ive seen that before too
[2014/11/18 11:14] Alicia.Raven @grid.spellscape.co.uk: do others get alot of failed web fetch messages for hg inventory?
[2014/11/18 11:14] Nebadon Izumi: but its been a while
[2014/11/18 11:14] Aine.Caoimhe @refugegrid.com:8002: Blue and I confirmed that the asset server is sending back the data
[2014/11/18 11:14] Justin Clark-Casey: because I've already done a lot of work on it without getting useful information
[2014/11/18 11:14] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online
[2014/11/18 11:14] Aine.Caoimhe @refugegrid.com:8002: but the region doesn't know what to do with it and asks for the same data all over again
[2014/11/18 11:15] Aine.Caoimhe @refugegrid.com:8002: eventually it will queue up as many requests as the region server has memory to hold
[2014/11/18 11:15] Seth.Nygard @refugegrid.com:8002: I can arrange that Justin. I want to be in part a small test grid to help development
[2014/11/18 11:15] Marcus.Llewellyn @grid.verymad.net:8002: Ew
[2014/11/18 11:15] Nebadon Izumi: wonder if 1 invnetory service is not enough
[2014/11/18 11:15] Nebadon Izumi: osgrid runs 5 inventory consoles
[2014/11/18 11:15] Aine.Caoimhe @refugegrid.com:8002: so even if the person logs out it will continue sending the fetch requests for minutes (even hours!)
[2014/11/18 11:15] Nebadon Izumi: with nginx round robin
[2014/11/18 11:15] Justin Clark-Casey: seth: cool, pls let me know if you do. I can't say exactly when I would get time to work on it but I do want to
[2014/11/18 11:15] Marcus.Llewellyn @grid.verymad.net:8002: I haven't seen that, but I haven't upgraded my sims since before OSCC.
[2014/11/18 11:15] Aine.Caoimhe @refugegrid.com:8002: and while it's doing that the rest of the region services are buggered
[2014/11/18 11:15] Justin Clark-Casey: seth: or potentially other core devs may be able to look at it again, like BlueWall
[2014/11/18 11:15] Aine.Caoimhe @refugegrid.com:8002: only way to fix it is restart the instance
[2014/11/18 11:16] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online
[2014/11/18 11:16] Nebadon Izumi: ya maybe its too m,uch for a single robust instance
[2014/11/18 11:16] Justin Clark-Casey: there probably does need to be a max limit on requests
[2014/11/18 11:16] Nebadon Izumi: is it a larger grid?
[2014/11/18 11:16] Nebadon Izumi: ya
[2014/11/18 11:16] Dahlia.Trimble @grid.wwweb3d.net:8002: hi
[2014/11/18 11:16] Justin Clark-Casey: opensim is in general bad at enforcing limits to protect itself
[2014/11/18 11:16] Nebadon Izumi: osgrid limits requests with nginx
[2014/11/18 11:16] Justin Clark-Casey: hi dahlia
[2014/11/18 11:16] Seth.Nygard @refugegrid.com:8002 is Online
[2014/11/18 11:16] Alicia.Raven @grid.spellscape.co.uk: hi dahlia
[2014/11/18 11:16] Marcus.Llewellyn @grid.verymad.net:8002: Heya dahlia
[2014/11/18 11:16] quaezar.Agnomen @sciencecircle.org:8002: Hi Dahlia
[2014/11/18 11:16] Nebadon Izumi: which is probably why osgrid runs so well
[2014/11/18 11:16] Seth.Nygard @refugegrid.com:8002: While I agree 1 Robust is light, the fetch issue never appears in sims off the same server, only to ones remote from it
[2014/11/18 11:17] Nebadon Izumi: wierd
[2014/11/18 11:17] Dahlia.Trimble @grid.wwweb3d.net:8002: fetch issue?
[2014/11/18 11:17] Nebadon Izumi: that suggests network problems
[2014/11/18 11:17] Aine.Caoimhe @refugegrid.com:8002: the interesting thing is it will happen to me even if I'm the only user online so it's not an issue of an overwhelmed asset server
[2014/11/18 11:17] Dahlia.Trimble @grid.wwweb3d.net:8002: I have a fetch issue also
[2014/11/18 11:17] Nebadon Izumi: inventory has not assocation to assets
[2014/11/18 11:17] Nebadon Izumi: completely seperate processes
[2014/11/18 11:17] Aine.Caoimhe @refugegrid.com:8002: kk...inventory server, sorry
[2014/11/18 11:17] Seth.Nygard @refugegrid.com:8002: Yes, and observed at osgrid, Metro, Refuge, and I was able to reproduce in my home setup as well
[2014/11/18 11:18] Nebadon Izumi: ya there was always people on osgrid who couldnt do inventory in their regions
[2014/11/18 11:18] Justin Clark-Casey: my previous guess was that all available inventory service requests got tied up for some reason, but my last batch of logging data did not show that
[2014/11/18 11:18] Nebadon Izumi: but got o a plaza and its fine
[2014/11/18 11:18] Nebadon Izumi: definitely something at the region level
[2014/11/18 11:18] Nebadon Izumi: not grid services level
[2014/11/18 11:18] Aine.Caoimhe @refugegrid.com:8002: I've never been able to get it to happen to me on my standalone althrough FS will sometimes pretend that it's fetching forever until you "empty" trash (even if trash is already empty)
[2014/11/18 11:18] Nebadon Izumi: it to me suggests bad networking maybe
[2014/11/18 11:19] Aine.Caoimhe @refugegrid.com:8002: but in those cases there's no HTTP traffic on the region
[2014/11/18 11:19] Marcus.Llewellyn @grid.verymad.net:8002: My inventory loading issues stopped on OSgrid once Melanie introduced throttling. It was perfect for me after that.
[2014/11/18 11:19] Nebadon Izumi: between regions and robust or a major bug in simulators that only effect certain people
[2014/11/18 11:19] Nebadon Izumi: im more prone to blame network problems
[2014/11/18 11:19] Dahlia.Trimble @grid.wwweb3d.net:8002: I always had slow inventory load on osgrid
[2014/11/18 11:19] Nebadon Izumi: but need better data to confirm that
[2014/11/18 11:19] Dahlia.Trimble @grid.wwweb3d.net:8002: even recently
[2014/11/18 11:19] Nebadon Izumi: ya i never had invnetory issues
[2014/11/18 11:19] Justin Clark-Casey: even if it is network problenms, what we really need is code that reports that, rather than leave the user guessing
[2014/11/18 11:19] Nebadon Izumi: and my inventory was gigantic
[2014/11/18 11:19] Nebadon Izumi: 50k items
[2014/11/18 11:19] Dahlia.Trimble @grid.wwweb3d.net:8002: takes several minutes, cant even rez my avatar
[2014/11/18 11:19] Justin Clark-Casey: of course, all those 'slow' requests then get annoying after a while :)
[2014/11/18 11:19] Nebadon Izumi: 1000s of folders
[2014/11/18 11:20] Nebadon Izumi: there was certain places i could visit
[2014/11/18 11:20] Nebadon Izumi: that sucked
[2014/11/18 11:20] Nebadon Izumi: bluewalls region had issues i recall
[2014/11/18 11:20] Nebadon Izumi: but i would go to my regions or other areas and it was fine
[2014/11/18 11:20] Aine.Caoimhe @refugegrid.com:8002: it doesn't seem to depend on inventory size at all...it was happening to Bluewall even with <1000 items in inventory
[2014/11/18 11:20] Dahlia.Trimble @grid.wwweb3d.net:8002: ya mine worked better on plazas
[2014/11/18 11:20] Nebadon Izumi: BlueWall has internet quality issues at times i recall
[2014/11/18 11:20] Aine.Caoimhe @refugegrid.com:8002: we spent a Sunday testing it and he posted the results to the Mantis
[2014/11/18 11:20] Richardus.Raymaker @hypergrid.org:8002: then its a good test sim
[2014/11/18 11:20] Nebadon Izumi: DSL connection in the boonies
[2014/11/18 11:21] Marcus.Llewellyn @grid.verymad.net:8002: I can't say inventory loading for me is fast. I wouldn't quite call it slow though either. It's a steady progression that usually take a minute or two.
[2014/11/18 11:22] Justin Clark-Casey: the problem is viewers should not really be reloading inventory all the time
[2014/11/18 11:22] Justin Clark-Casey: they should be able to use their cached informatin
[2014/11/18 11:22] Justin Clark-Casey: it's a puzzle as to why some are doing so many requests
[2014/11/18 11:22] Nebadon Izumi: ya unless your folder version # changes
[2014/11/18 11:22] Nebadon Izumi: it should never requrest again
[2014/11/18 11:22] Justin Clark-Casey: At one time, I heard firestorm had to do it for some of their AO stuff
[2014/11/18 11:22] Andrew Hellershanks is back
[2014/11/18 11:22] Dahlia.Trimble @grid.wwweb3d.net:8002: I have another interesting, possibly relatted problem. I loaded an old oar of wright plaza onto a region on my grid (entirely hosted on a single machine). The region was extremely laggy until the region asset cache filled
[2014/11/18 11:22] Nebadon Izumi: ya melanie mentioned major problems with viewer AO
[2014/11/18 11:23] Nebadon Izumi: and why its very bad
[2014/11/18 11:23] Nebadon Izumi: great for the users
[2014/11/18 11:23] Alicia.Raven @grid.spellscape.co.uk: it is possible to tell the viewer not to preload the whole inventory aswell, just load it when a folder is opened
[2014/11/18 11:23] Aine.Caoimhe @refugegrid.com:8002: AO and outfits seem to always need re-fetching
[2014/11/18 11:23] Nebadon Izumi: bad for the admins
[2014/11/18 11:23] Dahlia.Trimble @grid.wwweb3d.net:8002: once the cache had all assets, movement was perfect
[2014/11/18 11:23] Dahlia.Trimble @grid.wwweb3d.net:8002: had lots of slow messages on console
[2014/11/18 11:23] Dahlia.Trimble @grid.wwweb3d.net:8002: http
[2014/11/18 11:23] Justin Clark-Casey: I do think fetching assets from the serve is inefficient
[2014/11/18 11:23] Nebadon Izumi: the viewer AO if i recall
[2014/11/18 11:23] Nebadon Izumi: it requrests assets before invnetory does
[2014/11/18 11:23] Justin Clark-Casey: I've even seen 'slow' requests on a purely local grid with an unpopulated region asset cache
[2014/11/18 11:23] Nebadon Izumi: and it can cause overlap maybe
[2014/11/18 11:23] Dahlia.Trimble @grid.wwweb3d.net:8002: entire grid is on a single machine so couldnt be networking
[2014/11/18 11:23] Justin Clark-Casey: maybe it simply takes a lot of time to pull some of those giant assets out of the db
[2014/11/18 11:24] Aine.Caoimhe @refugegrid.com:8002: if a solution to that can't be found, could you add a region setting allowing you to prevent logins?
[2014/11/18 11:24] Dahlia.Trimble @grid.wwweb3d.net:8002: but it was as if the fetches were delaying normal sim operation
[2014/11/18 11:24] Dahlia.Trimble @grid.wwweb3d.net:8002: including physics
[2014/11/18 11:24] Marcus.Llewellyn @grid.verymad.net:8002: I always get slow requests ona region with an empty asset cache. I disable logins until it populates, or it's unusable.
[2014/11/18 11:24] Seth.Nygard @refugegrid.com:8002: Testing I've done certainly shows FS tends to show the inventory fetch issue more regularly than does Singularity, with everything else the same
[2014/11/18 11:24] Dahlia.Trimble @grid.wwweb3d.net:8002: and the cpu was idle most of the time
[2014/11/18 11:24] Nebadon Izumi: certain types of assets seem more problematic than others
[2014/11/18 11:24] Nebadon Izumi: i recall animations and sounds
[2014/11/18 11:24] Nebadon Izumi: reaaly taking an eternity to load
[2014/11/18 11:24] Aine.Caoimhe @refugegrid.com:8002: yes
[2014/11/18 11:24] Nebadon Izumi: but other stuff was instant
[2014/11/18 11:25] Nebadon Izumi: even very small sounds and animations
[2014/11/18 11:25] Nebadon Izumi: stuff that should be not much worse than a small linkset
[2014/11/18 11:25] Dahlia.Trimble @grid.wwweb3d.net:8002: so maybe the fetch threads interfere with the main sim thread?
[2014/11/18 11:25] Nebadon Izumi: my racer is a good example
[2014/11/18 11:25] Nebadon Izumi: of sounds not working
[2014/11/18 11:25] Nebadon Izumi: when you shift gears
[2014/11/18 11:25] Nebadon Izumi: it takes sometimes 5+ minutes
[2014/11/18 11:25] Nebadon Izumi: for the sounds to load
[2014/11/18 11:25] Marcus.Llewellyn @grid.verymad.net:8002: Wright Plaza has an insane number of textures. I had the info windows for texture queue and sim stats up, and I could see that when I had a full textures queue, sim ping went through the roof. When it emptied, it went back to normal.
[2014/11/18 11:25] Aine.Caoimhe @refugegrid.com:8002: if there was a way to force people to log into a grid-served region it would also remove the issue since it never seems to happen on those
[2014/11/18 11:26] Justin Clark-Casey: marcus: ping going up can have a number of causes
[2014/11/18 11:26] Justin Clark-Casey: marcus: it can go up if either viewer or simulator is overloaded
[2014/11/18 11:26] Justin Clark-Casey: or if network is poor
[2014/11/18 11:26] Dahlia.Trimble @grid.wwweb3d.net:8002: the ping was the same time as the console messages justin
[2014/11/18 11:26] Nebadon Izumi: Wright Plaza is nasty
[2014/11/18 11:26] Nebadon Izumi: never log into wright plaza without first doing fcache assets
[2014/11/18 11:26] Nebadon Izumi: and letting it complete
[2014/11/18 11:26] Justin Clark-Casey: marcus: for instance, it coul dbe that inserting lots of big textures into the viewer prevented it from servicing its own incoming udp in a timely manner
[2014/11/18 11:26] Nebadon Izumi: we have startup_comands.txt run it
[2014/11/18 11:27] Marcus.Llewellyn @grid.verymad.net:8002: Yes, but I could reproduce this easily. Full queue, high ping. Empty queue, low ping. Move my camera, and let the queue expode... high ping.. RInse and repeat.
[2014/11/18 11:27] Nebadon Izumi: and we do not allow logins until it finishes
[2014/11/18 11:27] Dahlia.Trimble @grid.wwweb3d.net:8002: so what I did was set the cache to no expire and reload the oar, now it runs perfectly smooth
[2014/11/18 11:27] Nebadon Izumi: ya that works too dahlia
[2014/11/18 11:27] Justin Clark-Casey: marcus: I would guess that's actually a viewer-side issue
[2014/11/18 11:27] Nebadon Izumi: a save oar will also cache everything
[2014/11/18 11:27] Marcus.Llewellyn @grid.verymad.net:8002: Dahlia was also experiencing slowdowns when I saw high ping times.
[2014/11/18 11:27] Dahlia.Trimble @grid.wwweb3d.net:8002: Marcus was on my wright region justin
[2014/11/18 11:28] Justin Clark-Casey: ok. How many cores does it have?
[2014/11/18 11:28] Alicia.Raven @grid.spellscape.co.uk: fcache assets command also good
[2014/11/18 11:28] Marcus.Llewellyn @grid.verymad.net:8002: So it wasn't just viewer side.
[2014/11/18 11:28] Nebadon Izumi: Wright Plaza is like a simulator nuclear bomb
[2014/11/18 11:28] Nebadon Izumi: lol
[2014/11/18 11:28] Dahlia.Trimble @grid.wwweb3d.net:8002: 4 real cones, no HT
[2014/11/18 11:28] Dahlia.Trimble @grid.wwweb3d.net:8002: *cores
[2014/11/18 11:28] Justin Clark-Casey: and the machine just runs the one region?
[2014/11/18 11:28] Dahlia.Trimble @grid.wwweb3d.net:8002: and all were idle
[2014/11/18 11:28] Justin Clark-Casey: I'd be interested to know what the experience is like with ghosts when it gets merged to master
[2014/11/18 11:28] Dahlia.Trimble @grid.wwweb3d.net:8002: no it runs many that were idle, along with the grid service
[2014/11/18 11:29] Marcus.Llewellyn @grid.verymad.net:8002: Once her assest cache filled up, we were walking around and stuff just fine.
[2014/11/18 11:29] Justin Clark-Casey: ghosts reduces some of the worst spots where opensim is extremely thread happy
[2014/11/18 11:29] Justin Clark-Casey: though asset fetching may still be one of those spots
[2014/11/18 11:29] Justin Clark-Casey: or it could be something else entirely....
[2014/11/18 11:29] Nebadon Izumi: ya it doesnt surprise me at all
[2014/11/18 11:29] Dahlia.Trimble @grid.wwweb3d.net:8002: lol thread happy is an understatement
[2014/11/18 11:29] Nebadon Izumi: even our very big iron has the same issue
[2014/11/18 11:30] Nebadon Izumi: i dont think hardware is a factor at all
[2014/11/18 11:30] Nebadon Izumi: even with 24 cores and gigabit network
[2014/11/18 11:30] Nebadon Izumi: wright plaza is harsh
[2014/11/18 11:30] Dahlia.Trimble @grid.wwweb3d.net:8002: anyway changing the cache to no expire helped a lot
[2014/11/18 11:30] Justin Clark-Casey nods
[2014/11/18 11:30] Justin Clark-Casey: we need better service side asset acache management options
[2014/11/18 11:31] Justin Clark-Casey: server side, sorry
[2014/11/18 11:31] Justin Clark-Casey: simulator side I mean really
[2014/11/18 11:31] Nebadon Izumi: our HTTP service is probably not the best
[2014/11/18 11:31] Nebadon Izumi: that may be a factor
[2014/11/18 11:31] Dahlia.Trimble @grid.wwweb3d.net:8002: well flotsam seems to work but the default 48 hour expire is pretty short
[2014/11/18 11:32] Aine.Caoimhe @refugegrid.com:8002: when asset cache is set to (for example) 1 day, does it only expire assets that haven't been accessed in the last 24 hours or does it purge *everything* that was initially cached 24 hours ago?
[2014/11/18 11:32] Justin Clark-Casey: I think we need an option to actually allow you to set the max size of the simulator asset cache
[2014/11/18 11:32] Justin Clark-Casey: it's bonkers we don't currently have that imo
[2014/11/18 11:32] Dahlia.Trimble @grid.wwweb3d.net:8002: I also pointed all the region caches to the same folder
[2014/11/18 11:32] Nebadon Izumi: eh
[2014/11/18 11:32] Marcus.Llewellyn @grid.verymad.net:8002: Neb, if wright is brought up with an empty asset cache on OSgrid, does it get slow requests at first? I'm wondering if there's different behavior between robust assets and sras.
[2014/11/18 11:32] Justin Clark-Casey: aine: should be just stuff that hasn't been accessed for 24 hours
[2014/11/18 11:32] Nebadon Izumi: size to me isnt much of a factor
[2014/11/18 11:32] Nebadon Izumi: when you could make 100 assets that are the same size as 100,000 assets
[2014/11/18 11:32] Justin Clark-Casey: right, but at the moment you have to manually check your have disk splace left for your cache
[2014/11/18 11:33] Justin Clark-Casey: which is bonkers
[2014/11/18 11:33] Nebadon Izumi: there were linksets in the OSgrid database that were 50mb
[2014/11/18 11:33] Aine.Caoimhe @refugegrid.com:8002: wow
[2014/11/18 11:33] Dahlia.Trimble @grid.wwweb3d.net:8002: linksets?
[2014/11/18 11:33] Justin Clark-Casey: tbh, I think at some point the data storage format has to change....
[2014/11/18 11:33] Nebadon Izumi: probably made by me
[2014/11/18 11:33] Nebadon Izumi: lol
[2014/11/18 11:33] Andrew Hellershanks: wow.
[2014/11/18 11:33] Justin Clark-Casey: there's a lot of redundant data in there, and the XML really does get huge and costly to parse
[2014/11/18 11:33] Nebadon Izumi: ya big assets they were just xml
[2014/11/18 11:33] Dahlia.Trimble @grid.wwweb3d.net:8002: gotta love xml *pukes*
[2014/11/18 11:33] Andrew Hellershanks: hehe, yeah
[2014/11/18 11:33] Nebadon Izumi: but we had 100s of not 1000s of assets that were 50mb
[2014/11/18 11:33] Andrew Hellershanks: :P
[2014/11/18 11:34] Nebadon Izumi: i dont think anything was larger than 50mb
[2014/11/18 11:34] Aine.Caoimhe @refugegrid.com:8002: it's that 20k-prim roller coaster you built....and it's physical :p
[2014/11/18 11:34] Nebadon Izumi: probably would time out if it was any larger
[2014/11/18 11:34] Dahlia.Trimble @grid.wwweb3d.net:8002: Im up to about 13 gb assets just on my baby grid
[2014/11/18 11:34] Andrew Hellershanks: oh, the roller coaster. I hope someone has a copy of that. That was brilliant.
[2014/11/18 11:34] Nebadon Izumi: OSCC grid is like 25gb
[2014/11/18 11:34] Aine.Caoimhe @refugegrid.com:8002: pertty sure Cute has a backup of the ones she made
[2014/11/18 11:34] Nebadon Izumi: the database dump
[2014/11/18 11:35] Nebadon Izumi: we have roller coasters backed up
[2014/11/18 11:35] Justin Clark-Casey: It would be interesting to have a completely mirrored asset database :)
[2014/11/18 11:35] Nicky Perian is Online
[2014/11/18 11:35] Dahlia.Trimble @grid.wwweb3d.net:8002: well the prims are probably in her region database
[2014/11/18 11:35] Justin Clark-Casey: where volunteers could effectively backup parts of it, assuming no-one treats the assets as sensitive
[2014/11/18 11:35] Nebadon Izumi: that is the plan for new OSgrid v2 justin
[2014/11/18 11:35] Dahlia.Trimble @grid.wwweb3d.net:8002: but I heard she had a grid
[2014/11/18 11:35] Justin Clark-Casey: though even then, maybe you could encryp them
[2014/11/18 11:35] Aine.Caoimhe @refugegrid.com:8002: Cute does have a grid now, yes
[2014/11/18 11:35] Justin Clark-Casey: nebadon: beyond having a redundant copy of all assets?
[2014/11/18 11:36] Nebadon Izumi: no it will basically just mirror everything
[2014/11/18 11:36] Nebadon Izumi: can could if needed function as a primary asset server
[2014/11/18 11:36] Justin Clark-Casey: but osgrid manages both mirrors right?
[2014/11/18 11:36] Nebadon Izumi: it wont be mysql mirror
[2014/11/18 11:36] Nebadon Izumi: assets will post live to both services
[2014/11/18 11:36] Nebadon Izumi: but reads will only happen on the primary
[2014/11/18 11:36] Dahlia.Trimble @grid.wwweb3d.net:8002: anyway I have a pretty smoothly running nostalgia version of wright plaza up if anyone wants to visit or help load test :) http://grid.wwweb3d.net:8002 Wright Plaza
[2014/11/18 11:36] Justin Clark-Casey: I'm thinking of something more exotic where other people can mirror the assets....
[2014/11/18 11:37] Andrew Hellershanks: DB replication is good. If any other machines in osgrid have the space they could also hold static snapshots of the main DB's saved from the replication slave DB server.
[2014/11/18 11:37] Justin Clark-Casey: only a pipe thought though, enough other tasks to do :)
[2014/11/18 11:37] Justin Clark-Casey: effectively, caching is a kind of mirroring but on a tiny section of assets
[2014/11/18 11:37] Aine.Caoimhe @refugegrid.com:8002: better yet, have each user mirroring their own inventory assets as part of the viewer code
[2014/11/18 11:38] Justin Clark-Casey: I guess then you have the question of whether an asset is tampered with or not though, etc.
[2014/11/18 11:38] Dahlia.Trimble @grid.wwweb3d.net:8002: those distributed nosql thingies might be worth considering
[2014/11/18 11:38] Justin Clark-Casey: aine: yeah, I would love to see that
[2014/11/18 11:38] Justin Clark-Casey: dahlia: yeah, I found tranq's talk v interesting on that. Had no chance to look at his example code though
[2014/11/18 11:38] Dahlia.Trimble @grid.wwweb3d.net:8002: oh thought he was talking inventory. I meant assets
[2014/11/18 11:38] Dahlia.Trimble @grid.wwweb3d.net:8002: but I didnt see his talk
[2014/11/18 11:38] Justin Clark-Casey: he was, but equally applicable to assets I think
[2014/11/18 11:39] Billy.Bradshaw @hg.viewtwo.net:8600: We have databases on separate machines, this enables the os to be tuned. As I understand things replication is a mysql feature.
[2014/11/18 11:39] Justin Clark-Casey: yes
[2014/11/18 11:39] Dahlia.Trimble @grid.wwweb3d.net:8002: I was playing with riak for use as an asset server a couple years back but I never got far
[2014/11/18 11:40] Aine.Caoimhe @refugegrid.com:8002: I would think having asset de-duplication as part of core would also help a lot
[2014/11/18 11:40] Andrew Hellershanks: I'd love to see that.
[2014/11/18 11:40] Justin Clark-Casey: it would reduce size but not the risk of failure
[2014/11/18 11:40] Andrew Hellershanks nods
[2014/11/18 11:41] Aine.Caoimhe @refugegrid.com:8002: but commonly-accessed things with common UUID could at least be held in memory cache so fewer DB queries
[2014/11/18 11:41] Dahlia.Trimble @grid.wwweb3d.net:8002: getting rid of materials as individual assets would probably help also but may be difficult considering all the content that uses them now
[2014/11/18 11:41] Andrew Hellershanks: When I can get my main computer up and running again I can revisit my asset blobs on disk patch.
[2014/11/18 11:41] Justin Clark-Casey: dahlia: it has to be one
[2014/11/18 11:41] Justin Clark-Casey: I suspect
[2014/11/18 11:41] Justin Clark-Casey: done
[2014/11/18 11:41] Dahlia.Trimble @grid.wwweb3d.net:8002: Justin, why?
[2014/11/18 11:41] Dahlia.Trimble @grid.wwweb3d.net:8002: oh done
[2014/11/18 11:42] Dahlia.Trimble @grid.wwweb3d.net:8002: ya it probably does
[2014/11/18 11:42] Justin Clark-Casey: aine: The thing is no, asset is really common fetched since they're only ever fetched once
[2014/11/18 11:42] Justin Clark-Casey: aine: So, people have found memroy caches not to be helpful in the past, esp as you would require a very large amount of memory to cache any significant portion of the scene
[2014/11/18 11:43] Justin Clark-Casey: tbh, the network latency swamps any effect anyway
[2014/11/18 11:43] Dahlia.Trimble @grid.wwweb3d.net:8002: most operating systems will use free memory to cache files anyway
[2014/11/18 11:43] Dahlia.Trimble @grid.wwweb3d.net:8002: so another memory cache is just redundant
[2014/11/18 11:44] Billy.Bradshaw @hg.viewtwo.net:8600: Recently I have scanned a few grids for for response times, and found several have port 3306 (mysql) open. Is there a reason for that, bit of a no no in our camp? Worried that some of this database corruption is due to hacking
[2014/11/18 11:44] Nebadon Izumi: nginx makes a good cache
[2014/11/18 11:44] Richardus.Raymaker @hypergrid.org:8002: why use memory cache if you have this days ssd
[2014/11/18 11:44] Justin Clark-Casey: billy: it's a no no, that's bad
[2014/11/18 11:44] Nebadon Izumi: thats how we protect the osgrid sras server
[2014/11/18 11:44] Nebadon Izumi: if someone makes a request to the same asset 500 times
[2014/11/18 11:44] Nebadon Izumi: it only hits nginx
[2014/11/18 11:44] Alicia.Raven @grid.spellscape.co.uk: we dont use port 3306 either lol
[2014/11/18 11:44] Nebadon Izumi: and not the sras/mysql server
[2014/11/18 11:45] Nebadon Izumi: and we limit how many requests per second any single ip address can make
[2014/11/18 11:45] Justin Clark-Casey: whilst immutable assets do cause the sorage problems, they have the big advantage of allowing caching everywhere
[2014/11/18 11:45] Richardus.Raymaker @hypergrid.org:8002: why would you have 3306 open to the internet. or at least put a firewall on it that only allow the admin ip
[2014/11/18 11:45] Nebadon Izumi: ya 3306 open to the world
[2014/11/18 11:45] Nebadon Izumi: very very veeeeeery bad idea
[2014/11/18 11:45] Dahlia.Trimble @grid.wwweb3d.net:8002: Id think nginx would be redundant also for a file based asset server
[2014/11/18 11:45] Nebadon Izumi: dahlia
[2014/11/18 11:45] Nebadon Izumi: it prevents mysql from flooding
[2014/11/18 11:45] Nebadon Izumi: takes away uneeded mysql calls
[2014/11/18 11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: *file based*
[2014/11/18 11:46] Nebadon Izumi: beyond that ya its probably redundant a bit
[2014/11/18 11:46] Nebadon Izumi: there is still mysql in file based
[2014/11/18 11:46] Nebadon Izumi: i dont think you totally erase the need for mysql
[2014/11/18 11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: in sras there is
[2014/11/18 11:46] Nebadon Izumi: you store UUID / hash
[2014/11/18 11:46] Nebadon Izumi: how would you do that otherwise?
[2014/11/18 11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: links?
[2014/11/18 11:47] Nebadon Izumi: that sounds potentially messy
[2014/11/18 11:47] Justin Clark-Casey: I'd read a lot about db vs file system asset caching and there's no conclusion that one of the other is better
[2014/11/18 11:47] Nebadon Izumi: hard to clean
[2014/11/18 11:47] Marcus.Llewellyn @grid.verymad.net:8002: If you start distributing assets to volunteers, how would any deduplication system work with that?
[2014/11/18 11:47] Dahlia.Trimble @grid.wwweb3d.net:8002: so does myswl
[2014/11/18 11:47] Dahlia.Trimble @grid.wwweb3d.net:8002: mysql
[2014/11/18 11:47] Justin Clark-Casey: marcus: yeah, perhaps too much of a pipe dream
[2014/11/18 11:47] Dahlia.Trimble @grid.wwweb3d.net:8002: thats what links were designed for
[2014/11/18 11:47] Nebadon Izumi: yea
[2014/11/18 11:47] Justin Clark-Casey: store them in the blockchain! :p
[2014/11/18 11:47] Andrew Hellershanks: In theory, straight file cache has a tiny bit less overhead.
[2014/11/18 11:47] Nebadon Izumi: i found databases like monogodb very hard to manage
[2014/11/18 11:48] Nebadon Izumi: i guess its good if you write all your own tools
[2014/11/18 11:48] Billy.Bradshaw @hg.viewtwo.net:8600: How relational is the data in the asset tables?
[2014/11/18 11:48] Dahlia.Trimble @grid.wwweb3d.net:8002: the simplest version is just a flat table
[2014/11/18 11:48] Justin Clark-Casey: billy: in original assets, none. In xassets there are some joins (shock)
[2014/11/18 11:49] Justin Clark-Casey: which may or may not be an issue for things like cassandra, though that could just implement in an entirely different way from how it's done in a relational db
[2014/11/18 11:49] Billy.Bradshaw @hg.viewtwo.net:8600: thanks Justin
[2014/11/18 11:49] Justin Clark-Casey: as the asset interface remains unchanged
[2014/11/18 11:50] Dahlia.Trimble @grid.wwweb3d.net:8002: anyway I kinda wonder if maybe region asset requests result in a bazillion "fire and forget" REST requests that just murder the threadpools
[2014/11/18 11:51] Justin Clark-Casey: dahlia: could be, thuogh it is limited to 30
[2014/11/18 11:51] Dahlia.Trimble @grid.wwweb3d.net:8002: and if maybe they can be done serially or something
[2014/11/18 11:51] Aine.Caoimhe @refugegrid.com:8002: the other thing on my "to whine about" list is the recent break to animations which is a pretty big deal (http://opensimulator.org/mantis/view.php?id=7357)
[2014/11/18 11:51] Dahlia.Trimble @grid.wwweb3d.net:8002: is the limit configurable?
[2014/11/18 11:51] Justin Clark-Casey: dahlia: the trouble with serial is that one long request then holds up everything
[2014/11/18 11:51] Justin Clark-Casey: dahlia: not sure
[2014/11/18 11:51] Justin Clark-Casey: which is a particular problem with hypergrid
[2014/11/18 11:51] Billy.Bradshaw @hg.viewtwo.net:8600: Glad for the response to the 3306 open thing, at least people reading these meetings will see it is not just me..
[2014/11/18 11:52] Dahlia.Trimble @grid.wwweb3d.net:8002: requests can also be queued on the asset server end
[2014/11/18 11:52] Justin Clark-Casey: aine: yes, I will need to look at the animations issue before doing any release
[2014/11/18 11:52] Justin Clark-Casey: aine: if it is a regression
[2014/11/18 11:52] Dahlia.Trimble @grid.wwweb3d.net:8002: s/can/could
[2014/11/18 11:52] Aine.Caoimhe @refugegrid.com:8002: it's somewhere between the last 2 I listed
[2014/11/18 11:53] Nebadon Izumi: Billy just use firewall
[2014/11/18 11:53] Nebadon Izumi: to only allow the ip addresses that need access to 3306
[2014/11/18 11:53] Richardus.Raymaker @hypergrid.org:8002: correct
[2014/11/18 11:53] Nebadon Izumi: otherwise your asking to be packet snifed on the data center network
[2014/11/18 11:53] Nebadon Izumi: and hacked
[2014/11/18 11:53] Billy.Bradshaw @hg.viewtwo.net:8600: Understand that, but if I can scan then not such restrictions in place, but yes..
[2014/11/18 11:53] Alicia.Raven @grid.spellscape.co.uk: nebadon, what if an admin has changing ip? id say its better to use a random port
[2014/11/18 11:54] Dahlia.Trimble @grid.wwweb3d.net:8002: Justin, maybe there's a way asset request tasks could be deprioritized?
[2014/11/18 11:54] Nebadon Izumi: then SSH into box
[2014/11/18 11:54] Nebadon Izumi: via a tunnel
[2014/11/18 11:54] Richardus.Raymaker @hypergrid.org:8002: then the admin need to adjust the firewall.
[2014/11/18 11:54] Nebadon Izumi: and connect to mysql over a secure tunnel
[2014/11/18 11:54] Justin Clark-Casey: dahlia: I don't know - I didn't get to that stage in my conference work, bu tit would have been a next area of focus
[2014/11/18 11:54] Alicia.Raven @grid.spellscape.co.uk: thats another point, never run ssh on the default port either lol
[2014/11/18 11:54] Justin Clark-Casey: luckily, it proved unnecessary in the end
[2014/11/18 11:54] Nebadon Izumi: alicia im not sure that matters
[2014/11/18 11:54] Nebadon Izumi: that wouldnt stop a skilled hacker
[2014/11/18 11:55] Nebadon Izumi: doesnt take very long to port scan a machine
[2014/11/18 11:55] Dahlia.Trimble @grid.wwweb3d.net:8002: ya I dont recall seeing any priority parameter back when Dan Lake and I were poking aorund in the treadpool code
[2014/11/18 11:55] Marcus.Llewellyn @grid.verymad.net:8002: A port scan will find it no matter what port it's on if you have it open to the public. The lesson is, don't open it to the public.
[2014/11/18 11:56] Justin Clark-Casey: the more scary one for me is people opening their private serive port to the public
[2014/11/18 11:56] Alicia.Raven @grid.spellscape.co.uk: we were getting dozens of loggin attempts from china and russia, better to block them at the firewall than waste time checking, and i checked logs, numbers went down from hundreds per day to zero just by changing the port
[2014/11/18 11:56] Seth.Nygard @refugegrid.com:8002: Changing the ports a service runs on will only delay a determined hacker by a couple minutes. It really does not do much to help protect a system. There are much better measures to take.
[2014/11/18 11:56] Justin Clark-Casey: far easier to do
[2014/11/18 11:56] Justin Clark-Casey: service
[2014/11/18 11:56] Nebadon Izumi: ya security by obscurity is a sure path to being overwhelmed
[2014/11/18 11:56] Justin Clark-Casey: alicia: yeah, I guess it does protect from the script kiddies who are 99% of the hackers out there
[2014/11/18 11:56] Dahlia.Trimble @grid.wwweb3d.net:8002: oh I use ssh tunnels for ports that are grid only use
[2014/11/18 11:57] Nebadon Izumi: brute force ssh attacks are not very common
[2014/11/18 11:57] Justin Clark-Casey: I think the private service interfaces should perhaps require a password by default when you set up opensim
[2014/11/18 11:57] Dahlia.Trimble @grid.wwweb3d.net:8002: tho diva added that password feature for grid services
[2014/11/18 11:57] Justin Clark-Casey: yeah, that
[2014/11/18 11:57] Billy.Bradshaw @hg.viewtwo.net:8600: If you have a reasonable firewall, you can use iptables to restrict the numbers of times a port can be accessed within a given time period, so prevent brute force attacks. I use Cisco Pix.
[2014/11/18 11:57] Nebadon Izumi: you attract alot of attention with brute force
[2014/11/18 11:57] Dahlia.Trimble @grid.wwweb3d.net:8002: I think its plaintext tho?
[2014/11/18 11:58] Dahlia.Trimble @grid.wwweb3d.net:8002: or maybe a hash
[2014/11/18 11:58] Justin Clark-Casey: dahlia: yeah, tjhat's another issue, lack of https use...
[2014/11/18 11:58] Richardus.Raymaker @hypergrid.org:8002: and waht happens. users are lazy and do '12345' ?
[2014/11/18 11:58] Dahlia.Trimble @grid.wwweb3d.net:8002: its not encrypted
[2014/11/18 11:58] Seth.Nygard @refugegrid.com:8002: What may be a good wiki page to do sometime is some information on proper setup running on public;y accessible IPs. That may help a lot of people who may simply not know what should be done
[2014/11/18 11:58] Justin Clark-Casey: dahlia: well, if most requests are internal to a network not quite so bad
[2014/11/18 11:58] Justin Clark-Casey: it would be more to protect those who just turn their firewall off, or make a mistake and think they jhave to open both 8002 and 8003 or similar
[2014/11/18 11:59] Dahlia.Trimble @grid.wwweb3d.net:8002: ya or if you share a datacenter internal network with people you dont know
[2014/11/18 11:59] Dahlia.Trimble @grid.wwweb3d.net:8002: I just use tunnels :)
[2014/11/18 11:59] Justin Clark-Casey: true, or any larger network
[2014/11/18 12:00] Richardus.Raymaker @hypergrid.org:8002: ovh. servers in own datacenter attacked other servers
[2014/11/18 12:00] Justin Clark-Casey: ok, I'm going off to catch up on my e-mail and other comms pile and at some poitn hopefully do some coding
[2014/11/18 12:00] Sheera.Khan @hypergrid.org:8002: one question if I may: whom may I remind of the missing chatlogs of the office hours in the opensimulator wiki?
[2014/11/18 12:00] Billy.Bradshaw @hg.viewtwo.net:8600: enemies both sides of the border lol
[2014/11/18 12:00] Andrew Hellershanks: wow. this hour flew by
[2014/11/18 12:01] Dahlia.Trimble @grid.wwweb3d.net:8002: anyway if anyone has a couple free minutes and wants to help me crash my server http://grid.wwweb3d.net:8002 Wright Plaza
[2014/11/18 12:01] Justin Clark-Casey: sheera: yeah, I've not even had time to post them over the last few weeks
[2014/11/18 12:01] Sheera.Khan @hypergrid.org:8002: very interesting topics :-)
[2014/11/18 12:01] Justin Clark-Casey: sheera: I planned to catch up today unless Neb wants to do it :)
[2014/11/18 12:01] Marcus.Llewellyn @grid.verymad.net:8002: Neb's asleep. Ssshhhhh.
[2014/11/18 12:01] Billy.Bradshaw @hg.viewtwo.net:8600: I will be the dahlia, name a time
[2014/11/18 12:01] Dahlia.Trimble @grid.wwweb3d.net:8002: now is ok :)
[2014/11/18 12:01] Sheera.Khan @hypergrid.org:8002: I would be glad to be able to read them :-)
[2014/11/18 12:01] Aine.Caoimhe @refugegrid.com:8002: have a great week everyone
[2014/11/18 12:02] Justin Clark-Casey: sheera: ok, hopefully they will be up later today
[2014/11/18 12:02] Billy.Bradshaw @hg.viewtwo.net:8600: ok, let me paste that address into map
[2014/11/18 12:02] Andrew Hellershanks: I should be catch up on some email.
[2014/11/18 12:02] Justin Clark-Casey: bye everyone
[2014/11/18 12:02] Marcus.Llewellyn @grid.verymad.net:8002 smiles.
[2014/11/18 12:02] Richardus.Raymaker @hypergrid.org:8002: bye marcus. im hopping behind you dahlia
[2014/11/18 12:02] Sheera.Khan @hypergrid.org:8002: thank you very much for that:-)
[2014/11/18 12:02] Alicia.Raven @grid.spellscape.co.uk: bye :)
