[Opensim-dev] some scalability tests...
diva at metaverseink.com
diva at metaverseink.com
Wed Jan 27 15:57:30 UTC 2010
Err.. that's EstablishAgentCommunication, not EnableChildAgent. Same
initials, different order :-)
diva at metaverseink.com wrote:
> More coolness!
>
> WRT opening up child agents in neighboring regions. Right now there's a
> very simple, but too inflexible, rule: the only child agents opened are
> those in immediate regions, and they open both the UDP and the CAPs
> connections. The inflexibility comes from our fuzzy understanding of how
> the viewer works, specifically wrt the events EnableSimulator and
> EnableChildAgent -- this last one wasn't being generated by OpenSim
> until last March or so. I think we (at least I) know more now than 1
> year ago. The idea is to make this more flexible, maybe configurable.
>
>
> Mic Bowman wrote:
>> final entry on this...
>>
>> we seem to have found a combination of settings, execution environment,
>> and changes to opensim that give us a reasonably stable experience
>> (there is nothing "normal" about feeding the viewer 100 regions to
>> render so keep it all in perspective, its most likely going to break if
>> you hang out there long enough :-)...
>>
>> * we got rid of the megaregions. we wanted a "many" region view that we
>> get with megaregons, but we didn't need the multi-region physics. we
>> also found that megaregions next to megaregions have some very painful
>> effects on border crossings especially going west & south. and big
>> megaregions are worse (as far as i could tell, the viewer thinks it
>> doesn't have connections outside its 10x10 range and the sim thinks the
>> viewer already has the connection and then weird stuff happens when the
>> region actually comes in view). instead of the megaregions we hacked
>> opensim to give out all of the regions within 5 coordinates of the
>> current region (which seems to be the max the viewer can handle). that
>> works out to about 100 regions that are visible. the change we made for
>> the test isn't particularly useful in general, but i think we can try to
>> get the viewers far clip distance to set the number of regions to send
>> back.
>>
>> * 64 bit was causing all kinds of problems with the jpeg decoder (no
>> surprise). given that the only jpeg being decoded was the map image,
>> there might be something in the way the map is created that causes the
>> problems. Regardless... given that every time you opened the map view in
>> the viewer, the region would grab you and not let go (no further TPs, no
>> border crossings, etc) it seemed like we needed a different solution. so
>> now we're running 16 simulators, each 8x8 for a 64 regions per simulator
>> and we're using the 32 bit launcher to force it to stay in 32 bit mode.
>>
>> * another thing that caused problems at times was the number of mysql
>> connections. increasing the number of simulators increases the number of
>> connections. db contention was causing some bad problems on startup...
>> we haven't dug into the root cause. instead, we've changed to use an
>> alternative mysql connection management solution that uses mysql
>> connection pooling. that seemed to get us past the initial problems
>> (yeah... you can always up the mysql connections... but more efficient
>> use of the connections seems like a good idea). john's working on a
>> patch against the current opensim master for that code.
>>
>> at any rate... it works. and for proof here's a picture of yellowstone
>> national park from the top of one of the mountains on the eastern side
>> of the park:
>> http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png
>>
>> (or come see it on the scisim grid.., search for the "geography" regions
>> on the map)
>>
>> --mic
>>
>>
>>
>>
>> On Fri, Jan 22, 2010 at 6:01 PM, <diva at metaverseink.com
>> <mailto:diva at metaverseink.com>> wrote:
>>
>> +100! Really great that Intel is pushing the performance limits...
>>
>> Diva
>> (coming home from an event in the university that had lots of local
>> industry, including gaming industry, even more convinced that the key
>> piece that is missing in all of it, for the longest time, is... an open
>> source MMO platform that *everyone* can explore with)
>>
>> Kyle wrote:
>> > Fantastic work Intel!
>> >
>> >
>> >
>> > *From:* opensim-dev-bounces at lists.berlios.de
>> <mailto:opensim-dev-bounces at lists.berlios.de>
>> > [mailto:opensim-dev-bounces at lists.berlios.de
>> <mailto:opensim-dev-bounces at lists.berlios.de>] *On Behalf Of *Mic Bowman
>> > *Sent:* Friday, January 22, 2010 7:19 PM
>> > *To:* opensim-dev at lists.berlios.de
>> <mailto:opensim-dev at lists.berlios.de>;
>> scisim-discuss at googlegroups.com <mailto:scisim-discuss at googlegroups.com>
>> > *Subject:* Re: [Opensim-dev] some scalability tests...
>> >
>> >
>> >
>> > Grid mode. Connected to SciSim. Thanks to some help from Brian,
>> we put
>> > the Yellowstone National Park terrain on it today. I think Brian
>> said it
>> > works out to about a 1:10 scale. With the far view distance its
>> pretty
>> > impressive (terrain textures are borked). The performance is
>> terrible,
>> > but good enough to look around some. Even a completely empty region
>> > consumes resources and 1000 of them consume a LOT of resources.
>> If you
>> > come over to sciencesim, look for "geography11 00". I'll probably be
>> > restarting the regions to get the map tiles updated & i'm not
>> making any
>> > promises to keep it up very long. But it does make for a very neat
>> > demonstration and its been a very useful experiment.
>> >
>> > --mic
>> >
>> > On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou
>> <impalah at gmail.com <mailto:impalah at gmail.com>
>> > <mailto:impalah at gmail.com <mailto:impalah at gmail.com>>> wrote:
>> >
>> > Just a stupid silly question: standalone or grid mode? If grid: where
>> > were the ugaim servers? Same machine?
>> >
>> > And a comment: 1024 regions? 8 hours for booting? Weird!
>> >
>> > 2010/1/22 Mic Bowman <cmickeyb at gmail.com
>> <mailto:cmickeyb at gmail.com> <mailto:cmickeyb at gmail.com
>> <mailto:cmickeyb at gmail.com>>>
>> >
>> > this is just fyi... and a very positive comment about how far opensim
>> > has come in recent months!
>> >
>> > as part of sizing the hw requirements for a mirror world project
>> we're
>> > exploring... we wanted to do some scalability tests on the
>> capacity of
>> > individual simulators in terms of the total number of regions. we
>> wanted
>> > baseline numbers that focus on just the most simple region
>> > configuration: a completely empty region with default terrain. that
>> > is... this is JUST simulator overhead.
>> >
>> > all tests are done on one of our blades... dual proc, quad core
>> with 16G
>> > ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
>> > regions in a 32x32 grid. the simulator configuration was our standard
>> > sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>> >
>> > the first configuration ran all 1024 regions in *one* simulator. i
>> > honestly figured this would crash quickly. it didn't. we managed
>> to get
>> > all 1024 regions created and running well enough to walk around.
>> it did
>> > take almost 8 hours to start. the first couple regions were
>> created in
>> > 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
>> > clearly there is something quadratic in there (stop using linear
>> lists,
>> > they are evil! :-). but it could have been the mono garbage
>> collector.
>> > who knows... not sure its worth too much investigation because i
>> can't
>> > imagine ever running a config like this for real. the simulator did
>> > crash when we opened the map in the viewer. the crash was caused
>> because
>> > we ran out of sockets. while it was running, the simulator used just
>> > over 10G of ram and was running at about 700% CPU utilization
>> (kind of
>> > scary to see load averages in the 900 range!).
>> >
>> > the second configuration ran 16 simulators each with 64 regions.
>> startup
>> > took about 30 minutes (each of the 16 simulators avoided the
>> quadratic
>> > "knee" we hit with the one big simulator). consumed about 11G of
>> memory
>> > and was again consuming essentially all cycles on the machine
>> > (completely idle regions aren't very idle). all 16 simulators
>> died just
>> > after startup with a "too many open files" error. not sure what
>> caused
>> > it, but all of them died loading the same terrain dll as part of some
>> > wildcard function looking for dlls. that one is an interesting bug.
>> >
>> > the final configuration and the one we're shooting for in the
>> short term
>> > runs 16 simulators, each with an 8x8 megaregion. again startup was
>> > around 40 minutes. we might be able to cut that time down by starting
>> > all 16 simulators simultaneously rather than 4 at a time. i
>> really just
>> > wanted to make sure that some of the spikes we see in startup didn't
>> > cause failures (some race condition causes all threads to consume
>> > maximum processor cycles randomly on startup right now). and,
>> well... it
>> > just worked. i figured the viewer would die horribly (it can't handle
>> > 250K "real" prims very well) but it survived just fine. turn off far
>> > clip with 8x8 megaregions providing neighbors and you are
>> "capable" of
>> > contacting simulators in a 24x24 region range!. the view is
>> pretty cool
>> > though it seems to not go beyond 12x12. :-) there are a bunch of
>> > problems (like you can't see the terrain in the region you're
>> standing
>> > in)... but there are a LOT of little humps of terrain in view.
>> oh... and
>> > it takes a lot of patience to get everything loaded.
>> >
>> > so... the conclusion... this opensim thing is pretty amazing!
>> good work
>> > everyone.
>> >
>> > --mic
>> >
>> > _______________________________________________
>> > Opensim-dev mailing list
>> > Opensim-dev at lists.berlios.de
>> <mailto:Opensim-dev at lists.berlios.de>
>> <mailto:Opensim-dev at lists.berlios.de
>> <mailto:Opensim-dev at lists.berlios.de>>
>> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > Opensim-dev mailing list
>> > Opensim-dev at lists.berlios.de
>> <mailto:Opensim-dev at lists.berlios.de>
>> <mailto:Opensim-dev at lists.berlios.de
>> <mailto:Opensim-dev at lists.berlios.de>>
>> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >
>> >
>> >
>> > No virus found in this incoming message.
>> > Checked by AVG - www.avg.com <http://www.avg.com>
>> > Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date:
>> 01/22/10
>> > 14:33:00
>> >
>> >
>> >
>> ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > Opensim-dev mailing list
>> > Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
>> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
More information about the Opensim-dev
mailing list