[Opensim-dev] Quantifying and Replicating Open Simulator Performance (UNCLASSIFIED)

[Diva Canto]

Excellent, thanks for sharing these papers!Looks like they have much in 
common with the one I sent out yesterday.

(as an aside, I recently started uploading preprints of my papers to 
arXiv, in the period between they are accepted until they are actually 
published. That way, a close-to-final version of the papers becomes 
available to everyone without stepping on the publishers' copyrights... 
this plus the automatic citation notifications of Google Schoolar makes 
it a powerful advertising platform for academics!)

Anyway, performance and non-functional testing of distributed systems is 
definitely "the lost world" of software engineering, and a lot needs to 
happen in order to get it out of the dark age. OpenSim is a fantastic 
framework for doing this kind of work!


On 8/20/2015 6:03 AM, Maxwell, Douglas CIV USARMY RDECOM (US) wrote:
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> We've had trouble replicating the successes advertised by other Grids and Open Simulator activities over the years.  The OSCC was an impressive achievement, however it took direct involvement and hand-tuning by Nebadon & others to help us get the MOSES grid in shape for last Spring's Federal Consortium for Virtual Worlds.  This isn't an indictment in any way of the people who worked on the high performance grids, simply a broad observation that grid deployment is complicated and the sprint-like development leaves documentation in an anemic shape.
>
> The MOSES team has decided to take a deliberate, scientific approach to quantifying affects of some of these variables on performance.  The tangible result you can see are the recent three large patches where we provided fixes and updates in the stats collectors in the Open Simulator.  Using these new reliable statistics measures, we have begun publishing our findings in the following recent venues:
>
> Summer Sim 2015 - these papers are behind a paywall, but it you contact me directly I can get you PDFs.
> Conference Program: http://scs.org/documents/SummerSim15/SummerSim15%20Program%20Final.pdf
>
> 1)  Vertical Scalability Benchmarking in Three-Dimensional Virtual World Simulation
> Abstract:
> "The United States military is investigating large-scale, realistic virtual world simulations to facilitate war-fighter training. As the simulation community strives towards meeting these military training objectives, methods must be developed and validated that measure scalability performance in these virtual world simulators. With such methods, the simulation community will be able to quantifiably compare scalability performance between system changes. This work contributes to the development and validation prerequisite by evaluating the effectiveness of commonly used system metrics to measure scalability in a three-dimensional virtual trainer. Specifically, the metrics of CPU utilization and simulation frames per second are evaluated for their effectiveness in vertical scalability benchmarking."
>
> 2)  Approach to Examine Efficacy of Game-Based and Virtual Simulation Training
> Abstract:
> "The United States Army has heavily leveraged, developed and expanded its use of virtual simulation training, as this class of simulation has been empirically demonstrated to be effective in the transfer of skills to the live environment. Game-based training, an alternative class of simulation, is characterized by its lower overhead and cost and potentially represents a less expensive alternative to virtual simulation training. In an effort to reduce the cost of training simulation, the U.S. Army has recently socialized the concept of potentially replacing select virtual simulation trainers with game-based simulations. While lowering the cost of simulation is a noble endeavor, the aforementioned concept requires further investigation as minimal empirical evidence exists regarding the effectiveness of game-based training, particularly at the collective echelon of training. In this paper, we lay the foundation to conduct an investigation of whether a game-based simulation may b
>   e as equally an effective collective training apparatus as a higher-cost, higher-fidelity, virtual simulation. Specifically, we discuss a planned Training Effectiveness Evaluation (TEE) of both the Aviation Combined Arms Tactical Trainer (AVCATT) and a game-based aviation simulation that will empirically determine whether or not virtual training in the AVCATT could potentially be replaced by an equally effective, but less costly, game-based simulation. We discourse on our proposed design of experiment, which will utilize qualified Army aviators performing a tactical, collective mission in two discrete training treatments (AVCATT and a game based simulation) at Fort Rucker, Alabama."
>
> Open Journal of Modeling and Simulation
>
> 3)  Analyzing Virtual World Region Fidelity on Scalability and Simulation Performance
> http://www.scirp.org/journal/PaperDownload.aspx?paperID=58322
>
> As you can see, we are not only making advancements with the Open Simulator, we are also testing our work with real soldiers.  It is important for you, the devs, to understand that the work you are doing is making a tangible difference.
>
> v/r -doug
>
> Dr. Douglas Maxwell
> Science and Technology Manager
> Virtual World Strategic Applications
> U.S. Army Research Lab
> Simulation & Training Technology Center (STTC)
> (c) (407) 242-0209
>
>
> -----Original Message-----
> From: opensim-dev-bounces at opensimulator.org [mailto:opensim-dev-bounces at opensimulator.org] On Behalf Of Diva Canto
> Sent: Wednesday, August 19, 2015 8:24 PM
> To: opensim-dev at opensimulator.org
> Subject: [Opensim-dev] Paper available
>
> May be of interest to some on this list:
> http://arxiv.org/abs/1508.04465
>
> Lots of references to mantises and assorted OpenSim things.
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>