[Opensim-dev] Quantifying and Replicating Open Simulator Performance (UNCLASSIFIED)

Thu Aug 20 13:03:37 UTC 2015

Classification: UNCLASSIFIED
Caveats: NONE

We've had trouble replicating the successes advertised by other Grids and Open Simulator activities over the years.  The OSCC was an impressive achievement, however it took direct involvement and hand-tuning by Nebadon & others to help us get the MOSES grid in shape for last Spring's Federal Consortium for Virtual Worlds.  This isn't an indictment in any way of the people who worked on the high performance grids, simply a broad observation that grid deployment is complicated and the sprint-like development leaves documentation in an anemic shape.

The MOSES team has decided to take a deliberate, scientific approach to quantifying affects of some of these variables on performance.  The tangible result you can see are the recent three large patches where we provided fixes and updates in the stats collectors in the Open Simulator.  Using these new reliable statistics measures, we have begun publishing our findings in the following recent venues:

Summer Sim 2015 - these papers are behind a paywall, but it you contact me directly I can get you PDFs.
Conference Program: http://scs.org/documents/SummerSim15/SummerSim15%20Program%20Final.pdf

1)  Vertical Scalability Benchmarking in Three-Dimensional Virtual World Simulation 
Abstract:  
"The United States military is investigating large-scale, realistic virtual world simulations to facilitate war-fighter training. As the simulation community strives towards meeting these military training objectives, methods must be developed and validated that measure scalability performance in these virtual world simulators. With such methods, the simulation community will be able to quantifiably compare scalability performance between system changes. This work contributes to the development and validation prerequisite by evaluating the effectiveness of commonly used system metrics to measure scalability in a three-dimensional virtual trainer. Specifically, the metrics of CPU utilization and simulation frames per second are evaluated for their effectiveness in vertical scalability benchmarking."

2)  Approach to Examine Efficacy of Game-Based and Virtual Simulation Training
Abstract:
"The United States Army has heavily leveraged, developed and expanded its use of virtual simulation training, as this class of simulation has been empirically demonstrated to be effective in the transfer of skills to the live environment. Game-based training, an alternative class of simulation, is characterized by its lower overhead and cost and potentially represents a less expensive alternative to virtual simulation training. In an effort to reduce the cost of training simulation, the U.S. Army has recently socialized the concept of potentially replacing select virtual simulation trainers with game-based simulations. While lowering the cost of simulation is a noble endeavor, the aforementioned concept requires further investigation as minimal empirical evidence exists regarding the effectiveness of game-based training, particularly at the collective echelon of training. In this paper, we lay the foundation to conduct an investigation of whether a game-based simulation may be as equally an effective collective training apparatus as a higher-cost, higher-fidelity, virtual simulation. Specifically, we discuss a planned Training Effectiveness Evaluation (TEE) of both the Aviation Combined Arms Tactical Trainer (AVCATT) and a game-based aviation simulation that will empirically determine whether or not virtual training in the AVCATT could potentially be replaced by an equally effective, but less costly, game-based simulation. We discourse on our proposed design of experiment, which will utilize qualified Army aviators performing a tactical, collective mission in two discrete training treatments (AVCATT and a game based simulation) at Fort Rucker, Alabama."

Open Journal of Modeling and Simulation

3)  Analyzing Virtual World Region Fidelity on Scalability and Simulation Performance
http://www.scirp.org/journal/PaperDownload.aspx?paperID=58322

As you can see, we are not only making advancements with the Open Simulator, we are also testing our work with real soldiers.  It is important for you, the devs, to understand that the work you are doing is making a tangible difference.

v/r -doug

Dr. Douglas Maxwell
Science and Technology Manager
Virtual World Strategic Applications
U.S. Army Research Lab
Simulation & Training Technology Center (STTC)
(c) (407) 242-0209

-----Original Message-----
From: opensim-dev-bounces at opensimulator.org [mailto:opensim-dev-bounces at opensimulator.org] On Behalf Of Diva Canto
Sent: Wednesday, August 19, 2015 8:24 PM
To: opensim-dev at opensimulator.org
Subject: [Opensim-dev] Paper available

May be of interest to some on this list:
http://arxiv.org/abs/1508.04465

Lots of references to mantises and assorted OpenSim things.
_______________________________________________
Opensim-dev mailing list
Opensim-dev at opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev

Classification: UNCLASSIFIED
Caveats: NONE