[Opensim-dev] Sim and Phys Frames per Second (FPS) Issues
Teravus Ovares
teravus at gmail.com
Sat Mar 14 19:40:42 UTC 2015
Sorry for the late response just looked at my email.
The multiplier originally had something to do with matching up the actual
physics frames with the times physics reported it's own frames since the
reporter was reporting at different times then the world was stepping and
the values were fluctuating improperly. In the end, I ended up setting it
to be 'similar' to the rate reported on Second Life mostly because when it
was implemented, I received repeated questions about it and caused more
questions then actually serving it's purpose.. so a 'relative' value
that someone could look at and feel comfortable with and be able to tell
when there was an issue was chosen so that the differences in the way the
physics engines and scene graph were run were not repeatedly and
erroneously targeted as the cause of bugs. I couldn't really alter the main
simulation scene loop frequency. The simulator simply couldn't handle it
and there were no process queues in the main simulation that we can skip
and process in the next frame if there was not enough time to process them
all in a frame.
The goal of the stats reporting was to help identify what the cause of a
slowdown was and people were seeing <11fps when they were used to 45. As
a result they often squarely put the blame on the physics layer when in
fact the issue was somewhere else. Early adopters were sensitive about
that number mostly because they are taught to look at it as a Simulator
Administrator/Estate manager in SL.
It sounds like since then, someone has done the physics step 5 times, which
is good. I'm not sure it would have supported that in the past.
Before you get the out the torches and pitch forks(Your 'curious' comment),
take a look at the following letter from 2009 and understand that you are
not the only proponent of using stats to try and identify the bottlenecks.
------------------------------------
Teravus Ovares <teravus at gmail.com>
11/27/09
to opensim-dev
Hey there,
A while back, we had somewhat reasonable statistics being generated and
presented to the client. They were not always accurate, but based on
what I saw, I could, pretty much pin certain parts of the simulator as the
limiting factor during load tests.
I'd say, the number 1 reason that they were semi-accurate and not
accurate.. in the past.. is because nobody ever thought about
instrumentation during the functionality design. It was always 'tacked
on later'. One example of this.. is the current AssetCache
implementation. There's no way, currently, to know, at a glance.. how
many external requests it has open. Additionally, it will be extremely
difficult to put one in because of the way the objects are designed and
accessed. To put one in, an event needs to be added to the IAssetService
interface and each AssetCache implementation will need an interlocked int
to count how many gets and puts it currently has open to the external data
source as well as it's own event calling schedule. Then, the
IAssetService property in Scene, (AssetService) will need an event
handler.. which updates the values in SimStatsReporter in Scene
(StatsReporter). This idea of external access resource instrumentation
should really have been built in to the design of the AssetService.
This last recent load test, there were no real statistics that I could use
to determine what the limiting factor was. Time Dilation was pegged at
1.0.. even when the simulator was obviously struggling. Total Frame
time (MS) was -50ms even when the simulation MS was 850ms and the Physics
ms was 250ms, so the inconsistencies made it impossible to know what part
of the simulator was struggling. Agent Updates were erratic.. sometimes
high..
sometimes low when the simulator was fine and when it was struggling.
Pending Uploads and Downloads were always 0, so there was no way to know
how well the simulator was downloading and uploading assets to and from the
grid. Packet stats were non-existant, so there was no way to know how
well the UDP handlers were faring under the load. When it crashed, it
crashed with a mono based stack trace which pointed to out of memory
errors, so the only way that you could, scientifically, find out what the
issue is.. is to run a load test under a memory profiler. We know,
that running a public load test under a memory profiler is quite
impractical.
To make something better, I need to know two things, where it is, and where
I want it to be. How can we make OpenSimulator better if we don't have
statistics that point to where we are currently?
On that note, I propose that, when designing objects for functionality in
OpenSimulator, that we also consider if the objects should be instrumented
and, what would be the best way to go about instrumenting the objects. We
should incorporate instrumentation into the design of the objects. Some
of that instrumentation is appropriate for a client to see, some of it
might not be. Consider that, many of them should be client facing and be
included in the SimStats that get sent to the client.. so that we can
have a reasonable idea of what's going on with a simulator at a glance.
Also, in the design of the instrumentation, we make sure that the
instrumentation is accurate and
lightweight.
The load test went reasonably... but, we didn't get half of the
information on the simulator that we needed to be able to improve it.
Please comment :) I look forward to hearing your responses.
Regards
Teravus
--------------------------------------------------------------------
That said, changing it now, I would be careful of the code that does
time dilation calculation. Time Dilation is 'supposed to be' a factor
calculated by dividing the physics scene time passed by real time passed.
This factor guides interpolation through velocity multiplication (Example:
Time Dilation:0.5, Linear Velocity: <1,1,1>m/s. would cause the viewer to
move the object <1*0.5,1*0.5,1*0.5>m/s until it gets a new update about the
object's position from the server) telling the viewer what to do with
moving objects between the times that it gets updates on the actual
positions of the objects. If you cause the Time Dilation calculation to
change to a more inaccurate result then be aware, objects will rubber-band
and stutter more often. Some of the time, objects will not visibly come to
a stop and fly off forever. It should be easy to test. Make a 30 0.5m
cubes and stack them up in 3 groups of ten 2x2x2 with 2 on the top with
each group about 2 meters above the ground. Make them physical and gauge
the amount of rubber banding as they bobble around. If it's worse after
then change then before the change then you have altered the calculation.
I use the term 'more inaccurate result' mostly because I'm not sure that
OpenSimulator is doing that calculation right as it is now. At one point
it was set to 1.0 regardless of what was happening so they didn't have to
think about it while working on things... and then someone different set
it to the scene graph time instead of the physics time which may or may not
be better then 1.0 all the time depending on how the updates actually get
out to the viewers.
If you wanted to be really sure, you could artificially add time to the
main physics loop and verify that rubber banding is minimal and that time
dilation goes down to reduce rubber banding.
Hopefully this was a helpful trip around OpenSimulator memory lane.
Teravus
On Tue, Mar 3, 2015 at 6:55 AM, Diva Canto <diva at metaverseink.com> wrote:
> Again, for people interested in learning more about these design issues,
> and the tradeoffs of different choices, I highly recommend this tutorial
> and associated code:
> http://buildnewgames.com/real-time-multiplayer/
>
> It's a wonderful little example that explains how interactive multi-user
> client-server simulations work. 100x simpler than OpenSim, but essentially
> the same concepts. It doesn't explain time dilation, but it's not hard to
> see where that comes in: in varying the time window of the physics loop.
> The update (simulation) loop is mainly about notifying clients about
> changes in the state of the simulation. We want the physics simulation to
> be high fidelity (meaning high loop rate), and the client updates to be as
> low as possible (low update loop rate).
>
> On 3/3/2015 2:09 AM, Tom Bess wrote:
>
> Thanks. You have clarified much. I realised the rift is viewer control but
> was not so clear about the impact/relationship between viewer and server
> frame rates and the interpolation needed.
>
> Thanks again.
>
> Tom Willans BSc(Hons) MBCS CITP
> Chartered IT Professional
>
> Managing Director Bessacarr Publications Ltd
> +44 (0)121 288 0281
> email: tom.willans at bessacarr.com
> skype: tom.willans
> Second Life and OSGrid: Tom Tiros
>
> Sent from my mobile
>
>
> On 3 Mar 2015, at 09:08, Dahlia Trimble <dahliatrimble at gmail.com> wrote:
>
> If my memory is correct, SL sims run at a default of 45
> frames/second. OpenSimulator runs at 11. I'm not certain exactly why 11 was
> chosen but I do know that increasing it increases the amount of work the
> simulator must do. E.g., if you go from 11 to 45 you quadruple the work the
> simulator must do
>
> The only signal I'm aware of that affects the viewer is "time dilation".
> This value varies from 0.0 to 1.0 where 1.0 indicates the simulator is
> running at full speed. If for some reason the physics engine cannot keep up
> with the simulation it can signal the viewers to slow movement. This value
> is sent with many UDP packets that are involved with the scene and moving
> objects, so the viewers will know this value as timely as possible.
>
> Running the simulation at 11 fps shoudl *not* affect devices such as the
> Rift as the camera is controlled entirely viewer side except under special
> circumstances such as a user surrendering camera control to a script. The
> viewer frame rate is *entirely independent* of the simulator frame rate and
> will run as fast as the hardware allows if configured to do so.
>
> Choosing an appropriate simulation frame rate would involve weighing the
> tradeoffs between simulation workload and responsiveness to user controls
> such as those that control avatar position movement. Bear in mind that
> there are also network delays involved; a simulation running at 11
> frames/second will respond to a user control every 89 ms but there will
> also be round-trip network delays which may typically be 100-200 ms but as
> high as 1.5 seconds or more. Increashing the simulation frame rate will
> likely not produce a perceivable difference in performance but will
> *significantly reduce* the capacity of the simulator to do things like host
> more avatars and scripted objects.
>
> Also bear in mind that much of the code assumes 11 frames/second and
> raising that may create motion-related and other bugs that may not be
> apparent until much later. OpenSimulator is tuned to function properly at
> 11 frames/second.
>
> To recap: the user camera is for the most part entirely controlled within
> the viewer and is unaffected by simulation frame rate. As such devices such
> as the Rift which manipulate the camera will not be affected.
>
> On Tue, Mar 3, 2015 at 12:09 AM, Tom Bess <tom.willans at bessacarr.com>
> wrote:
>
>>
>> I am reporting a comparison between sl and opensim and did not realise
>> this. Does sl run at a true 55fps? If so why bother? Presumably the viewer
>> needs 55fps sent to it to get its calculations correct as at 11fps opensim
>> does the same at sl albeit in larger steps.
>>
>> Would a faster fps improve the accuracy of devices such as the rift by
>> having to interpolate over a shorter period of time? Admittedly I suspect
>> viewer rendering needs to be improved as well as this aspect is holding my
>> experience back.
>>
>> I understand that other aspects may assume that 11fps is a fixed
>> constant and not allow for this to change presumably that can be changed
>> but you guys know more here.
>>
>> Thanks for the guide btw.
>>
>> Tom Willans BSc(Hons) MBCS CITP
>> Chartered IT Professional
>>
>> Managing Director Bessacarr Publications Ltd
>> +44 (0)121 288 0281 <%2B44%20%280%29121%20288%200281>
>> email: tom.willans at bessacarr.com
>> skype: tom.willans
>> Second Life and OSGrid: Tom Tiros
>>
>> Sent from my mobile
>>
>>
>> On 3 Mar 2015, at 05:18, Sean M <mondesire.sean at gmail.com> wrote:
>>
>> Thanks Justin, Dahlia, Michael, and everyone. I now have a better
>> understanding of the way FPS is calculated and of the correction factor.
>>
>> -Sean M.
>>
>> On Monday, March 2, 2015, Justin Clark-Casey <jjustincc at googlemail.com>
>> wrote:
>>
>>> I think this has already been said but just to summarize.
>>>
>>> * The 11 fps are the number of scene frames processed - opportunities
>>> where avatars may be notified about changes in the scene.
>>>
>>> * Each of these scene frames is associated with a physics process where
>>> 5 physics frames are processed in each frame. Hence 11 * 5 = 55 fps.
>>>
>>> * Why this number? Others may know better but my guess is that it's
>>> related to the frequency of updates expected by the viewer. Teravus may
>>> well know more if he's still around.
>>>
>>> * Changing m_reportedFpsCorrectionFactor will do nothing except change
>>> the server FPS stat.
>>>
>>> * Changing MinFrameTime will change the number of scene frames. From my
>>> work in the past, you would also need to adjust other parameters like
>>> physics frames to stop things going haywire (this was with ODE, Bullet
>>> might work differently). I expect you also wouldn't gain much if anything
>>> in scene fidelity.
>>>
>>> On 02/03/15 18:23, Sean M wrote:
>>>
>>>> Greetings,
>>>>
>>>> We at the MOSES project have noticed Simulation and Physics frames per
>>>> second (FPS) have a few issues that we are trying
>>>> to resolve. The issues are producing suspicious performance statistics
>>>> for the analysis of the current version of
>>>> OpenSim that we are running.
>>>>
>>>> First, there is a correction factor (m_reportedFpsCorrectionFactor)
>>>> that the raw SimFPS is multiplied by. The comment in
>>>> the following line is a bit curious because it indicates that the FPS
>>>> is artificially inflated to "lie" about the actual
>>>> FPS being so low:
>>>>
>>>> OpenSim/Region/Framework/__Scenes/SimStatsReporter.cs: Line 317
>>>> // We're going to lie about the FPS because we've been lying since
>>>> 2008. The actual FPS is currently
>>>> // locked at a maximum of 11. Maybe at some point this can change so
>>>> that we're not lying.
>>>> int reportedFPS = (int)(m_fps * m_reportedFpsCorrectionFactor);
>>>>
>>>> Also, lines 174 and 227 mention the use of this correction factor.
>>>>
>>>> Second, this multiplier also comes into play in the Scene where there
>>>> is a MinFrameTime, which seems to be the minimum
>>>> reported amount of time to process a frame:
>>>> OpenSim/Region/Framework/__Scenes/Scene.cs:Line 723
>>>>
>>>> Both of these variables, the correction factor and MinFrameTime, are
>>>> concerning from a statistics view point as they are
>>>> generating skewed and massaged numbers; therefore, I have a few
>>>> questions:
>>>>
>>>> 1) Is it commonly known that Sim and Phy FPSs are inflated to maintain
>>>> the "lie"? And if so, will it be corrected to be
>>>> an accurate reporting of processed frames per second?
>>>>
>>>> 2) What exactly are the definitions for OpenSim's Simulation (Sim) FPS,
>>>> Physics (Phy) FPS and a frame (I have found
>>>> conflicting and vague definitions on the wiki)?
>>>>
>>>> 3) What are the known performance consequences of setting the
>>>> m_reportedFpsCorrectionFactor to 1 and MinFrameTime to 0?
>>>>
>>>> Thanks,
>>>> Sean M.
>>>>
>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at opensimulator.org
>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>>
>>>>
>>>
>>> --
>>> Justin Clark-Casey (justincc)
>>> OSVW Consulting
>>> http://justincc.org
>>> http://twitter.com/justincc
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at opensimulator.org
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing listOpensim-dev at opensimulator.orghttp://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20150314/4df917d6/attachment-0001.html>
More information about the Opensim-dev
mailing list