Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007568opensim[REGION] Script Functionspublic2015-05-14 09:592015-05-22 14:32
ReporterMagnuz 
Assigned Tonebadon 
PrioritynormalSeverityminorReproducibilitysometimes
StatusclosedResolutionfixed 
PlatformIntel Core i7OSUbuntuOS Version14.04
Product Versionmaster (dev code) 
Target VersionFixed in Versionmaster (dev code) 
Summary0007568: llCastRay V3 needs throttling
DescriptionllCastRay V3 could overload or even crash a sim owing to heavy geometry mesh generation. To make it safe to use, the function needs throttling, not to load the sim excessively.
Steps To ReproduceHave several scripted items making llCastRay requests in an object dense sim as fast as possible. Notice sim performance degrading and maybe crash. (I never managed to crash the sim myself though.)
Additional InformationMono 3.12.1
TagsNo tags attached.
Git Revision or version number0.8.2 dev
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
EnvironmentMono / Linux32
Mono VersionOther
ViewerSingularity 1.8.6 OS X
Attached Filespatch file icon 0001-Throttle-llCastRay-V3.patch [^] (8,057 bytes) 2015-05-14 09:59 [Show Content]

- Relationships
related to 0007546closednebadon llCastRay only partly implemented 
related to 0007582closednebadon llCastRay v3 very poor performance (High CPU) 
related to 0007575closednebadon llCastRay V3 performance improved by mesh caching 

-  Notes
(0028374)
Magnuz (reporter)
2015-05-14 10:01

Patch to throttle llCastRay V3, similar but not identical to the throttling in SL:

Throttles are based on a rolling interval rather than a number of frames (4) in SL.

Default limits are quite different compared to SL owing to a slower implementation and varying server performances, compared to the rather uniform SL high end servers.

Throttle parameters are configurable to fine-tune performance depending on server performance and scene.

The suggested defaults are sane but not 100% safe. E.g. 15-20 avatars firing cast ray weapons at full throttle simultaneously should be able to grind the sim to a halt.
(0028375)
Magnuz (reporter)
2015-05-14 10:07

To use or test llCastRay V3, add and adjust the following to OpenSim.ini (from OpenSimDefaults.ini):

[LL-Functions]
    ; ...

    ; Use llCastRay V3 if true
    ; Gives better accuracy and can be faster on some servers, but slower on others,
    ; compared to previous version of llCastRay
    ; Generates geometry meshes and can therefore use much system resources
    UseLlCastRayV3 = true

    ; Accepted calculation precision error in calculations in llCastRay V3
    FloatToleranceInLlCastRay = 0.000001

    ; Accepted distance difference between duplicate hits in llCastRay V3
    FloatTolerance2InLlCastRay = 0.0001

    ; Detail level when rendering prims in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    PrimDetailLevelInLlCastRay = 1

    ; Detail level when rendering sculpts in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    SculptDetailLevelInLlCastRay = 1

    ; Detail level when rendering meshes in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    MeshDetailLevelInLlCastRay = 3

    ; Detail level when rendering avatar capsules in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    AvatarDetailLevelInLlCastRay = 1

    ; Maximum number of returned hits from llCastRay V3
    MaxHitsInLlCastRay = 16

    ; Maximum number of returned hits per prim from llCastRay V3
    MaxHitsPerPrimInLlCastRay = 16

    ; Maximum number of returned hits per object from llCastRay V3
    MaxHitsPerObjectInLlCastRay = 16

    ; Report ray intersections with surfaces on exits from a prim as hits in llCastRay V3 if true
    DetectExitHitsInLlCastRay = false

    ; Filter on parts instead of groups in llCastRay V3 if true
    FilterPartsInLlCastRay = false

    ; Detect attachments in llCastRay V3 if true
    DoAttachmentsInLlCastRay = false

    ; Throttle period length in ms before which all old llCastRay use is discarded in llCastRay V3
    ; The sum of AvailableTimeInMsPerRegionInLlCastRay and all AvailableTimeInMsPerAvatarInLlCastRay should not exceed this
    ThrottleTimeInMsInLlCastRay = 200

    ; Available time in ms for llCastRay per throttle period and 65536 m2 land area in llCastRay V3
    AvailableTimeInMsPerRegionInLlCastRay = 40

    ; Available time in ms for llCastRay per throttle period and avatar when script in attachment or vehicle in llCastRay V3
    AvailableTimeInMsPerAvatarInLlCastRay = 10

    ; Required available time in ms left to perform a new llCastRay in llCastRay V3
    RequiredAvailableTimeInMsInLlCastRay = 2

    ; Maximum available time in ms possible in llCastRay V3, not to get too high values with varregions
    MaximumAvailableTimeInMsInLlCastRay = 40
(0028418)
Magnuz (reporter)
2015-05-21 22:28
edited on: 2015-05-21 22:29

nebadon load tested llCastRay V3 and reported a CPU load of 170-180% (per core) in Mantis 7582, but it wasn't clear if that was with or without this throttling, and that Mantis was closed before I could ask there.

Testing nebadon's rig with this throttling keeps CPU load below 20%, as expected (40 ms castray time per 200 ms throttle period = 40/200 = 0.2 = 20%), both on the high-end server and an ordinary laptop I use myself for testing. Without this throttling, I could push the high-end server to a CPU load of almost 400% (per core) with another load test rig, but with this throttling, that rig too keeps the CPU load below 20%.

(0028419)
nebadon (administrator)
2015-05-21 22:31

ok let me test with the patches, though I think I already have those patches in this simulator and I noticed no change.
(0028420)
nebadon (administrator)
2015-05-21 22:48

ok in my confusion of testing I guess I didnt have them applied on my last test, I saw a large reduction as well down to about 67% cpu from 180%, good stuff both patches are applied
(0028421)
nebadon (administrator)
2015-05-21 22:48

patch applied : http://opensimulator.org/viewgit/?a=commit&p=opensim&h=7d26815d0e615b60c708768e8aab19ffd4d118fb [^]
(0028434)
Magnuz (reporter)
2015-05-22 14:32

Verified code and function, so closing.

The difference in CPU load appears to be because the 67% load is with MinTimerPeriod reduced from the standard 0.5 s to 0.05 s, while the 17-19% is with MinTimerPeriod at the standard 0.5 s.

This shouldn't make a difference in the throttling result, but it does, owing to calls stacking up waiting for available castray time. This becomes more prominent when more calls are made, and nebadon's test rig can theoretically make 240 calls per second with MinTimerPeriod 0.05 s, but only 24 calls per second with MinTimerPeriod 0.5 s.

When castray time becomes available, all stacked calls start almost simultaneously, without realizing other threads already will use up the time available, so the time available becomes "over-used".

A better throttling solution that can handle many parallel calls is needed, but the present one at least improves the situation some.

- Issue History
Date Modified Username Field Change
2015-05-14 09:59 Magnuz New Issue
2015-05-14 09:59 Magnuz File Added: 0001-Throttle-llCastRay-V3.patch
2015-05-14 10:01 Magnuz Note Added: 0028374
2015-05-14 10:01 Magnuz Status new => patch included
2015-05-14 10:02 Magnuz Relationship added related to 0007546
2015-05-14 10:07 Magnuz Note Added: 0028375
2015-05-16 14:09 Magnuz Relationship added related to 0007575
2015-05-21 22:28 Magnuz Note Added: 0028418
2015-05-21 22:28 Magnuz Relationship added related to 0007582
2015-05-21 22:29 Magnuz Note Edited: 0028418 View Revisions
2015-05-21 22:31 nebadon Note Added: 0028419
2015-05-21 22:48 nebadon Note Added: 0028420
2015-05-21 22:48 nebadon Note Added: 0028421
2015-05-21 22:48 nebadon Status patch included => resolved
2015-05-21 22:48 nebadon Resolution open => fixed
2015-05-21 22:48 nebadon Assigned To => nebadon
2015-05-22 14:32 Magnuz Note Added: 0028434
2015-05-22 14:32 Magnuz Status resolved => closed
2015-05-22 14:32 Magnuz Fixed in Version => master (dev code)


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker