MantisBT - opensim
View Issue Details
0007568opensim[REGION] Script Functionspublic2015-05-14 09:592015-05-22 14:32
Magnuz 
nebadon 
normalminorsometimes
closedfixed 
Intel Core i7Ubuntu14.04
master (dev code) 
master (dev code) 
0.8.2 dev
Grid (1 Region per Sim)
BulletSim
Mono / Linux32
Other
Singularity 1.8.6 OS X
0007568: llCastRay V3 needs throttling
llCastRay V3 could overload or even crash a sim owing to heavy geometry mesh generation. To make it safe to use, the function needs throttling, not to load the sim excessively.
Have several scripted items making llCastRay requests in an object dense sim as fast as possible. Notice sim performance degrading and maybe crash. (I never managed to crash the sim myself though.)
Mono 3.12.1
No tags attached.
related to 0007546closed nebadon llCastRay only partly implemented 
related to 0007582closed nebadon llCastRay v3 very poor performance (High CPU) 
related to 0007575closed nebadon llCastRay V3 performance improved by mesh caching 
patch 0001-Throttle-llCastRay-V3.patch (8,057) 2015-05-14 09:59
http://opensimulator.org/mantis/file_download.php?file_id=4201&type=bug
Issue History
2015-05-14 09:59MagnuzNew Issue
2015-05-14 09:59MagnuzFile Added: 0001-Throttle-llCastRay-V3.patch
2015-05-14 10:01MagnuzNote Added: 0028374
2015-05-14 10:01MagnuzStatusnew => patch included
2015-05-14 10:02MagnuzRelationship addedrelated to 0007546
2015-05-14 10:07MagnuzNote Added: 0028375
2015-05-16 14:09MagnuzRelationship addedrelated to 0007575
2015-05-21 22:28MagnuzNote Added: 0028418
2015-05-21 22:28MagnuzRelationship addedrelated to 0007582
2015-05-21 22:29MagnuzNote Edited: 0028418bug_revision_view_page.php?bugnote_id=28418#r4195
2015-05-21 22:31nebadonNote Added: 0028419
2015-05-21 22:48nebadonNote Added: 0028420
2015-05-21 22:48nebadonNote Added: 0028421
2015-05-21 22:48nebadonStatuspatch included => resolved
2015-05-21 22:48nebadonResolutionopen => fixed
2015-05-21 22:48nebadonAssigned To => nebadon
2015-05-22 14:32MagnuzNote Added: 0028434
2015-05-22 14:32MagnuzStatusresolved => closed
2015-05-22 14:32MagnuzFixed in Version => master (dev code)

Notes
(0028374)
Magnuz   
2015-05-14 10:01   
Patch to throttle llCastRay V3, similar but not identical to the throttling in SL:

Throttles are based on a rolling interval rather than a number of frames (4) in SL.

Default limits are quite different compared to SL owing to a slower implementation and varying server performances, compared to the rather uniform SL high end servers.

Throttle parameters are configurable to fine-tune performance depending on server performance and scene.

The suggested defaults are sane but not 100% safe. E.g. 15-20 avatars firing cast ray weapons at full throttle simultaneously should be able to grind the sim to a halt.
(0028375)
Magnuz   
2015-05-14 10:07   
To use or test llCastRay V3, add and adjust the following to OpenSim.ini (from OpenSimDefaults.ini):

[LL-Functions]
    ; ...

    ; Use llCastRay V3 if true
    ; Gives better accuracy and can be faster on some servers, but slower on others,
    ; compared to previous version of llCastRay
    ; Generates geometry meshes and can therefore use much system resources
    UseLlCastRayV3 = true

    ; Accepted calculation precision error in calculations in llCastRay V3
    FloatToleranceInLlCastRay = 0.000001

    ; Accepted distance difference between duplicate hits in llCastRay V3
    FloatTolerance2InLlCastRay = 0.0001

    ; Detail level when rendering prims in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    PrimDetailLevelInLlCastRay = 1

    ; Detail level when rendering sculpts in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    SculptDetailLevelInLlCastRay = 1

    ; Detail level when rendering meshes in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    MeshDetailLevelInLlCastRay = 3

    ; Detail level when rendering avatar capsules in llCastRay V3
    ; 0 = Low, 1 = Medium, 2 = High, 3 = Highest, higer level gives better accuracy but slower call
    AvatarDetailLevelInLlCastRay = 1

    ; Maximum number of returned hits from llCastRay V3
    MaxHitsInLlCastRay = 16

    ; Maximum number of returned hits per prim from llCastRay V3
    MaxHitsPerPrimInLlCastRay = 16

    ; Maximum number of returned hits per object from llCastRay V3
    MaxHitsPerObjectInLlCastRay = 16

    ; Report ray intersections with surfaces on exits from a prim as hits in llCastRay V3 if true
    DetectExitHitsInLlCastRay = false

    ; Filter on parts instead of groups in llCastRay V3 if true
    FilterPartsInLlCastRay = false

    ; Detect attachments in llCastRay V3 if true
    DoAttachmentsInLlCastRay = false

    ; Throttle period length in ms before which all old llCastRay use is discarded in llCastRay V3
    ; The sum of AvailableTimeInMsPerRegionInLlCastRay and all AvailableTimeInMsPerAvatarInLlCastRay should not exceed this
    ThrottleTimeInMsInLlCastRay = 200

    ; Available time in ms for llCastRay per throttle period and 65536 m2 land area in llCastRay V3
    AvailableTimeInMsPerRegionInLlCastRay = 40

    ; Available time in ms for llCastRay per throttle period and avatar when script in attachment or vehicle in llCastRay V3
    AvailableTimeInMsPerAvatarInLlCastRay = 10

    ; Required available time in ms left to perform a new llCastRay in llCastRay V3
    RequiredAvailableTimeInMsInLlCastRay = 2

    ; Maximum available time in ms possible in llCastRay V3, not to get too high values with varregions
    MaximumAvailableTimeInMsInLlCastRay = 40
(0028418)
Magnuz   
2015-05-21 22:28   
(edited on: 2015-05-21 22:29)
nebadon load tested llCastRay V3 and reported a CPU load of 170-180% (per core) in Mantis 7582, but it wasn't clear if that was with or without this throttling, and that Mantis was closed before I could ask there.

Testing nebadon's rig with this throttling keeps CPU load below 20%, as expected (40 ms castray time per 200 ms throttle period = 40/200 = 0.2 = 20%), both on the high-end server and an ordinary laptop I use myself for testing. Without this throttling, I could push the high-end server to a CPU load of almost 400% (per core) with another load test rig, but with this throttling, that rig too keeps the CPU load below 20%.

(0028419)
nebadon   
2015-05-21 22:31   
ok let me test with the patches, though I think I already have those patches in this simulator and I noticed no change.
(0028420)
nebadon   
2015-05-21 22:48   
ok in my confusion of testing I guess I didnt have them applied on my last test, I saw a large reduction as well down to about 67% cpu from 180%, good stuff both patches are applied
(0028421)
nebadon   
2015-05-21 22:48   
patch applied : http://opensimulator.org/viewgit/?a=commit&p=opensim&h=7d26815d0e615b60c708768e8aab19ffd4d118fb [^]
(0028434)
Magnuz   
2015-05-22 14:32   
Verified code and function, so closing.

The difference in CPU load appears to be because the 67% load is with MinTimerPeriod reduced from the standard 0.5 s to 0.05 s, while the 17-19% is with MinTimerPeriod at the standard 0.5 s.

This shouldn't make a difference in the throttling result, but it does, owing to calls stacking up waiting for available castray time. This becomes more prominent when more calls are made, and nebadon's test rig can theoretically make 240 calls per second with MinTimerPeriod 0.05 s, but only 24 calls per second with MinTimerPeriod 0.5 s.

When castray time becomes available, all stacked calls start almost simultaneously, without realizing other threads already will use up the time available, so the time available becomes "over-used".

A better throttling solution that can handle many parallel calls is needed, but the present one at least improves the situation some.