NINJA Physics: Networked INteractive Jointed Assemblies for OpenSim

This project proposes a new physics extension called NINJA Physics. The NINJA Physics extension enhances the ODE physics in OpenSim with joints: movable linkages between prims. This will allow wheeled vehicles, ragdolls, and mechanical/robotic assemblies to be interactively built within the standard viewer and controllable through scripting.

A demonstration of the idea can be seen here: http://jp.youtube.com/watch?v=nirTXPO-opE

NINJA stands for a Networked INteractive Jointed Assembly. "Jointed Assembly" refers to an articulated rigid body. The "INteractive" aspect refers to interactive editing in 3D space of jointed assemblies using the viewer and intuitive 3D placement of joint objects as normal prims. The "Networked" aspect refers to many users simultaneously interacting with jointed assemblies in a networked virtual world.

The core ODE physics engine already provides support for joints; it is OpenSim and the viewer that lack support. This project therefore aims to extend the OpenSim physics interface to access the ODE joint functionality. Furthermore, this project must also provide a means of creating and editing joints within the standard viewer, which has no native support for joints. In the future, viewer changes and protocol changes can be considered for native joint support.

This new kind of physics, jointed physics, is separate from and complementary to the existing LSL script-based vehicles. Separate from NINJA Physics, other activity has recently accelerated on LSL script-based vehicles, with some water-based vehicles already working (e.g. sailboats). The new jointed/NINJA physics will allow more new kinds of vehicles not currently possible with scripting. Two example of new kinds of vehicles are cars with wheels that actually turn and rotate, and robots that walk by moving limbs.

Philosophy

The development of NINJA Physics follows these principles.

1. Break nothing. When turned off, NINJA Physics should have no observable effect on simulator behavior. When turned on, existing physics behavior without joints should be exactly the same as before. Additionally, new physics behavior with joints should be available to the user.
2. Standard viewer compatibility. Editing of joints must be possible using only the affordances of the standard viewer. This means we must enter new information (joint information) in an existing GUI not designed for that purpose. We do this by repurposing existing prim editor fields (Name and Description) and using a special naming convention to specify joints. The requirment of standard viewer compatibility conflicts slightly with the requirement of breaking nothing; by using a special naming convention, it is possible that existing regions might already use that naming convention, which would lead to altered behavior. In practice, this will likely not be a great concern.
3. Code readability. The first version of the NINJA Physics patch is implemented in as straightforward a way as possible to allow understanding and critique of the basic approach. At this stage, no overly-clever optimizations and only necessary abstractions have been applied.
4. Generalizability. The first version of NINJA Physics is implemented using ODE Physics, but the same implementation should be usable for any physics engine that supports joints.

User documentation

Quick start

To try out NINJA Physics, follow these steps.

1. Apply the patch ninja-allparts.patch to your OpenSim source tree and compile. Available at http://opensimulator.org/mantis/view.php?id=2874 I am happy to report that NINJA Physics is included in trunk SVN as of r7875, so update your repository and it should be available. Important note if NINJA Physics is not working for you: The NINJA Physics in SVN r7875 should work but requires C# 3.0. The NINJA Physics in r7878 does not work. If you use SVN r7878 or above, please apply the additional patch in Mantis #2928 at http://opensimulator.org/mantis/view.php?id=2928
2. Copy demo-playground.tgz to your opensim/bin directory. Available at http://forge.opensimulator.org/gf/download/frsrelease/142/304/demo-playground.tgz
3. Edit OpenSim.ini to run in standalone mode with ODE physics. Set use_NINJA_physics_joints=true.
4. Start OpenSim.exe. In the region console type load-oar demo-playground.tgz to load demo data.
5. Login to the region and navigate to (128,128). Select all prims in the swing object (Ctrl+3, mouse drag). If you have a powerful CPU, you can also select everything in the region, but note that processing joints takes some CPU power, so you may want to select a smaller group of objects to start.
6. In the object editing window, click on the "Physical" checkbox in the "Object" tab to turn the selection physical, then press ESC to exit selection mode. The prims and the joints go "live".
7. Click on the jointed assembly and drag it around. Also, the following commands spoken in chat can control the ragdoll: "arise", "kneel", "die".

Definitions

A joint is a constraint on the motion of rigid bodies. Currently we consider two types of joints: a ball-and-socket joint and a hinge joint. A ball-and-socket joint is similar to the joint connecting your upper arm to your shoulder; the joint constrains the position of the two bodies, but allows free rotation. A hinge joint is similar to a door hinge: it constrains the position of two bodies but allows rotation about only one axis, the hinge axis. By using joints, we can create complex assemblies of prims in OpenSim that move with realistically-constrained motion.

A joint proxy object is a prim created in the viewer and existing within the region to represent a joint. Its position, relative to the prims it affects, defines the position around which the prims movement will be constrained. For hinge joints, the orientation of the joint proxy object defines the hinge axis; specifically, the x-axis of the proxy object is defined as the axis about which the connected prims will be allowed to rotate.

A physics engine joint is the data structure within the physics engine that actually constrains the behavior of the rigid bodies as simulated in the simulator. The physics engine joint is initially specified by and later visualized by the joint proxy object.

Enabling NINJA Physics in OpenSim.ini

In OpenSim.ini, set "physics = OpenDynamicsEngine" and "use_NINJA_physics_joints = true".

Creating a ball-and-socket joint

Create two box prims side-by-side with space in between them. Leave them non-physical for now. Give each one a unique name (right click, edit, General tab, Name). Example names are "box1" and "box2". Leave the dimensions at (0.5, 0.5, 0.5). These 2 prims will be connected by a ball joint.

Create a small sphere. Make its dimensions something like (0.1, 0.1, 0.1). This prim represents a ball-and-socket joint. Position the sphere between the 2 boxes. Assign the small sphere the name of "balljoint-uniquename", where "uniquename" is a unique name not used by any other prim. Give the sphere a Description of "box1 box2", which are the names of the 2 prims to be connected, separated by a space.

Close all windows in the viewer. Press Ctrl+3 to enter edit mode. Drag the mouse to highlight the 2 boxes to be connected and the small sphere.

In the object editing window, click on the "Physical" checkbox in the "Object" tab.

Drag the boxes around and notice they remain connected by a joint.

Creating a hinge joint

A similar procedure is used for creating bodies connected with hinge joints, with the following differces:

The joint object should be named "hingejoint-uniquename", where uniquename is a unique name not used elsewhere in the region.

The joint object's position determines the anchor of the hinge.

The joint object's X axis orientation determines the axis of rotation of the hinge. The easiest way to visualize this is to create a prim whose size is elongated along its x axis; set the object size directly in the object editor to something like (1.0, 0.1, 0.1).

Hinges can be used for simulating wheels. Connected bodies can rotate freely around the hinge, like a wheel around an axle.

Activating and deactivating bodies and joints

Setting a joint proxy object to "physical" causes creation of that joint in the physics engine. Setting the joint proxy object to non-physical, or deleting the joint proxy object, or taking the joint proxy object causes deletion of the corresponding joint in the physics engine. (This is why each joint must have a unique name, so that the physics engine knows which joint is being referred to.) Deleting joints in an active jointed assembly will cause the connected bodies to disconnect. Deleting, deactivating, or taking a prim in an active jointed assembly will cause all joints connected to that prim to deactivate.

Troubleshooting

• Joint error messages displayed in-world like script compiler errors. If, for example, you create a hinge joint (by naming a prim "hingejoint-uniquename") but specify non-existent or non-physical body names in the Description field, the joint will not be able to be created and a warning icon appears in-world on top of the joint. Click on the warning icon to see the error message. Deactivate, correct, and re-activate the joint.
• Duplicate joint names cause problems. If you keep getting errors with a joint, it is possible that some other prim somewhere in the region has the same name. Try changing the name of the joint proxy object. You should only change the name when the joint is deactivated.
• Duplicate prim names cause problems. If you are repeatedly unable to connect a prim with a joint, it is possible that some other prim somewhere in the region has the same name, and that the joint is connecting to that prim instead of the one you want. Try changing the name of the prim being connected. You should only change the name when the prim is deactivated.

Tips for effectively using joints

• The joint proxy object itself does not collide with anything and has no geometric representation in the physics scene. While the joint proxy is active (physical), you cannot directly physically manipulate the joint proxy (by dragging it in the viewer or through script physics). When active, its position always automatically follows the position of the jointed bodies.
• Prims connected by a joint do not collide with each other.
• Prims connected by a joint can overlap on top each other and on top of the joint proxy object, effectively hiding the joint proxy object.
• You can also make the joint proxy object very small or transparent to hide it.
• If you hide the joint proxy, it can be difficult to debug when things go wrong. Hide joints only after you've verified that the joints are created properly.
• Prims connected by joints should probably be simple shapes like boxes or spheres for simpler collision. Complex shapes with twists or cuts cause creation of concave meshes with sharp corners which may not collide smoothly (leading to jitter) or quickly. Unfortunately, it doesn't seem possible to use ODE capsules (often used for ragdoll limbs) for collision detection because there is no way to specify a "capsule" geometry for a prim in the viewer.
• To take a jointed assembly into inventory:
  • Select all
  • Deactivate
  • Link
  • Take
• To rez a jointed assembly from inventory:
  • Rez. Do not rez the same jointed assembly more than once or you will have prims and joints with duplicate names, which causes unexpected behavior as the joints don't know which of the duplicated prims they are supposed to connect.
  • Move to desired location
  • Unlink
  • Select all
  • Activate
• Friction, collision bounce, and collision filtering settings in OpenSim.ini can be tweaked for different effects.
• Workflow for creating a complex jointed assembly
  • Create a few prims
  • Create a few joints
  • Select them all
  • Keeping the selection, click "[x] Physical" to turn on physics
  • Keeping the selection, observe if joint error message icons appear
  • Keeping the selection, cllick "[ ] Physical" to turn off physics
  • Fix any joint error messages that occured (e.g. mistyped or duplicate prim names)
  • Repeat
  • When all joint error messages are fixed for the group of prims under consideration, continue to construct more of the assembly

Developer documentation

Patch structure

The patch for NINJA Physics is divided into 6 parts for better readability.

Part 1: data structures

Patch ninja-part1-joint_data_structures.patch declares the PhysicsJoint class and its subclass OdePhysicsJoint for storing joint information.

Part 2: abstract physics interface

Patch ninja-part2-physics_interfaces.patch alters PhysicsScene.cs with new methods for manipulating joints. The main idea is that a user of the physics engine can make joint requests or be notified of joint activity. Joint requests include requesting the physics engine to create or delete joints. Joint activity notification takes place through events that are fired whenever a joint is moved or a joint is deleted. These events allow the external scene to update scene information (position and orientation of joint proxy objects) to accurately reflect the state of the joints.

This part of the patch also modifies PhysicsActor.cs to contain the scene object part name and description. The physics engine links bodies together based by looking for physics actors whose names match the joint description.

Part 3: scene

Patch ninja-part3-scene.patch alters Scene.cs to allow NINJA Physics to be turned on/off. It also extends the behavior of DeleteSceneObject to delete the joint in the physics engine if the scene object corresponding to the joint (i.e. the joint proxy object) is deleted.

Also, this part of the patch alters SceneGraph.cs with callback handlers to respond to joint movement or joint deletion notifications from the physics engine. When a joint in the physics engine is moved/deactivated, its representation in the scene (the joint proxy object) is correspondingly moved/stopped from moving. Note that the joint has no direct representation (i.e. no body or geom) in the physics engine, so the physics engine has no knowledge of the scene graph's joint proxy object, necessitating this callback approach.

In v2 of this patch (now included in trunk), the joint callback handlers have been moved into Scene.cs.

Part 4: SceneObjectPart

Patch ninja-part4-sceneobjectpart patches SceneObjectPart.cs to detect if the SceneObjectPart is a joint (based on its name of "balljoint-*" or "hingejoint-*"). If it is a joint, then requests to make this SceneObjectPart physical/non-physical are translated into calls to the physics scene to request creation/deletion of joints.

Part 5: ODE joints implementation

Patch ninja-part5-ode.patch is the ODE implementation of NINJA Physics.

ODEPrim.cs is patched to remove an optimization that prevents updates from being sent when the object position is almost stationary. This prevents rotational-only motion from being sent to the client, such as a fan blade rotating on a hinge. In v2 of the patch (included in trunk) the optimization - which is actually a throttle that is necessary - has been re-instated, additionally adding a check for rotational-only motion to allow rotation updates to be transmitted even in the absence of translational motion.

OdePlugin.cs implements the creation and deletion of joints within ODE. Internal data structures keep track of associations between joints and bodies. Unprocessed joint requests (creation or deletion) are processed in the Simulate loop.

Joint deletion requests are handled immediately and the joint deleted callback is invoked to nofity the external scene of the joint change.

Joint creation requests are not handled immediately, because a joint can only be created if the bodies it is connecting already exist in the physics scene. Therefore, a joint creation request adds the new joint into a pending list. After processing of prim taints, a check is performed to see if any pending joints are still waiting for their respective bodies to appear in the physics engine. If such pending joints exist, and all of their bodies also exist, the joint is removed from the pending list and is actually created in ODE. Internal data structures also keep track of the association between bodies and the joints connected to that body.

Finally, whenever an physics actor (body) moves, it is checked if that body is connected by joints. If so, movement of the body implies movement of the joints connected to that body. Therefore, a joint moved callback is invoked to notify the external scene of the joint motion.

The DumpJointInfo function is useful for debugging.

Part 6: OpenSim.ini.example

Patch ninja-part6-ini.patch shows the OpenSim.ini option to control NINJA Physics.

Joint states

Joints pass through the following states during their lifetime:

• Requested creation. In this state, the joint is not yet processed by main physics simulation loop.
• Pending. In this state, the joint is processed by main physics loop but does not yet exist in the physics engine. It is waiting for its constituent bodies to exist in the physics engine.
• Active. In this state, the joint exists in ODE and affects the ODE simulation. Furthermore, motion of the joint generates a joint moved event that can be subscribed to and acted upon.
• Requesting deletion. In this state, the joint has requested to be deleted from the main physics simulation loop, but that request has not been processed by the main simulate loop. Therefore the joint is still being simulated.
• About to delete. In this state, the joint is about to be deleted and a joint deactivated event is sent to notify subscribers of the joint's impending death.
• Deleted. The joint no longer exists in ODE, does not affect the physical simulation, and generates no further events.

Limitations

This section describes some current limitations and shortcomings of the current implementation and areas of continuing and future work.

Naming limitations

• Each joint and each prim connected by a joint must have a unique name.
• Changes of joint names or prim names while the prim/joint is physical are not handled.
• When creating a joint, in the description field, use exactly one space between the names of the connected prims.

Linked or attached prims

• A linked prim cannot currently be part of a jointed assembly.
• Don't try to attach (to your avatar) prims that are part of a jointed assembly. This would however be a very cool feature in the future, but needs more research about how to implement it.

Timing issues

• Extremely rapid toggling between physical and non-physical states, with the final state being "physical", can sometimes incorrectly lead to the joint's last state being "non-physical".

Persistence of joints

• All jointed assemblies in the region should be deactivated before shutting down the region. Upon region restart, select and re-activate the jointed assemblies. However, this leads to gradual accumulation of joint error (see below).
• "Live" jointed assemblies are not properly restored on region shutdown/startup. In other words, if you activate a jointed assembly then, while the jointed assembly is still simulating, shutdown the region, upon region startup the recreated joints are not all perfectly reconstructed. It almost works, but some joint orientations are incorrect on region startup. This is a bug and will hopefully soon be fixed. Additionally, accumulation of joint error still occurs (see below).
• A region with deactivated jointed assemblies can be saved/loaded with save-oar/load-oar. However, trying to use save-oar/load-oar with live jointed assemblies that are actively simulating doesn't work due to threading/atomicity issues (see below).

To summarize:

• Deactivate jointed assemblies before persisting them.
• Expect gradual accumulation of joint error over time (see below).
• Periodically delete and re-rez the jointed assembly from inventory to correct joint error.

Atomicity issues

• Non-atomicity of viewer selection processing. If you select a group of non-physical prims and joints, and turn them all physical by clicking "physical" in the viewer, it is not guaranteed that the entire selected group will be turned physical in one tick of the simulation. It is therefore possible that part of the assembly will go "live" and start moving before all of the joints in the assembly have been constructed. This will lead to joint positions different than the original static specification of the joints.
• Non-atomicity of load-oar. Due to multi-threading, physics simulation continues while a load-oar is executing. This again means that parts of an assembly will go "live" and start moving before all of the joints in the assembly have been constructed.

Accumulation of joint error

Repeatedly deactivating and reactivating a jointed assembly will lead to non-reversible accumulation of error in the joints. The joint proxy object saves the "position" of the joint but this information is not complete information about the joint. In particular, it does not contain any information about the joint error. For instance, imagine a ball and socket joint where, due to extreme forces acting on the parts, the ball and socket have been forced apart, leading to joint error. Currently, the joint proxy object only saves the joint position relative to one body, which is analagous to saving only the position of the ball but not the position of the socket. Upon deactivating and reactivating a joint with joint error, the current body positions are assumed to be 0% error, which means that the original joint error has been permanently "forgotten".

Client-side issues

The client has no knowledge of joints and renders each prim separately. It is possible that network lags will lead to temporary visual breakage of the joints on the client because the jointed prims' positions/orientations/velocities are updated separately without consideration of the joints.

Multi-user testing

This hasn't been tested in a multi-user environment. Single-user testing revealed occasional deadlocks which should hopefully now all be solved, but multi-user testing may expose more corner cases. My hope is that interested users of the OpenSim community can help test various multi-user scenarios. It will also be interesting to see how network latency and server load affect the user experience.

Links

• GForge project homepage http://forge.opensimulator.org/gf/project/ninjaphysics/