Varregion

Varregion

"Varregion" is a feature of OpenSimulator that enables region sizes larger than 256x256. The region is just larger so it acts like a regular region but with borders farther apart.

The implementation uses the Aurora large region protocol extensions so the existing Firestorm and Singularity Aurora support will now work for OpenSimulator.

Restrictions

Some restrictions apply:

• The dimensions must be a multiple of 256
• The dimensions must be square if used with Singularity (as of 20131104)
• There must be no adjacent regions (at least one empty 256m space bordering all sides of the large region)
• You must use BulletSim (as of 20131104, ODE has not been modified for varregions)
  • You must use BulletSim's height map terrain implementation. Add to your INI files:

[BulletSim]
    TerrainImplementation = 0

Configuration

The size is be specified in the Region.ini file:

    [MyRegionName]
    RegionUUID = 95ec77ec-58c5-4ce2-9ff3-b6d1900d78a2
    Location = 1000,1000
    SizeX = 1024
    SizeY = 1024
    InternalAddress = 0.0.0.0
    InternalPort = 9200
    AllowAlternatePorts = False
    ExternalHostName = SYSTEMIP

If size is not specified, it will, of course, default to the legacy size of 256.

If the given dimensions do not fit the restrictions, acceptable values are computed and warning and error messages are output into the log.

The terrain heightmap is stored in the database as an array of 'shorts' (two bytes) so, if you have a very large region you might need to set the database maximum size to a larger value. For instance, a 2618x2618 region (11x11 legacy regions) turns into a 56 megabyte heightmap and, for MySQL, you would need to set max_allowed_packet to something like 64M.

Loading Terrain

TODO: discussion of reading in terrain information

Varregions and OAR Files

TODO: Discussion on saving and restoring varregions.

Implementation Discussion

Since this will be a major change to OpenSimulator that touches a lot of different parts, subsequent posts, will discuss the changes I'm making.

TerrainData

One major problem is passing the terrain data from the region to the protocol stack. The existing implementation passed an array of floats that were presumed to be a 256x256 array of region terrain heights. The TerrainChannel class is an attempt to hide the terrain implementation from TerrainModule . TerrainChannel can't be passed into the protocol stack (LLClientView) because TerrainChannel is defined as part of OpenSim.Region.Framework which is not visible to the protocol code.

My solution is to create the TerrainData class in OpenSim.Framework. TerrainData just wraps the data structure for the terrain and additionally has the attributes giving X and Y size.

I didn't want to change the signature of IClientAPI since so many external modules rely on it. It should be changed to pass TerrainData rather than a float[]. I decided to not change IClientAPI but rather have LLClientView ignore the passed array and instead reach back into the associated scene and fetch the TerrainData instance.

There is one subclass of TerrainData: HeightmapTerrainData which keeps the terrain as a compressed heightmap. The height of each point is stored as a short which is height * compressionFactor where compressionFactor is usually "100". This creates a compact storage of the terrain heights with two decimal points of resolution.

The LLLP ("Linden Lab Legacy Protocol") sends terrain height data in compressed "patches" of 16x16 areas of the terrain height. This is a protocol feature that is implemented in TerrainChannel. I feel that underlying protocol optimizations shouldn't appear up the stack so, in creating TerrainData, I tried to hide terrain patches. I mean, someday terrain will be generalized meshes. Right?

Terrain in the Database

Previously, terrain height maps were saved in the database as a blob of 256x256 doubles. To have different region sizes, that format had to change. There is an existing database field revision that stored the time the terrain was saved. This revision information wasn't used for anything so this field was co-opted to contain a revision code for the height field blob.

There are three forms for the height map blob: legacy, compressed2D and regular2D. "legacy" is, of course, the previous 256x256 collections of doubles. "regular2D" contains the X, Y dimensions and enough floats for that area's heights. "compressed2D" contains the X,Y and compressionFactor followed by enough shorts for the height map.

The database readers and writers default to the legacy format and, if the region happens to have the dimensions 256x256, it is stored using the legacy format. This is an attempt to keep downward compatibility.

All of the database modules (MySQL, SQLite, MSSQL and PSQL) have been modified to store terrain this new way.

Things Known to be Broken

• Rezzing a prim from inventory at greater than <256,256>
• Saving/restoring heightmap when restarting region
  • Start large region, edit large part, restart region, see that edits are not saved

Implementation Notes

Branch in the Repository

As of November 4, 2013, the varregion implementation is in the 'varregion' branch in the OpenSimulator git source repository. This branch is periodically merged with the 'master' branch so it is roughly the 'master' branch plus the varregion functionality.

Since the varregion implementation is supposed to be 100% downward compatible, this branch will eventually be merged into the 'master' branch once the not-breaking-existing-configuration feature is tested and verified.

In an existing clone of the OpenSimulator git source repository, do:

# setup local branch that tracks the remote repository branch
git branch --track varregion origin/varregion
# update the working set with the varregion sources
git checkout varregion

# after you are done testing varregion, restore the 'master' sources with:
git checkout master

Stuff to Work On

What follows are notes I am making as things that might need work are found in OpenSimulator.

• ITerrainLoader implementations
  • How to handle tiles.
  • How to handle large regions sizes.
  • Terrain/FileLoaders/*.cs all return an ITerrainChannel
  • Need to fix all the file reader/writers
  • FileLoaders/LLRAW.cs has several "256"s rather than constant references
• Teleporting: should be able to teleport to anywhere in a large region
  • "MyRegion/550/687/40"
• HG code needs to allow addresses anywhere into large regions
• When GridService return adjacent regions, make sure it does the right thing for large regions
• Consider moving terrain XML serialization into TerrainData
• Consider removing all instances of TerrainChannel.GetFloatsSerialized()
  • Replace with passing around TerrainData
• Code in EntityTransferModule that checks distance to decide of to make new connection
  • search for references to 'Constants.RegionSize'
  • Consider removing the distance code and replacing with call to grid service
• WorldMapModule.cs does a lot of arithmetic depending on constants that are really Contants.RegionSize
• Util.IsOutsideView uses Constants.RegionSize. Is use is ScenePresence a problem?
  • Used in ScenePresence.AdjustKnownSeeds()
  • Is this another instance like EntityTransferModule that needs to goto the grid service?
• Move all the short[] heightmap representation stuff into TerrainData.cs (partially done)
  • Is it possible to move all the patch stuff out of TerrainModule/TerrainChannel?
  • Clean up the use of m_revert. It looks like it is not saved after terrain is modified.
• Need to look through RegionCombinerModule.cs and see what safety checks are needed
  • Maybe just prevent combination if not legacy region size
• In LSL_Api.cs, llEdgeOfWorld() does some neighbor computation. Check for ok'ness.
• Blog entry on moving terrain info into class
  • Moving class from OpenSim.Region.Framework into OpenSim.Framework
• LandManagementModule.SendParcelOverlay sends land sale/ownership info for 4x4m areas
  • Looks like it expects to send exactly one 1024 byte block (for the 256x256 legacy region
  • Check viewer code for what will happen for larger regions
• The code for cloud and wind needs to be enhanced for larger regions
  • New layer types added
• LLClientView.SendMapBlockSplit() needs to have region size to send with the map info
• Ward3DMap/TerrainSplat.cs is filled with "256"s.
  • Doesn't even bother to use the symbolic constants.
• Warp3DMap/Warp3DImageModule.cs is filled with "256"s
• WorldMap/WorldMapModules.cs is filled with "256"S
• Services/MapImageService/MapImageService.cs relies on a constant width of 256
• Region/Framework/Scenes/Tests/BorderTests.cs might need work if non-standard regions are tested
• Verify terrain tests still work (for 256m regions)

Testing

• Terrain/parcel operations
  • Default 256 standalone region
    • Clean database. Start single standalone region. Verify 'pimple' region. Verify one parcel. Flatten region.
    • Create parcels adjacent to each edge and standalone in middle. Change names of each parcel and verify. Restart simulator and verify parcels exist and work correctly.
    • Save five parcel region as OAR. Clear database. Restart region. Load oar. Verify 5 parcels of correct location, name and features.
  • Adjacent 256 standalone regions
  • 768x768 standalone region
  • Changing land from 256 to 768
• Teleporting
• Adjacency
• Hypergrid