Varregion

From OpenSimulator

Revision as of 08:25, 29 December 2013 by Misterblue (Talk | contribs)

Jump to: navigation, search

Varregion

Contents

"Varregion" is a feature of OpenSimulator that enables region sizes larger than 256x256. The region is just larger so it acts like a regular region but with borders farther apart.

The implementation uses the Aurora large region protocol extensions so the existing Firestorm and Singularity Aurora support will now work for OpenSimulator.

A growing list of protocol changes to implement "varregion" is at Varregion Protocol.

Restrictions

Some restrictions apply:

  • The dimensions must be a multiple of 256
  • The dimensions must be square if used with Singularity (as of 20131104)
  • There must be no adjacent regions (at least one empty 256m space bordering all sides of the large region)
  • You must use BulletSim (as of 20131104, ODE has not been modified for varregions)
    • You must use BulletSim's height map terrain implementation. Add to your INI files: 
[BulletSim]
    TerrainImplementation = 0

Configuration

The size is be specified in the Region.ini file: 

    [MyRegionName]
    RegionUUID = 95ec77ec-58c5-4ce2-9ff3-b6d1900d78a2
    Location = 1000,1000
    SizeX = 1024
    SizeY = 1024
    InternalAddress = 0.0.0.0
    InternalPort = 9200
    AllowAlternatePorts = False
    ExternalHostName = SYSTEMIP

If size is not specified, it will, of course, default to the legacy size of 256.

If the given dimensions do not fit the restrictions, acceptable values are computed and warning and error messages are output into the log.

If converting from a mega-region, remember to set "CombineContiguousRegions = false".

for llRezObject to work beyond 256m set "ScriptDistanceLimitFactor = 52.0" (for 512mx512m) in the [Xengine] section of OpenSim.ini.

The terrain heightmap is stored in the database as an array of 'shorts' (two bytes) so, if you have a very large region you might need to set the database maximum size to a larger value. For instance, a 2816x2816 region (11x11 legacy regions) turns into a 56 megabyte heightmap and, for MySQL, you would need to set max_allowed_packet to something like 64M.

Loading Terrain

TODO: discussion of reading in terrain information

Varregions and OAR Files

TODO: Discussion on saving and restoring varregions.

Implementation Discussion

Since this will be a major change to OpenSimulator that touches a lot of different parts, subsequent posts, will discuss the changes I'm making.

TerrainData

One major problem is passing the terrain data from the region to the protocol stack. The existing implementation passed an array of floats that were presumed to be a 256x256 array of region terrain heights. The TerrainChannel class is an attempt to hide the terrain implementation from TerrainModule . TerrainChannel can't be passed into the protocol stack (LLClientView) because TerrainChannel is defined as part of OpenSim.Region.Framework which is not visible to the protocol code.

My solution is to create the TerrainData class in OpenSim.Framework. TerrainData just wraps the data structure for the terrain and additionally has the attributes giving X and Y size.

I didn't want to change the signature of IClientAPI since so many external modules rely on it. It should be changed to pass TerrainData rather than a float[]. I decided to not change IClientAPI but rather have LLClientView ignore the passed array and instead reach back into the associated scene and fetch the TerrainData instance.

There is one subclass of TerrainData: HeightmapTerrainData which keeps the terrain as a compressed heightmap. The height of each point is stored as a short which is height * compressionFactor where compressionFactor is usually "100". This creates a compact storage of the terrain heights with two decimal points of resolution.

The LLLP ("Linden Lab Legacy Protocol") sends terrain height data in compressed "patches" of 16x16 areas of the terrain height. This is a protocol feature that is implemented in TerrainChannel. I feel that underlying protocol optimizations shouldn't appear up the stack so, in creating TerrainData, I tried to hide terrain patches. I mean, someday terrain will be generalized meshes. Right?

Terrain in the Database

Previously, terrain height maps were saved in the database as a blob of 256x256 doubles. To have different region sizes, that format had to change. There is an existing database field revision that stored the time the terrain was saved. This revision information wasn't used for anything so this field was co-opted to contain a revision code for the height field blob.

There are three forms for the height map blob: legacy, compressed2D and regular2D. "legacy" is, of course, the previous 256x256 collections of doubles. "regular2D" contains the X, Y dimensions and enough floats for that area's heights. "compressed2D" contains the X,Y and compressionFactor followed by enough shorts for the height map.

The database readers and writers default to the legacy format and, if the region happens to have the dimensions 256x256, it is stored using the legacy format. This is an attempt to keep downward compatibility.

All of the database modules (MySQL, SQLite, MSSQL and PSQL) have been modified to store terrain this new way.

Sensing Border Crossing

Most of the ‘move to new region’ code is based on checking boundaries. There is much code related to computing if an object or avatar has crossed a region boundary and then computing the address of the next region from same. Introducing variable sized regions messes a lot of this computation up. That is, the code doing the arithmetic usually assumes it knows the address of the next region based on a known region size and them can compute the location of the next region based on that. With varregions those assumptions no longer hold. Varregion implementation means that the computation of region base locations and border locations moves to the GridService who is the entity who really knows the size of all the regions and what is adjacent to what.

The realization that location to region computation is really a GridService operation lead me to totally rip apart the grid boundary checking code and replace it with two functions: Scene.PositionIsInCurrentRegion(Vector3 pos) and then EntityTransferModule.GetRegionContainingWorldLocation(double X, double Y). The former function tests to see if the object/avatar has moved out of the current region and the latter gets the region moved into. (Side note: GetRegionContainingWorldLocation should really be a function on IGridService but that exercise is left for future hacking).

These changes leave all the 'border' code in limbo -- the generation of the border lists is still there but it is not being used. This should eventually be cleaned up. Also, the computation of neighbor regions is scattered around with routines on IGridService, in EntityTransferModule and a bunch of bookkeeping in Scene. This too should be cleaned up.

Things Known to be Broken

  • Rezzing a prim from inventory at greater than <256,256>
  • Saving/restoring heightmap when restarting region
    • Start large region, edit large part, restart region, see that edits are not saved

Implementation Notes

Branch in the Repository

As of November 4, 2013, the varregion implementation is in the 'varregion' branch in the OpenSimulator git source repository. This branch is periodically merged with the 'master' branch so it is roughly the 'master' branch plus the varregion functionality.

Since the varregion implementation is supposed to be 100% downward compatible, this branch will eventually be merged into the 'master' branch once the not-breaking-existing-configuration feature is tested and verified.

In an existing clone of the OpenSimulator git source repository, do: 

# setup local branch that tracks the remote repository branch
git branch --track varregion origin/varregion
# update the working set with the varregion sources
git checkout varregion

# after you are done testing varregion, restore the 'master' sources with:
git checkout master

Stuff to Work On

What follows are notes I am making as things that might need work are found in OpenSimulator.

  • Consolidate neighbor region computation code (Scene, EntityTransferModule and IGridService all have 'get neighbor' code)
  • Clean up/eliminate border list creation and use code
  • Region handles are world coords and also the base of the region
    • verify that everyone who converts coords to handle (Util.RegionWorldLocToHandle) is supplying the base of the region.
  • ITerrainLoader implementations
    • How to handle tiles.
    • How to handle large regions sizes.
    • Terrain/FileLoaders/*.cs all return an ITerrainChannel
    • Need to fix all the file reader/writers
    • FileLoaders/LLRAW.cs has several "256"s rather than constant references
  • Teleporting: should be able to teleport to anywhere in a large region
    • "MyRegion/550/687/40"
  • HG code needs to allow addresses anywhere into large regions
  • When GridService return adjacent regions, make sure it does the right thing for large regions
  • Consider moving terrain XML serialization into TerrainData
  • Consider removing all instances of TerrainChannel.GetFloatsSerialized()
    • Replace with passing around TerrainData
  • Code in EntityTransferModule that checks distance to decide of to make new connection
    • search for references to 'Constants.RegionSize'
    • Consider removing the distance code and replacing with call to grid service
  • WorldMapModule.cs does a lot of arithmetic depending on constants that are really Contants.RegionSize
  • Util.IsOutsideView uses Constants.RegionSize. Is use is ScenePresence a problem?
    • Used in ScenePresence.AdjustKnownSeeds()
    • Is this another instance like EntityTransferModule that needs to goto the grid service?
  • Move all the short[] heightmap representation stuff into TerrainData.cs (partially done)
    • Is it possible to move all the patch stuff out of TerrainModule/TerrainChannel?
    • Clean up the use of m_revert. It looks like it is not saved after terrain is modified.
  • Need to look through RegionCombinerModule.cs and see what safety checks are needed
    • Maybe just prevent combination if not legacy region size
  • In LSL_Api.cs, llEdgeOfWorld() does some neighbor computation. Check for ok'ness.
  • Blog entry on moving terrain info into class
    • Moving class from OpenSim.Region.Framework into OpenSim.Framework
  • LandManagementModule.SendParcelOverlay sends land sale/ownership info for 4x4m areas
    • Looks like it expects to send exactly one 1024 byte block (for the 256x256 legacy region
    • Check viewer code for what will happen for larger regions
  • The code for cloud and wind needs to be enhanced for larger regions
    • New layer types added
  • LLClientView.SendMapBlockSplit() needs to have region size to send with the map info
  • Ward3DMap/TerrainSplat.cs is filled with "256"s.
    • Doesn't even bother to use the symbolic constants.
  • Warp3DMap/Warp3DImageModule.cs is filled with "256"s
  • WorldMap/WorldMapModules.cs is filled with "256"S
  • Services/MapImageService/MapImageService.cs relies on a constant width of 256
  • Region/Framework/Scenes/Tests/BorderTests.cs might need work if non-standard regions are tested
  • Verify terrain tests still work (for 256m regions)
  • Region coords are sometimes 'int' and sometimes 'uint' with conversions EVERYWHERE
    • pass over everything and convert region coordinates it 'uint's

Viewer

  • LLSurface is wired to expect adjacent regions of same size
  • Terrain surface image is sized with a 'static": " static S32 sTextureSize; // Size of the surface texture"
  • From FreeNode:#SingularityViewer 20131125: 
<Aleric> radams1, SianaGearz, frnic : I couldn't help myself doing a profile anyway..  The reason the viewer is slow is because it's busy "drawing" the terrain. "drawing" between quotes because my guess is that it is trying to draw ALL terrain and then clip that afterwards to the frustrum... 
<Aleric> So, this could be greatly improved by disregarding terrain way way earlier when it is beyond the drawing range anyway.
<SianaGearz> OK
<Aleric> I disabled all rendering, even added a 'return' in terrain render
<Aleric> but.. no improvement of FPS!
<frnic> oops
<Aleric> LLSurfacePatch::updateVisibility() is still eating a lot of cpu.
<Aleric> LLSurface::updatePatchVisibilities does 97% of the calls to LLSurfacePatch::updateVisibility
<Aleric> which then spends most of it's time calling LLCamera::AABBInFrustumNoFarClip


Testing

  • Terrain/parcel operations
    • Default 256 standalone region
      • Clean database. Start single standalone region. Verify 'pimple' region. Verify one parcel. Flatten region.
      • Create parcels adjacent to each edge and standalone in middle. Change names of each parcel and verify. Restart simulator and verify parcels exist and work correctly.
      • Save five parcel region as OAR. Clear database. Restart region. Load oar. Verify 5 parcels of correct location, name and features.
    • Adjacent 256 standalone regions
    • 768x768 standalone region
    • Changing land from 256 to 768
  • Teleporting
  • Adjacency
  • Hypergrid
Retrieved from "http://opensimulator.org/index.php?title=Varregion&oldid=34766"
Personal tools
General
About This Wiki