Revision as of 18:56, 2 March 2012

Date

November 2011.

Status

Draft and being test implemented. Everything here is open for change and discussion.

Proposers

Justin Clark-Casey (justincc)

Introduction

The asset service stores practically all sim data (textures, scripts, etc.). Many assets are exact duplicates of existing assets, except stored under a different asset ID (and occasionally different metadata).

This feature would be to create an asset service which stores hashes of all assets, enabling duplicate assets to be detected and only one copy stored.

The intention is not to replace existing facilities such as the Simple Ruby Asset Server but rather to raise the level of the default asset service in OpenSimulator, in order to cope with future demands placed on it by upload and other content import mechanisms.

Proposal

A high proportion of uploaded assets, whether uploaded via the viewer or through mechanisms such as OARs and IARs are duplicates of existing assets stored by the asset service. On OSGrid, for example, Nebadon Izumi, OSGrid president, found that 20-25% of all assets were duplicates when the switched over from the default OpenSim asset service to Simple Ruby Asset Server (SRAS).

Here, I propose to establish an asset service that hashes all assets and so detects and eliminates duplicates. This would function along similar lines as coyled's existing SRAS.

Design

There are two major alternatives.

Design 1: Add hash and pointer column to existing asset table

The first design would see a hash column and a pointer column added to the existing asset table. The hash column would be a primary key.

Pseudo-code for adding a new asset

On asset add
  Hash new asset
  Compare to existing hashes
  If match
    create new asset table entry storing metadata and pointer to existing asset hash
  else
    create new asset table entry storing data, hash and metadata

Pseudo-code for retrieving an asset

On asset get
  select existing asset based on input id
  If match
    If asset contains data directly
      return existing asset
    else
      look up asset pointed to by reference
      returning asset using initial metadata and pointer-referenced data
  else
    return no such asset

Design 2: Create two separate tables assetsmeta and assetsdata

The second design would see the creation of two separate tables. The assetsmeta would be

This matches the existing assets table except that the data column is no longer present and a sha256 column has been added instead.

The assetsdata table would be

This could be replaced by other storage mechanism options (e.g. filesystem) in the future.

Pseudo-code for adding a new asset

On asset add
  Hash new asset
  Compare to existing hashes
  If match
    create new assetmeta entry pointing to existing assetdata entry
  If no match
    create new asset data entry
    create new assetmeta entry pointing to new assetdata entry

Pseudo-code for retrieving an asset

On asset get
  select existing asset based on input id
  If match
    Fetch asset data from asset data
    Return asset metadata + data
  else
    return no such asset

Discussion

I vastly prefer a two table design to extending the existing single table. To be honest, the single table design is only included for comparison purposes.

With a single table design one gets columns with no data but pointers to other assets, and assets with data but blank pointers. The point where metadata is retrieved is also non-obvious.

Also, the two table design can cope with asset removal (with the addition of referencing counting to assetsdata). This is extremely difficult with a one table design.

Migration of asset data from the existing asset service might be slightly simpler with the single table design. However, migration is likely to be difficult in both cases (see below).

Asset service interfaces

I do not envisaging the OpenSimulator asset service interfaces (IAssetService) to change in any way.

Compression

Asset data will probably not be compressed. Though it reduces storage size, it also requires CPU time to compress and decompress. As we aren't transmitting data over the wire, network considerations do not come into play.

Migration

Migration of assets is extremely difficult due to the vast number of them.

In fact, I would propose that in this case, asset migration is not done within the OpenSimulator runtime. Rather, a parallel asset service would be created and a separate executable to take an asset from the original asset service and add it to the new one. Migration doesn't need to be done all at once - it can be done gradually over time (e.g. by periodic running of a migration tool) and then the services switched over via config change.

The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSim.

Development

The service will be developed as a new package within OpenSimulator core. The original asset service package will never be altered. Choice of asset service will be done via config as usual.

Initial development of the service may also take place in a separate branch until the data schemas have been sorted out.

Testing

A test implementation is being done in the xassetservice OpenSimulator git branch. You can try this out by building the code and then putting the following config in StandaloneCommon.ini or any other of the *Common ini files in the existing [AssetService] section.

[AssetService]

   LocalServiceModule = "OpenSim.Services.AssetService.dll:XAssetService"
   StorageProvider = "OpenSim.Data.MySQL.dll:MySQLXAssetData"

There is currently only an implementation for MySQL. This implementation is not complete and should not be tried for anything other than testing purposes, it is subject to change without notice in a way that will not be backward compatible.