[Opensim-dev] Proposal: Implement a de-duplicating core ROBUST asset service

[Justin Clark-Casey]

Hi folks.  As we know, the space required for asset storage in OpenSimulator grows continuously over time.

I think this is inevitable in a web-like virtual world - distributed garbage collection is practically impossible. 
However, the current OpenSimulator asset service stores much more data than it needs to since it fails to identify 
assets that are exact duplicates of each other.

Previous work in places such as OSGrid, which now uses Dave Coyle's Simple Ruby Asset Service (SRAS) [1], reveals that 
preventing duplicate assets has a significant effect on storage requirements (I can't remember the exact figures but I 
think that it's >30%).

Therefore, I propose to introduce a new core asset service (xassetservice) that will implement asset de-duplication via 
asset data hashing.  This has already been shown to work by SRAS.  I regard this feature as critical for future plans to 
extend IAR loading and to improve the 3-months-out-of-the-box OpenSimulator experience.  It does not aim to replace 
external projects such as SRAS for heavy users.

I already have a working implementation in the xassetservice git branch (configuration instructions at [2]).  This 
should not be used in way except for testing - it is still in the prototype stage and can change at any time.  Only a 
MySQL implementation exists right now.

The plan would be to have xassetservice exist alongside and independent of the existing asset service.  Only one service 
can be used at a time and this is determined via config files.  After considerable testing, xassetservice would become 
the default asset service for new OpenSimulator installations.  The existing asset service would continue alongside for 
a very, very, very long time.

Since asset datasets are so large and critical there would be no automatic migration between assetservice data and 
xassetservice.  Instead, there would be an external migration tool.

I may also take this opportunity to implement other asset service features, such as data storage on disk instead of 
database (possibly nicking stuff from kcozen's previous patch for this) and maybe compression (though I'm currently 
thinking that the cons of this outweight the pros).

More detail is at [3].  Comments or alternative implementation suggestions from developers, etc., are welcome.

[1] https://github.com/coyled/sras
[2] http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service#Testing
[3] http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service

-- 
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc