[Opensim-dev] Proposal: Implement a de-duplicating core ROBUST asset service

[Cristina Videira Lopes]

+1 on this.

On 3/2/2012 6:51 PM, Justin Clark-Casey wrote:
> Hi folks.  As we know, the space required for asset storage in 
> OpenSimulator grows continuously over time.
>
> I think this is inevitable in a web-like virtual world - distributed 
> garbage collection is practically impossible. However, the current 
> OpenSimulator asset service stores much more data than it needs to 
> since it fails to identify assets that are exact duplicates of each 
> other.
>
> Previous work in places such as OSGrid, which now uses Dave Coyle's 
> Simple Ruby Asset Service (SRAS) [1], reveals that preventing 
> duplicate assets has a significant effect on storage requirements (I 
> can't remember the exact figures but I think that it's >30%).
>
> Therefore, I propose to introduce a new core asset service 
> (xassetservice) that will implement asset de-duplication via asset 
> data hashing.  This has already been shown to work by SRAS.  I regard 
> this feature as critical for future plans to extend IAR loading and to 
> improve the 3-months-out-of-the-box OpenSimulator experience.  It does 
> not aim to replace external projects such as SRAS for heavy users.
>
> I already have a working implementation in the xassetservice git 
> branch (configuration instructions at [2]).  This should not be used 
> in way except for testing - it is still in the prototype stage and can 
> change at any time.  Only a MySQL implementation exists right now.
>
> The plan would be to have xassetservice exist alongside and 
> independent of the existing asset service.  Only one service can be 
> used at a time and this is determined via config files.  After 
> considerable testing, xassetservice would become the default asset 
> service for new OpenSimulator installations.  The existing asset 
> service would continue alongside for a very, very, very long time.
>
> Since asset datasets are so large and critical there would be no 
> automatic migration between assetservice data and xassetservice.  
> Instead, there would be an external migration tool.
>
> I may also take this opportunity to implement other asset service 
> features, such as data storage on disk instead of database (possibly 
> nicking stuff from kcozen's previous patch for this) and maybe 
> compression (though I'm currently thinking that the cons of this 
> outweight the pros).
>
> More detail is at [3].  Comments or alternative implementation 
> suggestions from developers, etc., are welcome.
>
> [1] https://github.com/coyled/sras
> [2] 
> http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service#Testing
> [3] 
> http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service
>