[Opensim-dev] Proposal: Implement a de-duplicating core ROBUST asset service

[James Stallings II]

brilliant direction. +1

On Wed, Mar 7, 2012 at 9:44 PM, Cristina Videira Lopes <lopes at ics.uci.edu>wrote:

> +1 on this.
>
>
> On 3/2/2012 6:51 PM, Justin Clark-Casey wrote:
>
>> Hi folks.  As we know, the space required for asset storage in
>> OpenSimulator grows continuously over time.
>>
>> I think this is inevitable in a web-like virtual world - distributed
>> garbage collection is practically impossible. However, the current
>> OpenSimulator asset service stores much more data than it needs to since it
>> fails to identify assets that are exact duplicates of each other.
>>
>> Previous work in places such as OSGrid, which now uses Dave Coyle's
>> Simple Ruby Asset Service (SRAS) [1], reveals that preventing duplicate
>> assets has a significant effect on storage requirements (I can't remember
>> the exact figures but I think that it's >30%).
>>
>> Therefore, I propose to introduce a new core asset service
>> (xassetservice) that will implement asset de-duplication via asset data
>> hashing.  This has already been shown to work by SRAS.  I regard this
>> feature as critical for future plans to extend IAR loading and to improve
>> the 3-months-out-of-the-box OpenSimulator experience.  It does not aim to
>> replace external projects such as SRAS for heavy users.
>>
>> I already have a working implementation in the xassetservice git branch
>> (configuration instructions at [2]).  This should not be used in way except
>> for testing - it is still in the prototype stage and can change at any
>> time.  Only a MySQL implementation exists right now.
>>
>> The plan would be to have xassetservice exist alongside and independent
>> of the existing asset service.  Only one service can be used at a time and
>> this is determined via config files.  After considerable testing,
>> xassetservice would become the default asset service for new OpenSimulator
>> installations.  The existing asset service would continue alongside for a
>> very, very, very long time.
>>
>> Since asset datasets are so large and critical there would be no
>> automatic migration between assetservice data and xassetservice.  Instead,
>> there would be an external migration tool.
>>
>> I may also take this opportunity to implement other asset service
>> features, such as data storage on disk instead of database (possibly
>> nicking stuff from kcozen's previous patch for this) and maybe compression
>> (though I'm currently thinking that the cons of this outweight the pros).
>>
>> More detail is at [3].  Comments or alternative implementation
>> suggestions from developers, etc., are welcome.
>>
>> [1] https://github.com/coyled/sras
>> [2] http://opensimulator.org/wiki/**Feature_Proposals/**
>> Deduplicating_Asset_Service#**Testing<http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service#Testing>
>> [3] http://opensimulator.org/wiki/**Feature_Proposals/**
>> Deduplicating_Asset_Service<http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service>
>>
>>  ______________________________**_________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>



-- 
===================================
http://simhost.com
http://twitter.com/jstallings2
http://www.linkedin.com/pub/5/770/a49
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20120307/87027957/attachment-0001.html>