[Opensim-dev] Proposal: Implement a de-duplicating core ROBUST asset service

[Chris]

I think it sounds like a great idea! I do have a question on this 
approach though out of curiosity.

How would this method handle assets (textures for instance) that are 
referenced on two different objects (or perhaps referenced inside the 
object's inventories) that are essentially the same but have different 
UUIDs? For instance, Object A is a house and has a wallpaper texture 
with a UUID b8982190-6537-11e1-b86c-0800200c9a66 and Object B is 
different house that uses the same exact wallpaper texture (Shares the 
same hash) but has a UUID of 5fa5c9f6-448f-4c9b-8d32-75f226b2173c. Would 
de-duplication destroy one of the assets, resulting in one of the houses 
to have no texture?

An example of this situation arising would be if hypothetically I were 
to build something on another grid using full perm and open licensed 
textures and then export my creation using Imprudence for instance which 
if all the permissions, creator name, and grid conditions are satisfied 
will also export the textures with it, and then import to OpenSim using 
its automatic texture upload and texture reassigning capabilities to 
recreate my object with all textures intact. If these assets already 
exist in the database at the time of upload (perhaps I've created and 
imported something else earlier back that uses assets that share the 
same hash but not the same UUID) will the newly created object be 
missing textures?

Would there be an option in the configuration files for one to turn this 
feature off if needed, perhaps to avoid the above scenario?

On 3/2/2012 8:51 PM, Justin Clark-Casey wrote:
> Hi folks.  As we know, the space required for asset storage in 
> OpenSimulator grows continuously over time.
>
> I think this is inevitable in a web-like virtual world - distributed 
> garbage collection is practically impossible. However, the current 
> OpenSimulator asset service stores much more data than it needs to 
> since it fails to identify assets that are exact duplicates of each 
> other.
>
> Previous work in places such as OSGrid, which now uses Dave Coyle's 
> Simple Ruby Asset Service (SRAS) [1], reveals that preventing 
> duplicate assets has a significant effect on storage requirements (I 
> can't remember the exact figures but I think that it's >30%).
>
> Therefore, I propose to introduce a new core asset service 
> (xassetservice) that will implement asset de-duplication via asset 
> data hashing.  This has already been shown to work by SRAS.  I regard 
> this feature as critical for future plans to extend IAR loading and to 
> improve the 3-months-out-of-the-box OpenSimulator experience.  It does 
> not aim to replace external projects such as SRAS for heavy users.
>
> I already have a working implementation in the xassetservice git 
> branch (configuration instructions at [2]).  This should not be used 
> in way except for testing - it is still in the prototype stage and can 
> change at any time.  Only a MySQL implementation exists right now.
>
> The plan would be to have xassetservice exist alongside and 
> independent of the existing asset service.  Only one service can be 
> used at a time and this is determined via config files.  After 
> considerable testing, xassetservice would become the default asset 
> service for new OpenSimulator installations.  The existing asset 
> service would continue alongside for a very, very, very long time.
>
> Since asset datasets are so large and critical there would be no 
> automatic migration between assetservice data and xassetservice.  
> Instead, there would be an external migration tool.
>
> I may also take this opportunity to implement other asset service 
> features, such as data storage on disk instead of database (possibly 
> nicking stuff from kcozen's previous patch for this) and maybe 
> compression (though I'm currently thinking that the cons of this 
> outweight the pros).
>
> More detail is at [3].  Comments or alternative implementation 
> suggestions from developers, etc., are welcome.
>
> [1] https://github.com/coyled/sras
> [2] 
> http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service#Testing
> [3] 
> http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service
>


-- 
OpenSim: 9 Region Standalone on 0.7.3 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD Phenom II X4 840 3.2 GHz
Memory: 11 GB DDR3
Database: MySQL 5.1.11