[Opensim-dev] Proposal: Implement a de-duplicating core ROBUST asset service

[M.E. Verhagen]

+1

spitting up the asset tabel is brilliant.
loading an iar or oar of a few 100mb wich was previously saved on the same
grid would than not increase the database with the few 100mb but just a few
100k :)

Op zaterdag 3 maart 2012 schreef Justin Clark-Casey (
jjustincc at googlemail.com) het volgende:
> Hi folks.  As we know, the space required for asset storage in
OpenSimulator grows continuously over time.
>
> I think this is inevitable in a web-like virtual world - distributed
garbage collection is practically impossible. However, the current
OpenSimulator asset service stores much more data than it needs to since it
fails to identify assets that are exact duplicates of each other.
>
> Previous work in places such as OSGrid, which now uses Dave Coyle's
Simple Ruby Asset Service (SRAS) [1], reveals that preventing duplicate
assets has a significant effect on storage requirements (I can't remember
the exact figures but I think that it's >30%).
>
> Therefore, I propose to introduce a new core asset service
(xassetservice) that will implement asset de-duplication via asset data
hashing.  This has already been shown to work by SRAS.  I regard this
feature as critical for future plans to extend IAR loading and to improve
the 3-months-out-of-the-box OpenSimulator experience.  It does not aim to
replace external projects such as SRAS for heavy users.
>
> I already have a working implementation in the xassetservice git branch
(configuration instructions at [2]).  This should not be used in way except
for testing - it is still in the prototype stage and can change at any
time.  Only a MySQL implementation exists right now.
>
> The plan would be to have xassetservice exist alongside and independent
of the existing asset service.  Only one service can be used at a time and
this is determined via config files.  After considerable testing,
xassetservice would become the default asset service for new OpenSimulator
installations.  The existing asset service would continue alongside for a
very, very, very long time.
>
> Since asset datasets are so large and critical there would be no
automatic migration between assetservice data and xassetservice.  Instead,
there would be an external migration tool.
>
> I may also take this opportunity to implement other asset service
features, such as data storage on disk instead of database (possibly
nicking stuff from kcozen's previous patch for this) and maybe compression
(though I'm currently thinking that the cons of this outweight the pros).
>
> More detail is at [3].  Comments or alternative implementation
suggestions from developers, etc., are welcome.
>
> [1] https://github.com/coyled/sras
> [2]
http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service#Testing
> [3]
http://opensimulator.org/wiki/Feature_Proposals/Deduplicating_Asset_Service
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>

-- 
Groningen en Hannover Opensims:   secondlife://meverhagen.nl:8002:Hannover
ZW/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20120303/5fe2f1a8/attachment-0001.html>