|Anonymous | Login | Signup for a new account||2020-01-24 00:53 PST|
|Main | My View | View Issues | Change Log | Roadmap | Summary | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0005429||opensim||[GRID] Asset Service||public||2011-04-05 08:24||2012-03-23 13:49|
|Product Version||master (dev code)|
|Target Version||Fixed in Version|
|Summary||0005429: Add optional ability to store asset blobs on disk|
|Description||The attached patch (created using git patch) modifies OpenSim to save and retrieve asset blobs to and from disk when used with a MySQL database.|
The patch should *NOT* be commited to the master branch of OpenSim yet as it is a proof of concept at this time. More work needs to be done before it is ready to be integrated in to the main OpenSim source tree. It is provided to allow for further testing and for feedback.
To be done:
1. Storing of blobs on disk should be made a configuration option.
2. Can the asset read/write be made available for use across all databases?
The patch should not be used on a production system. Backup the database before using this patch. Asset blobs will no longer be stored in the database once the patch has been applied. When an asset record is updated the blob field of the asset table will be "lost" (replaced with a 0 byte entry).
I heard comments on #opensim about MySQL databases which have "blown up" in the past when they get to around 50G requiring the indexes to be rebuilt. The problem was reportedly due to the storage of blobs in the database. It made me nervous as I had a table around that size. Wanting to avoid having the problem (if it could still happen) I came up with this patch.
ABOUT THE CHANGES:
The filename for a blob file is based on an sha1 sum of the blob data. The filename is stored in the 'hash' field of the asset record. On disk, the files are stored under a directory called 'assetblobs' using two levels of sub-directories based on the first 3 and second 3 characters of the filename.
The patch includes a migration to update the asset table. The PHP file attached to this report will copy asset blobs from the database to disk and should be run *BEFORE* starting up a patched OpenSim.
1. How can this be made optional?
2. Should temporary asset blobs on disk be stored with '.tmp' appended to the filename to make it easier to find/remove temporary assets if manual clean up of assets is ever needed?
3. Can it be abstracted in a way that will allow other database storage systems (ie. SQLite, MSSQL) used with OpenSim to use the new asset storage/retrieval method?
|Tags||No tags attached.|
|Git Revision or version number||18e206d2ed5cc1828f2ada27ff4698bdc0b84352|
|Run Mode||Standalone (1 Region)|
|Environment||Mono / Linux32|
|Attached Files|| asset-blobs-on-disk.patch [^] (9,135 bytes) 2011-04-05 08:24 [Show Content]
copy-assets-to-disk.php [^] (3,160 bytes) 2011-04-05 08:50
007_AssetStore.sql [^] (134 bytes) 2012-03-23 13:47
|One additional note about the patch... I left in a lot of debugging messages I was using to make certain the generation of the sha1 sum was the same between the PHP code and C# and that the code was saving and retrieving assets properly.|
Hi kcozens. I took asset-blobs-on-disk out for a whirl and it works great (rezzed a simple cube on standalone and restarted).
The following thoughts are a little off-the-top-of-my-head late on a Friday, so please take them as such.
So, how does this compare with the on-disk asset storage code of OpenSim.Region.CoreModules/Asset/FlotsamAssetCache.cs? I suppose that the two are fulfilling rather different functions (one is caching temporarily and not reloading files on startup afaik, while the other is permanent storage). If your code was at the caching level (with a permanent and reloadable cache that did not pass asset data on to the Data layer) then it would work with all storage adaptors, though that is probably inappropriate.
On the Data layer, one way to make it optional would be to introduce a separate interface (IBlobStore?) for different implementations of blob storage. Yours could be one while the existing in table storage for each db could be another (assuming we are happy to keep both data and hash columns, which I don't think is a problem). Configuration could then take place in the [DatabaseService] section (BlobStorageProvider?).
Your IBlobStorage implementation could go in OpenSim.Data.Disk or similar.
OpenSim.Data.MySql (and SQLite and MSSQL) would use whichever one was configured, though it would have to be clear that one can't easily swap back and forth between the options.
Marking temporary blobs with .tmp seems a reasonable approach to me.
Using PHP code to transfer could be an issue - I think we would need a C# utility. How important this is would depend on what happened to the defaults. In any case, I would forsee having this as an option at first to see if there are any issues.
One really interesting thing would be to see performance metrics between the in database storage and separate file storage. Do you think there is likely to be a difference?
edited on: 2012-03-23 17:19
I have attached a small file containing the SQL commands I use to add and a hash field to the assets table and initialize it.
|2011-04-05 08:24||kcozens||New Issue|
|2011-04-05 08:24||kcozens||File Added: asset-blobs-on-disk.patch|
|2011-04-05 08:24||kcozens||Git Revision||=> 18e206d2ed5cc1828f2ada27ff4698bdc0b84352|
|2011-04-05 08:24||kcozens||SVN Revision||=> 0|
|2011-04-05 08:24||kcozens||Run Mode||=> Standalone (1 Region)|
|2011-04-05 08:24||kcozens||Physics Engine||=> BasicPhysics|
|2011-04-05 08:24||kcozens||Environment||=> Mono / Linux32|
|2011-04-05 08:24||kcozens||Mono Version||=> 2.4|
|2011-04-05 08:50||kcozens||File Added: copy-assets-to-disk.php|
|2011-04-05 11:33||kcozens||Note Added: 0018207|
|2011-04-05 11:33||kcozens||Status||new => patch included|
|2011-04-08 16:42||justincc||Note Added: 0018216|
|2012-03-23 13:47||kcozens||File Added: 007_AssetStore.sql|
|2012-03-23 13:49||kcozens||Note Added: 0021137|
|2012-03-23 17:19||kcozens||Note Edited: 0021137||View Revisions|
|Copyright © 2000 - 2012 MantisBT Group|