Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005429opensim[GRID] Asset Servicepublic2011-04-05 08:242012-03-23 13:49
Reporterkcozens 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Statuspatch includedResolutionopen 
PlatformOSOS Version
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0005429: Add optional ability to store asset blobs on disk
DescriptionThe attached patch (created using git patch) modifies OpenSim to save and retrieve asset blobs to and from disk when used with a MySQL database.

The patch should *NOT* be commited to the master branch of OpenSim yet as it is a proof of concept at this time. More work needs to be done before it is ready to be integrated in to the main OpenSim source tree. It is provided to allow for further testing and for feedback.

To be done:
1. Storing of blobs on disk should be made a configuration option.
2. Can the asset read/write be made available for use across all databases?
Additional InformationIMPORTANT:
The patch should not be used on a production system. Backup the database before using this patch. Asset blobs will no longer be stored in the database once the patch has been applied. When an asset record is updated the blob field of the asset table will be "lost" (replaced with a 0 byte entry).

BACKGROUND:
I heard comments on #opensim about MySQL databases which have "blown up" in the past when they get to around 50G requiring the indexes to be rebuilt. The problem was reportedly due to the storage of blobs in the database. It made me nervous as I had a table around that size. Wanting to avoid having the problem (if it could still happen) I came up with this patch.

ABOUT THE CHANGES:
The filename for a blob file is based on an sha1 sum of the blob data. The filename is stored in the 'hash' field of the asset record. On disk, the files are stored under a directory called 'assetblobs' using two levels of sub-directories based on the first 3 and second 3 characters of the filename.
 
The patch includes a migration to update the asset table. The PHP file attached to this report will copy asset blobs from the database to disk and should be run *BEFORE* starting up a patched OpenSim.

QUESTIONS:
1. How can this be made optional?
2. Should temporary asset blobs on disk be stored with '.tmp' appended to the filename to make it easier to find/remove temporary assets if manual clean up of assets is ever needed?
3. Can it be abstracted in a way that will allow other database storage systems (ie. SQLite, MSSQL) used with OpenSim to use the new asset storage/retrieval method?
TagsNo tags attached.
Git Revision or version number18e206d2ed5cc1828f2ada27ff4698bdc0b84352
Run ModeStandalone (1 Region)
Physics EngineBasicPhysics
Script Engine
EnvironmentMono / Linux32
Mono Version2.4
Viewer
Attached Filespatch file icon asset-blobs-on-disk.patch [^] (9,135 bytes) 2011-04-05 08:24 [Show Content]
? file icon copy-assets-to-disk.php [^] (3,160 bytes) 2011-04-05 08:50
? file icon 007_AssetStore.sql [^] (134 bytes) 2012-03-23 13:47

- Relationships

-  Notes
(0018207)
kcozens (administrator)
2011-04-05 11:33

One additional note about the patch... I left in a lot of debugging messages I was using to make certain the generation of the sha1 sum was the same between the PHP code and C# and that the code was saving and retrieving assets properly.
(0018216)
justincc (administrator)
2011-04-08 16:42

Hi kcozens. I took asset-blobs-on-disk out for a whirl and it works great (rezzed a simple cube on standalone and restarted).

The following thoughts are a little off-the-top-of-my-head late on a Friday, so please take them as such.

So, how does this compare with the on-disk asset storage code of OpenSim.Region.CoreModules/Asset/FlotsamAssetCache.cs? I suppose that the two are fulfilling rather different functions (one is caching temporarily and not reloading files on startup afaik, while the other is permanent storage). If your code was at the caching level (with a permanent and reloadable cache that did not pass asset data on to the Data layer) then it would work with all storage adaptors, though that is probably inappropriate.

On the Data layer, one way to make it optional would be to introduce a separate interface (IBlobStore?) for different implementations of blob storage. Yours could be one while the existing in table storage for each db could be another (assuming we are happy to keep both data and hash columns, which I don't think is a problem). Configuration could then take place in the [DatabaseService] section (BlobStorageProvider?).

Your IBlobStorage implementation could go in OpenSim.Data.Disk or similar.

OpenSim.Data.MySql (and SQLite and MSSQL) would use whichever one was configured, though it would have to be clear that one can't easily swap back and forth between the options.

Marking temporary blobs with .tmp seems a reasonable approach to me.

Using PHP code to transfer could be an issue - I think we would need a C# utility. How important this is would depend on what happened to the defaults. In any case, I would forsee having this as an option at first to see if there are any issues.

One really interesting thing would be to see performance metrics between the in database storage and separate file storage. Do you think there is likely to be a difference?
(0021137)
kcozens (administrator)
2012-03-23 13:49
edited on: 2012-03-23 17:19

I have attached a small file containing the SQL commands I use to add and a hash field to the assets table and initialize it.


- Issue History
Date Modified Username Field Change
2011-04-05 08:24 kcozens New Issue
2011-04-05 08:24 kcozens File Added: asset-blobs-on-disk.patch
2011-04-05 08:24 kcozens Git Revision => 18e206d2ed5cc1828f2ada27ff4698bdc0b84352
2011-04-05 08:24 kcozens SVN Revision => 0
2011-04-05 08:24 kcozens Run Mode => Standalone (1 Region)
2011-04-05 08:24 kcozens Physics Engine => BasicPhysics
2011-04-05 08:24 kcozens Environment => Mono / Linux32
2011-04-05 08:24 kcozens Mono Version => 2.4
2011-04-05 08:50 kcozens File Added: copy-assets-to-disk.php
2011-04-05 11:33 kcozens Note Added: 0018207
2011-04-05 11:33 kcozens Status new => patch included
2011-04-08 16:42 justincc Note Added: 0018216
2012-03-23 13:47 kcozens File Added: 007_AssetStore.sql
2012-03-23 13:49 kcozens Note Added: 0021137
2012-03-23 17:19 kcozens Note Edited: 0021137 View Revisions


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker