Feature Proposals/Deduplicating Asset Service - Revision history

Justincc: /* Migration */

2013-03-16T00:12:06Z

‎Migration

Justincc: /* Migration */

2013-03-15T20:36:44Z

‎Migration

Justincc: /* Migration */

2013-03-15T20:28:21Z

‎Migration

Justincc: /* Testing */

2012-03-18T23:30:49Z

‎Testing

Justincc: /* Hash querying */

2012-03-05T23:34:37Z

‎Hash querying

Justincc at 23:34, 5 March 2012

2012-03-05T23:34:12Z

Justincc at 23:27, 5 March 2012

2012-03-05T23:27:30Z

Justincc: Asset deletion

2012-03-05T23:26:38Z

Asset deletion

MakoBot: Robot: Replacing 'OpenSim' to 'OpenSimulator', which is the precise name

2012-03-04T06:23:44Z

Robot: Replacing 'OpenSim' to 'OpenSimulator', which is the precise name

MakoBot: Robot: Cosmetic changes

2012-03-04T03:02:19Z

Robot: Cosmetic changes

@@ Line 171: / Line 171: @@
 * If something goes wrong then recovery might be harder (although restoration from any database snapshot should be fine).
 * Implementation is more complex.
 == Development ==

@@ Line 145: / Line 145: @@
 == Migration ==
-Migration of assets is extremely difficult due to the vast number of them.
+=== Do asset migration entirely outside of OpenSimulator runtimes ===
-In fact, I would propose that in this case, asset migration is not done within the OpenSimulator runtime. Rather, a parallel asset service would be created and a separate executable to take an asset from the original asset service and add it to the new one. Migration doesn't need to be done all at once - it can be done gradually over time (e.g. by periodic running of a migration tool) and then the services switched over via config change.
+==== Pros ====
-The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSimulator.
+==== Cons ====
+* During migration, two copies of asset data will be in the database.  Therefore, one will need to have double the hard drive space of the existing asset database.  In practice, this should be slightly less for large databases when assets are deduplicated.
 === Cons ===
-A problem with this approach is that during migration, multiple copies of asset data will be in the database which could take up a lot of room.
 An alternative would be for XAssetService to optionally reference a backend vanilla asset database.
@@ Line 160: / Line 163: @@
 One could also choose that moment to migrate the asset.
 == Development ==

← Older revision		Revision as of 20:28, 15 March 2013
Line 151:		Line 151:

	The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSimulator.		The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSimulator.
		+
		+	=== Cons ===
		+	A problem with this approach is that during migration, multiple copies of asset data will be in the database which could take up a lot of room.
		+
		+	An alternative would be for XAssetService to optionally reference a backend vanilla asset database.
		+
		+	New assets could go in the xassets tables. Requests for assets would query xasset db plugin first, then drop back to the older assets db plugin.
		+
		+	One could also choose that moment to migrate the asset.

	== Development ==		== Development ==

@@ Line 162: / Line 162: @@
 A test implementation is being done in the xassetservice OpenSimulator git branch. You can try this out by building the code and then putting the following config in StandaloneCommon.ini or any other of the *Common ini files in the existing [AssetService] section.
-[AssetService]
+ [AssetService]
      LocalServiceModule = "OpenSim.Services.AssetService.dll:XAssetService"
      StorageProvider = "OpenSim.Data.MySQL.dll:MySQLXAssetData"

← Older revision		Revision as of 23:34, 5 March 2012
Line 174:		Line 174:

	This involves three messages rather than one (=> query data hash, <= reply yes/no, => upload metadata/data) but results in far lower data upload. Whether this is a worthwhile trade-off is as of yet unknown.		This involves three messages rather than one (=> query data hash, <= reply yes/no, => upload metadata/data) but results in far lower data upload. Whether this is a worthwhile trade-off is as of yet unknown.
		+
		+	This also involves an extension of the existing asset service interface.

← Older revision		Revision as of 23:34, 5 March 2012
Line 167:		Line 167:

	There is currently only an implementation for MySQL. This implementation is not complete and should not be tried for anything other than testing purposes, it is subject to change without notice in a way that will not be backward compatible.		There is currently only an implementation for MySQL. This implementation is not complete and should not be tried for anything other than testing purposes, it is subject to change without notice in a way that will not be backward compatible.
		+
		+	= Further possible work =
		+
		+	== Hash querying ==
		+	If hashes of assets are stored. then when an asset is uploaded to a simulator, instead of forwarding the uploaded blob to the asset service for hashing and comparison, the simulator could hash the asset itself instead and query the xassetservice as to whether it already has that asset data. If it does, then the simulator only needs to upload the new metadata rather than the data.
		+
		+	This involves three messages rather than one (=> query data hash, <= reply yes/no, => upload metadata/data) but results in far lower data upload. Whether this is a worthwhile trade-off is as of yet unknown.

@@ Line 132: / Line 132: @@
 Migration of asset data from the existing asset service might be slightly simpler with the single table design. However, migration is likely to be difficult in both cases (see below).
-=== Asset service interfaces ===
+== Asset service interfaces ==
 I do not envisaging the OpenSimulator asset service interfaces (IAssetService) to be changed in any way by this proposal. This means that existing asset services will require no alteration.
-=== Asset deletion ===
+== Asset deletion ==
 At this stage, the xassetservice will not support asset deletion (though metadata can still be deleted).  This could be done if the hash column in the metadata table became a secondary key, if we assume that an asset blob with only one reference is deletable, as the current asset service assumes.  However, only a very small subset of assets get requests for deletion so I do not regard this as a major issue.

← Older revision		Revision as of 23:26, 5 March 2012
Line 135:		Line 135:

	I do not envisaging the OpenSimulator asset service interfaces (IAssetService) to be changed in any way by this proposal. This means that existing asset services will require no alteration.		I do not envisaging the OpenSimulator asset service interfaces (IAssetService) to be changed in any way by this proposal. This means that existing asset services will require no alteration.
		+
		+	=== Asset deletion ===
		+
		+	At this stage, the xassetservice will not support asset deletion (though metadata can still be deleted). This could be done if the hash column in the metadata table became a secondary key, if we assume that an asset blob with only one reference is deletable, as the current asset service assumes. However, only a very small subset of assets get requests for deletion so I do not regard this as a major issue.

	== Compression ==		== Compression ==

@@ Line 21: / Line 21: @@
 = Proposal =
-A high proportion of uploaded assets, whether uploaded via the viewer or through mechanisms such as OARs and IARs are duplicates of existing assets stored by the asset service. On OSGrid, for example, Nebadon Izumi, OSGrid president, found that 20-25% of all assets were duplicates when the switched over from the default OpenSim asset service to [https://github.com/coyled/sras Simple Ruby Asset Server (SRAS)].
+A high proportion of uploaded assets, whether uploaded via the viewer or through mechanisms such as OARs and IARs are duplicates of existing assets stored by the asset service. On OSGrid, for example, Nebadon Izumi, OSGrid president, found that 20-25% of all assets were duplicates when the switched over from the default OpenSimulator asset service to [https://github.com/coyled/sras Simple Ruby Asset Server (SRAS)].
 Here, I propose to establish an asset service that hashes all assets and so detects and eliminates duplicates. This would function along similar lines as coyled's existing SRAS.
@@ Line 146: / Line 146: @@
 In fact, I would propose that in this case, asset migration is not done within the OpenSimulator runtime. Rather, a parallel asset service would be created and a separate executable to take an asset from the original asset service and add it to the new one. Migration doesn't need to be done all at once - it can be done gradually over time (e.g. by periodic running of a migration tool) and then the services switched over via config change.
-The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSim.
+The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSimulator.
 == Development ==

@@ Line 3: / Line 3: @@
 November 2011.
-=Status=
+= Status =
-Draft and being test implemented.  Everything here is open for change and discussion.
+Draft and being test implemented. Everything here is open for change and discussion.
-=Proposers=
+= Proposers =
 Justin Clark-Casey (justincc)
-=Introduction=
+= Introduction =
-The asset service stores practically all sim data (textures, scripts, etc.).  Many assets are exact duplicates of existing assets, except stored under a different asset ID (and occasionally different metadata).
+The asset service stores practically all sim data (textures, scripts, etc.). Many assets are exact duplicates of existing assets, except stored under a different asset ID (and occasionally different metadata).
 This feature would be to create an asset service which stores hashes of all assets, enabling duplicate assets to be detected and only one copy stored.
@@ Line 19: / Line 19: @@
 The intention is not to replace existing facilities such as the [https://github.com/coyled/sras Simple Ruby Asset Server] but rather to raise the level of the default asset service in OpenSimulator, in order to cope with future demands placed on it by upload and other content import mechanisms.
-=Proposal=
+= Proposal =
-A high proportion of uploaded assets, whether uploaded via the viewer or through mechanisms such as OARs and IARs are duplicates of existing assets stored by the asset service.  On OSGrid, for example, Nebadon Izumi, OSGrid president, found that 20-25% of all assets were duplicates when the switched over from the default OpenSim asset service to [https://github.com/coyled/sras Simple Ruby Asset Server (SRAS)].
+A high proportion of uploaded assets, whether uploaded via the viewer or through mechanisms such as OARs and IARs are duplicates of existing assets stored by the asset service. On OSGrid, for example, Nebadon Izumi, OSGrid president, found that 20-25% of all assets were duplicates when the switched over from the default OpenSim asset service to [https://github.com/coyled/sras Simple Ruby Asset Server (SRAS)].
-Here, I propose to establish an asset service that hashes all assets and so detects and eliminates duplicates.  This would function along similar lines as coyled's existing SRAS.
+Here, I propose to establish an asset service that hashes all assets and so detects and eliminates duplicates. This would function along similar lines as coyled's existing SRAS.
-==Design==
+== Design ==
 There are two major alternatives.
-===Design 1: Add hash and pointer column to existing asset table===
+=== Design 1: Add hash and pointer column to existing asset table ===
-The first design would see a hash column and a pointer column added to the existing asset table.  The hash column would be a primary key.
+The first design would see a hash column and a pointer column added to the existing asset table. The hash column would be a primary key.
 Pseudo-code for adding a new asset
@@ Line 57: / Line 57: @@
       return no such asset
-===Design 2: Create two separate tables assetsmeta and assetsdata===
+=== Design 2: Create two separate tables assetsmeta and assetsdata ===
-The second design would see the creation of two separate tables.  The assetsmeta would be
+The second design would see the creation of two separate tables. The assetsmeta would be
 {| border="1"
@@ Line 122: / Line 122: @@
       return no such asset
-===Discussion===
+=== Discussion ===
-I vastly prefer a two table design to extending the existing single table.  To be honest, the single table design is only included for comparison purposes.
+I vastly prefer a two table design to extending the existing single table. To be honest, the single table design is only included for comparison purposes.
-With a single table design one gets columns with no data but pointers to other assets, and assets with data but blank pointers.  The point where metadata is retrieved is also non-obvious.
+With a single table design one gets columns with no data but pointers to other assets, and assets with data but blank pointers. The point where metadata is retrieved is also non-obvious.
-Also, the two table design can cope with asset removal (with the addition of referencing counting to assetsdata).  This is extremely difficult with a one table design.
+Also, the two table design can cope with asset removal (with the addition of referencing counting to assetsdata). This is extremely difficult with a one table design.
-Migration of asset data from the existing asset service might be slightly simpler with the single table design.  However, migration is likely to be difficult in both cases (see below).
+Migration of asset data from the existing asset service might be slightly simpler with the single table design. However, migration is likely to be difficult in both cases (see below).
-===Asset service interfaces===
+=== Asset service interfaces ===
-I do not envisaging the OpenSimulator asset service interfaces (IAssetService) to be changed in any way by this proposal.  This means that existing asset services will require no alteration.
+I do not envisaging the OpenSimulator asset service interfaces (IAssetService) to be changed in any way by this proposal. This means that existing asset services will require no alteration.
-==Compression==
+== Compression ==
-Asset data will probably not be compressed.  Though it reduces storage size, it also requires CPU time to compress and decompress.  As we aren't transmitting data over the wire, network considerations do not come into play.
+Asset data will probably not be compressed. Though it reduces storage size, it also requires CPU time to compress and decompress. As we aren't transmitting data over the wire, network considerations do not come into play.
-==Migration==
+== Migration ==
 Migration of assets is extremely difficult due to the vast number of them.
-In fact, I would propose that in this case, asset migration is not done within the OpenSimulator runtime.  Rather, a parallel asset service would be created and a separate executable to take an asset from the original asset service and add it to the new one.  Migration doesn't need to be done all at once - it can be done gradually over time (e.g. by periodic running of a migration tool) and then the services switched over via config change.
+In fact, I would propose that in this case, asset migration is not done within the OpenSimulator runtime. Rather, a parallel asset service would be created and a separate executable to take an asset from the original asset service and add it to the new one. Migration doesn't need to be done all at once - it can be done gradually over time (e.g. by periodic running of a migration tool) and then the services switched over via config change.
-The de-duplicating asset service would be created as a parallel one to the existing asset service.  Once ready, it would become the default, though the older asset service would remain and be deprecated.  Only after a long period would the older asset service be removed from OpenSim.
+The de-duplicating asset service would be created as a parallel one to the existing asset service. Once ready, it would become the default, though the older asset service would remain and be deprecated. Only after a long period would the older asset service be removed from OpenSim.
-==Development==
+== Development ==
-The service will be developed as a new package within OpenSimulator core.  The original asset service package will never be altered.  Choice of asset service will be done via config as usual.
+The service will be developed as a new package within OpenSimulator core. The original asset service package will never be altered. Choice of asset service will be done via config as usual.
 Initial development of the service may also take place in a separate branch until the data schemas have been sorted out.
-==Testing==
+== Testing ==
-A test implementation is being done in the xassetservice OpenSimulator git branch.  You can try this out by building the code and then putting the following config in StandaloneCommon.ini or any other of the *Common ini files in the existing [AssetService] section.
+A test implementation is being done in the xassetservice OpenSimulator git branch. You can try this out by building the code and then putting the following config in StandaloneCommon.ini or any other of the *Common ini files in the existing [AssetService] section.
 [AssetService]
@@ Line 162: / Line 162: @@
      StorageProvider = "OpenSim.Data.MySQL.dll:MySQLXAssetData"
-There is currently only an implementation for MySQL.  This implementation is not complete and should not be tried for anything other than testing purposes, it is subject to change without notice in a way that will not be backward compatible.
+There is currently only an implementation for MySQL. This implementation is not complete and should not be tried for anything other than testing purposes, it is subject to change without notice in a way that will not be backward compatible.