FSAssets Service

From OpenSimulator

(Difference between revisions)
Jump to: navigation, search
(FS Assets)
 
(9 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
{{Languages|FSAssets_Service}}
 
=FS Assets=
 
=FS Assets=
FSAssets is intended for larger grids where the size of the database is expected to exceed 50GB. This option will save the assets to the file system as opposed to the default service which stores assets as blobs in the database. This option also provides deduplication abilities, each asset is hashed when it is received for storage and if the asset already exists, the asset service will link to the existing file rather than store two copies.
+
FSAssets is intended for grids where the size of the database is expected to exceed 50GB. This option will save the assets to the file system as opposed to the default service which stores assets as blobs in the database. This option also provides deduplication abilities, each asset is hashed when it is received for storage and if the asset already exists, the asset service will link to the existing file rather than store two copies.
 +
 
 +
==Performance Warning==
 +
FSAssets stores data as many small files. Depending on your workload and hardware, this can produce high random I/O and metadata activity and may overload slower disks (for example older HDDs, weak NAS storage, or heavily shared virtual disks). Use fast storage for BaseDirectory/SpoolDirectory, monitor disk latency and queue depth, and plan capacity for inode/file-count growth.
 +
 
 +
The hash-based directory layout helps distribute files across subfolders, but it does not remove scaling limits of filesystem metadata/index structures. At very large file counts (millions of files), operations like backup, restore, replication, antivirus scans, snapshots, and consistency checks can become a major bottleneck.
 +
 
 +
Real-world example (small grid):
 +
* Asset files: 833659
 +
* Asset size: 125 GB
  
 
==Config Options==
 
==Config Options==
Line 16: Line 26:
 
  ;FallbackService = "OpenSim.Services.AssetService.dll:AssetService"
 
  ;FallbackService = "OpenSim.Services.AssetService.dll:AssetService"
 
  ;DaysBetweenAccessTimeUpdates = 30
 
  ;DaysBetweenAccessTimeUpdates = 30
 +
;SecondaryInstance = false
 +
;ShowConsoleStats = true
  
 
  ;StorageProvider = ""
 
  ;StorageProvider = ""
 
  ;ConnectionString = ""
 
  ;ConnectionString = ""
 
  ;Realm = "fsassets"
 
  ;Realm = "fsassets"
 +
 +
;UseOsgridFormat = false
 +
;DefaultAssetLoader = ""
 +
;AssetLoaderArgs = ""
  
 
* '''BaseDirectory'''
 
* '''BaseDirectory'''
 
::This is the folder in which the asset data will be saved.
 
::This is the folder in which the asset data will be saved.
 
* '''SpoolDirectory'''
 
* '''SpoolDirectory'''
::The spool directory is a folder used for temporary storage while the asset is hashed and compressed before it gets moved to the BaseDirectory. This should be on the same file system as the base directory.
+
::The spool directory is a folder used for temporary storage while the asset is hashed and compressed before it gets moved to the BaseDirectory. This must be on the same file system as the base directory.
 
* '''FallbackService'''
 
* '''FallbackService'''
 
::If you are upgrading from the default asset service, uncomment this and set the default asset connector. This will allow FSAssets to retrieve an existing asset from the old service and add it to the BaseDirectory. The fallback service is checked when an asset is requested but not found by FSAssets.
 
::If you are upgrading from the default asset service, uncomment this and set the default asset connector. This will allow FSAssets to retrieve an existing asset from the old service and add it to the BaseDirectory. The fallback service is checked when an asset is requested but not found by FSAssets.
 
* '''DaysBetweenAccessTimeUpdates'''
 
* '''DaysBetweenAccessTimeUpdates'''
 
::By default, the access time is updated each time an asset is requested. This option can be used to skip this update for assets that already have a recent access time and reduce calls to the database.
 
::By default, the access time is updated each time an asset is requested. This option can be used to skip this update for assets that already have a recent access time and reduce calls to the database.
 +
* '''SecondaryInstance'''
 +
::When true, this instance does not act as the main FSAssets instance in the process. Useful if multiple FSAssetConnector instances are created in one process.
 +
* '''ShowConsoleStats'''
 +
::Enables periodic FSAssets read/miss statistics output.
 
* '''StorageProvider'''
 
* '''StorageProvider'''
 
::To change where the metadata is saved you can set a specific storage provider and connection string. If this is disabled, the default storage provider given in the database section will be used to save metadata.
 
::To change where the metadata is saved you can set a specific storage provider and connection string. If this is disabled, the default storage provider given in the database section will be used to save metadata.
Line 35: Line 55:
 
* '''Realm'''
 
* '''Realm'''
 
::The table name to save metadata in the database. Default is 'fsassets'
 
::The table name to save metadata in the database. Default is 'fsassets'
 +
* '''UseOsgridFormat'''
 +
::Changes the folder structure from 00/00/00 to 000/000 for compatibility reasons. False means 00/00/00 which is encouraged.
 +
* '''DefaultAssetLoader'''
 +
::Optional default asset loader plugin used at startup.
 +
* '''AssetLoaderArgs'''
 +
::Optional argument string passed to DefaultAssetLoader.
  
 
==Migration==
 
==Migration==
While FSAssets will retrieve and convert an existing assets when they are first requested using the 'FallBackService' if available, this will not automatically convert all the old assets to the new service. To convert all assets from the default service there is a console command provided:
+
While FSAssets will use the FallbackService if set, to convert any existing assets when they are requested and not found, this will not automatically convert all the old assets to the new service. To convert all assets from the default service there is a console command provided:
 
+
<pre>
 
  import <conn> <table> [<start> <count>]
 
  import <conn> <table> [<start> <count>]
 
+
</pre>
 
The import command expects a database connection string and the name of the legacy asset table to be passed as parameters. The following example shows how to start the import process for a MySQL database, just change the connection details to match your database schema, username and password:
 
The import command expects a database connection string and the name of the legacy asset table to be passed as parameters. The following example shows how to start the import process for a MySQL database, just change the connection details to match your database schema, username and password:
 
+
<pre>
 
  import "Data Source=localhost;Database=SCHEMA;User ID=USER;Password=PASS;Old Guids=true;" assets
 
  import "Data Source=localhost;Database=SCHEMA;User ID=USER;Password=PASS;Old Guids=true;" assets
 
+
</pre>
 
Depending on the size of your existing assets table, the import process will take some time to complete. The optional parameters, start and count allow you to specify the position and number of rows to convert.
 
Depending on the size of your existing assets table, the import process will take some time to complete. The optional parameters, start and count allow you to specify the position and number of rows to convert.
 +
 +
There is also a force import mode that overwrites existing content for matching asset IDs:
 +
<pre>
 +
force import <conn> <table> [<start> <count>]
 +
</pre>
 +
Use this only when you explicitly want legacy data to replace current FSAssets content.
 +
 +
If the import command fails with a timeout error, add "Command Timeout=300;" to the connection string:
 +
 +
<pre>
 +
import "Data Source=localhost;Port=3306;Database=Robust;User ID=SomeRobustuser;Password=Somerobustpassword;Old Guids=true;Command Timeout=300;" assets
 +
</pre>
 +
 +
Timeout units and defaults depend on the database provider/driver. In most setups this value is interpreted as seconds.
 +
 +
==Import and Export Notes==
 +
* '''Import'''
 +
::Supported by the FS command set via import and force import.
 +
* '''Export'''
 +
::There is no dedicated bulk export command in the FS command set. For full exports/backups, include both:
 +
::# the asset files under BaseDirectory
 +
::# the FSAssets metadata table (Realm, default "fsassets") from the configured database
 +
::For single assets, dump asset <ID> may be available depending on enabled asset service handlers/modules.
 +
 +
==Hypergrid Assets==
 +
If your grid has hypergrid enabled, you probably have it setup to use the HGAssetService for sharing assets with other grids. In this situation you will need to change the HGAssetService section of your Robust.ini to point to the FSAssets version of the HGAssetService known as HGFSAssetService.
 +
 +
To enable the HGFSAssetService, find the HGAssetService section of your Robust.ini, the LocalServiceModule should be changed to:
 +
 +
  LocalServiceModule = "OpenSim.Services.HypergridService.dll:HGFSAssetService"
 +
 +
The following HGAssetService options are also used by HGFSAssetService:
 +
 +
UserAccountsService = "..."
 +
;DisallowExport = "..."
 +
;DisallowImport = "..."
 +
 +
HomeURI is also required for Hypergrid operation and is usually provided from Startup/Hypergrid/HGAssetService sections.

Latest revision as of 07:37, 9 April 2026

Contents

[edit] FS Assets

FSAssets is intended for grids where the size of the database is expected to exceed 50GB. This option will save the assets to the file system as opposed to the default service which stores assets as blobs in the database. This option also provides deduplication abilities, each asset is hashed when it is received for storage and if the asset already exists, the asset service will link to the existing file rather than store two copies.

[edit] Performance Warning

FSAssets stores data as many small files. Depending on your workload and hardware, this can produce high random I/O and metadata activity and may overload slower disks (for example older HDDs, weak NAS storage, or heavily shared virtual disks). Use fast storage for BaseDirectory/SpoolDirectory, monitor disk latency and queue depth, and plan capacity for inode/file-count growth.

The hash-based directory layout helps distribute files across subfolders, but it does not remove scaling limits of filesystem metadata/index structures. At very large file counts (millions of files), operations like backup, restore, replication, antivirus scans, snapshots, and consistency checks can become a major bottleneck.

Real-world example (small grid):

  • Asset files: 833659
  • Asset size: 125 GB

[edit] Config Options

To enable FSAssets, open Robust.ini or Robust.HG.ini depending on whether you are running a standard or hypergrid enabled grid and find the AssetService section. Change the LocalServiceModule from the current option to:

LocalServiceModule = "OpenSim.Services.FSAssetService.dll:FSAssetConnector"

Also uncomment the following required settings, these are specific to FSAssets:

BaseDirectory = "./fsassets/data"
SpoolDirectory = "./fsassets/tmp"

The following are optional FSAsset config options:

;FallbackService = "OpenSim.Services.AssetService.dll:AssetService"
;DaysBetweenAccessTimeUpdates = 30
;SecondaryInstance = false
;ShowConsoleStats = true
;StorageProvider = ""
;ConnectionString = ""
;Realm = "fsassets"
;UseOsgridFormat = false
;DefaultAssetLoader = ""
;AssetLoaderArgs = ""
  • BaseDirectory
This is the folder in which the asset data will be saved.
  • SpoolDirectory
The spool directory is a folder used for temporary storage while the asset is hashed and compressed before it gets moved to the BaseDirectory. This must be on the same file system as the base directory.
  • FallbackService
If you are upgrading from the default asset service, uncomment this and set the default asset connector. This will allow FSAssets to retrieve an existing asset from the old service and add it to the BaseDirectory. The fallback service is checked when an asset is requested but not found by FSAssets.
  • DaysBetweenAccessTimeUpdates
By default, the access time is updated each time an asset is requested. This option can be used to skip this update for assets that already have a recent access time and reduce calls to the database.
  • SecondaryInstance
When true, this instance does not act as the main FSAssets instance in the process. Useful if multiple FSAssetConnector instances are created in one process.
  • ShowConsoleStats
Enables periodic FSAssets read/miss statistics output.
  • StorageProvider
To change where the metadata is saved you can set a specific storage provider and connection string. If this is disabled, the default storage provider given in the database section will be used to save metadata.
  • ConnectionString
DB connection string for the optional StorageProvider mentioned above.
  • Realm
The table name to save metadata in the database. Default is 'fsassets'
  • UseOsgridFormat
Changes the folder structure from 00/00/00 to 000/000 for compatibility reasons. False means 00/00/00 which is encouraged.
  • DefaultAssetLoader
Optional default asset loader plugin used at startup.
  • AssetLoaderArgs
Optional argument string passed to DefaultAssetLoader.

[edit] Migration

While FSAssets will use the FallbackService if set, to convert any existing assets when they are requested and not found, this will not automatically convert all the old assets to the new service. To convert all assets from the default service there is a console command provided:

 import <conn> <table> [<start> <count>]

The import command expects a database connection string and the name of the legacy asset table to be passed as parameters. The following example shows how to start the import process for a MySQL database, just change the connection details to match your database schema, username and password:

 import "Data Source=localhost;Database=SCHEMA;User ID=USER;Password=PASS;Old Guids=true;" assets

Depending on the size of your existing assets table, the import process will take some time to complete. The optional parameters, start and count allow you to specify the position and number of rows to convert.

There is also a force import mode that overwrites existing content for matching asset IDs:

 force import <conn> <table> [<start> <count>]

Use this only when you explicitly want legacy data to replace current FSAssets content.

If the import command fails with a timeout error, add "Command Timeout=300;" to the connection string:

import "Data Source=localhost;Port=3306;Database=Robust;User ID=SomeRobustuser;Password=Somerobustpassword;Old Guids=true;Command Timeout=300;" assets

Timeout units and defaults depend on the database provider/driver. In most setups this value is interpreted as seconds.

[edit] Import and Export Notes

  • Import
Supported by the FS command set via import and force import.
  • Export
There is no dedicated bulk export command in the FS command set. For full exports/backups, include both:
  1. the asset files under BaseDirectory
  2. the FSAssets metadata table (Realm, default "fsassets") from the configured database
For single assets, dump asset <ID> may be available depending on enabled asset service handlers/modules.

[edit] Hypergrid Assets

If your grid has hypergrid enabled, you probably have it setup to use the HGAssetService for sharing assets with other grids. In this situation you will need to change the HGAssetService section of your Robust.ini to point to the FSAssets version of the HGAssetService known as HGFSAssetService.

To enable the HGFSAssetService, find the HGAssetService section of your Robust.ini, the LocalServiceModule should be changed to:

 LocalServiceModule = "OpenSim.Services.HypergridService.dll:HGFSAssetService"

The following HGAssetService options are also used by HGFSAssetService:

UserAccountsService = "..."
;DisallowExport = "..."
;DisallowImport = "..."

HomeURI is also required for Hypergrid operation and is usually provided from Startup/Hypergrid/HGAssetService sections.

Personal tools
General
About This Wiki