FSAssets Service
From OpenSimulator
| (3 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
{{Languages|FSAssets_Service}} | {{Languages|FSAssets_Service}} | ||
=FS Assets= | =FS Assets= | ||
| − | FSAssets is intended for | + | FSAssets is intended for grids where the size of the database is expected to exceed 50GB. This option will save the assets to the file system as opposed to the default service which stores assets as blobs in the database. This option also provides deduplication abilities, each asset is hashed when it is received for storage and if the asset already exists, the asset service will link to the existing file rather than store two copies. |
| + | |||
| + | ==Performance Warning== | ||
| + | FSAssets stores data as many small files. Depending on your workload and hardware, this can produce high random I/O and metadata activity and may overload slower disks (for example older HDDs, weak NAS storage, or heavily shared virtual disks). Use fast storage for BaseDirectory/SpoolDirectory, monitor disk latency and queue depth, and plan capacity for inode/file-count growth. | ||
| + | |||
| + | The hash-based directory layout helps distribute files across subfolders, but it does not remove scaling limits of filesystem metadata/index structures. At very large file counts (millions of files), operations like backup, restore, replication, antivirus scans, snapshots, and consistency checks can become a major bottleneck. | ||
| + | |||
| + | Real-world example (small grid): | ||
| + | * Asset files: 833659 | ||
| + | * Asset size: 125 GB | ||
==Config Options== | ==Config Options== | ||
| Line 17: | Line 26: | ||
;FallbackService = "OpenSim.Services.AssetService.dll:AssetService" | ;FallbackService = "OpenSim.Services.AssetService.dll:AssetService" | ||
;DaysBetweenAccessTimeUpdates = 30 | ;DaysBetweenAccessTimeUpdates = 30 | ||
| + | ;SecondaryInstance = false | ||
| + | ;ShowConsoleStats = true | ||
;StorageProvider = "" | ;StorageProvider = "" | ||
;ConnectionString = "" | ;ConnectionString = "" | ||
;Realm = "fsassets" | ;Realm = "fsassets" | ||
| + | |||
| + | ;UseOsgridFormat = false | ||
| + | ;DefaultAssetLoader = "" | ||
| + | ;AssetLoaderArgs = "" | ||
* '''BaseDirectory''' | * '''BaseDirectory''' | ||
| Line 30: | Line 45: | ||
* '''DaysBetweenAccessTimeUpdates''' | * '''DaysBetweenAccessTimeUpdates''' | ||
::By default, the access time is updated each time an asset is requested. This option can be used to skip this update for assets that already have a recent access time and reduce calls to the database. | ::By default, the access time is updated each time an asset is requested. This option can be used to skip this update for assets that already have a recent access time and reduce calls to the database. | ||
| + | * '''SecondaryInstance''' | ||
| + | ::When true, this instance does not act as the main FSAssets instance in the process. Useful if multiple FSAssetConnector instances are created in one process. | ||
| + | * '''ShowConsoleStats''' | ||
| + | ::Enables periodic FSAssets read/miss statistics output. | ||
* '''StorageProvider''' | * '''StorageProvider''' | ||
::To change where the metadata is saved you can set a specific storage provider and connection string. If this is disabled, the default storage provider given in the database section will be used to save metadata. | ::To change where the metadata is saved you can set a specific storage provider and connection string. If this is disabled, the default storage provider given in the database section will be used to save metadata. | ||
| Line 36: | Line 55: | ||
* '''Realm''' | * '''Realm''' | ||
::The table name to save metadata in the database. Default is 'fsassets' | ::The table name to save metadata in the database. Default is 'fsassets' | ||
| + | * '''UseOsgridFormat''' | ||
| + | ::Changes the folder structure from 00/00/00 to 000/000 for compatibility reasons. False means 00/00/00 which is encouraged. | ||
| + | * '''DefaultAssetLoader''' | ||
| + | ::Optional default asset loader plugin used at startup. | ||
| + | * '''AssetLoaderArgs''' | ||
| + | ::Optional argument string passed to DefaultAssetLoader. | ||
==Migration== | ==Migration== | ||
| − | While FSAssets will use the | + | While FSAssets will use the FallbackService if set, to convert any existing assets when they are requested and not found, this will not automatically convert all the old assets to the new service. To convert all assets from the default service there is a console command provided: |
<pre> | <pre> | ||
import <conn> <table> [<start> <count>] | import <conn> <table> [<start> <count>] | ||
| Line 47: | Line 72: | ||
</pre> | </pre> | ||
Depending on the size of your existing assets table, the import process will take some time to complete. The optional parameters, start and count allow you to specify the position and number of rows to convert. | Depending on the size of your existing assets table, the import process will take some time to complete. The optional parameters, start and count allow you to specify the position and number of rows to convert. | ||
| + | |||
| + | There is also a force import mode that overwrites existing content for matching asset IDs: | ||
| + | <pre> | ||
| + | force import <conn> <table> [<start> <count>] | ||
| + | </pre> | ||
| + | Use this only when you explicitly want legacy data to replace current FSAssets content. | ||
| + | |||
| + | If the import command fails with a timeout error, add "Command Timeout=300;" to the connection string: | ||
| + | |||
| + | <pre> | ||
| + | import "Data Source=localhost;Port=3306;Database=Robust;User ID=SomeRobustuser;Password=Somerobustpassword;Old Guids=true;Command Timeout=300;" assets | ||
| + | </pre> | ||
| + | |||
| + | Timeout units and defaults depend on the database provider/driver. In most setups this value is interpreted as seconds. | ||
| + | |||
| + | ==Import and Export Notes== | ||
| + | * '''Import''' | ||
| + | ::Supported by the FS command set via import and force import. | ||
| + | * '''Export''' | ||
| + | ::There is no dedicated bulk export command in the FS command set. For full exports/backups, include both: | ||
| + | ::# the asset files under BaseDirectory | ||
| + | ::# the FSAssets metadata table (Realm, default "fsassets") from the configured database | ||
| + | ::For single assets, dump asset <ID> may be available depending on enabled asset service handlers/modules. | ||
==Hypergrid Assets== | ==Hypergrid Assets== | ||
| Line 54: | Line 102: | ||
LocalServiceModule = "OpenSim.Services.HypergridService.dll:HGFSAssetService" | LocalServiceModule = "OpenSim.Services.HypergridService.dll:HGFSAssetService" | ||
| + | |||
| + | The following HGAssetService options are also used by HGFSAssetService: | ||
| + | |||
| + | UserAccountsService = "..." | ||
| + | ;DisallowExport = "..." | ||
| + | ;DisallowImport = "..." | ||
| + | |||
| + | HomeURI is also required for Hypergrid operation and is usually provided from Startup/Hypergrid/HGAssetService sections. | ||
Latest revision as of 07:37, 9 April 2026
Contents |
[edit] FS Assets
FSAssets is intended for grids where the size of the database is expected to exceed 50GB. This option will save the assets to the file system as opposed to the default service which stores assets as blobs in the database. This option also provides deduplication abilities, each asset is hashed when it is received for storage and if the asset already exists, the asset service will link to the existing file rather than store two copies.
[edit] Performance Warning
FSAssets stores data as many small files. Depending on your workload and hardware, this can produce high random I/O and metadata activity and may overload slower disks (for example older HDDs, weak NAS storage, or heavily shared virtual disks). Use fast storage for BaseDirectory/SpoolDirectory, monitor disk latency and queue depth, and plan capacity for inode/file-count growth.
The hash-based directory layout helps distribute files across subfolders, but it does not remove scaling limits of filesystem metadata/index structures. At very large file counts (millions of files), operations like backup, restore, replication, antivirus scans, snapshots, and consistency checks can become a major bottleneck.
Real-world example (small grid):
- Asset files: 833659
- Asset size: 125 GB
[edit] Config Options
To enable FSAssets, open Robust.ini or Robust.HG.ini depending on whether you are running a standard or hypergrid enabled grid and find the AssetService section. Change the LocalServiceModule from the current option to:
LocalServiceModule = "OpenSim.Services.FSAssetService.dll:FSAssetConnector"
Also uncomment the following required settings, these are specific to FSAssets:
BaseDirectory = "./fsassets/data" SpoolDirectory = "./fsassets/tmp"
The following are optional FSAsset config options:
;FallbackService = "OpenSim.Services.AssetService.dll:AssetService" ;DaysBetweenAccessTimeUpdates = 30 ;SecondaryInstance = false ;ShowConsoleStats = true
;StorageProvider = "" ;ConnectionString = "" ;Realm = "fsassets"
;UseOsgridFormat = false ;DefaultAssetLoader = "" ;AssetLoaderArgs = ""
- BaseDirectory
- This is the folder in which the asset data will be saved.
- SpoolDirectory
- The spool directory is a folder used for temporary storage while the asset is hashed and compressed before it gets moved to the BaseDirectory. This must be on the same file system as the base directory.
- FallbackService
- If you are upgrading from the default asset service, uncomment this and set the default asset connector. This will allow FSAssets to retrieve an existing asset from the old service and add it to the BaseDirectory. The fallback service is checked when an asset is requested but not found by FSAssets.
- DaysBetweenAccessTimeUpdates
- By default, the access time is updated each time an asset is requested. This option can be used to skip this update for assets that already have a recent access time and reduce calls to the database.
- SecondaryInstance
- When true, this instance does not act as the main FSAssets instance in the process. Useful if multiple FSAssetConnector instances are created in one process.
- ShowConsoleStats
- Enables periodic FSAssets read/miss statistics output.
- StorageProvider
- To change where the metadata is saved you can set a specific storage provider and connection string. If this is disabled, the default storage provider given in the database section will be used to save metadata.
- ConnectionString
- DB connection string for the optional StorageProvider mentioned above.
- Realm
- The table name to save metadata in the database. Default is 'fsassets'
- UseOsgridFormat
- Changes the folder structure from 00/00/00 to 000/000 for compatibility reasons. False means 00/00/00 which is encouraged.
- DefaultAssetLoader
- Optional default asset loader plugin used at startup.
- AssetLoaderArgs
- Optional argument string passed to DefaultAssetLoader.
[edit] Migration
While FSAssets will use the FallbackService if set, to convert any existing assets when they are requested and not found, this will not automatically convert all the old assets to the new service. To convert all assets from the default service there is a console command provided:
import <conn> <table> [<start> <count>]
The import command expects a database connection string and the name of the legacy asset table to be passed as parameters. The following example shows how to start the import process for a MySQL database, just change the connection details to match your database schema, username and password:
import "Data Source=localhost;Database=SCHEMA;User ID=USER;Password=PASS;Old Guids=true;" assets
Depending on the size of your existing assets table, the import process will take some time to complete. The optional parameters, start and count allow you to specify the position and number of rows to convert.
There is also a force import mode that overwrites existing content for matching asset IDs:
force import <conn> <table> [<start> <count>]
Use this only when you explicitly want legacy data to replace current FSAssets content.
If the import command fails with a timeout error, add "Command Timeout=300;" to the connection string:
import "Data Source=localhost;Port=3306;Database=Robust;User ID=SomeRobustuser;Password=Somerobustpassword;Old Guids=true;Command Timeout=300;" assets
Timeout units and defaults depend on the database provider/driver. In most setups this value is interpreted as seconds.
[edit] Import and Export Notes
- Import
- Supported by the FS command set via import and force import.
- Export
- There is no dedicated bulk export command in the FS command set. For full exports/backups, include both:
- the asset files under BaseDirectory
- the FSAssets metadata table (Realm, default "fsassets") from the configured database
- For single assets, dump asset <ID> may be available depending on enabled asset service handlers/modules.
- There is no dedicated bulk export command in the FS command set. For full exports/backups, include both:
[edit] Hypergrid Assets
If your grid has hypergrid enabled, you probably have it setup to use the HGAssetService for sharing assets with other grids. In this situation you will need to change the HGAssetService section of your Robust.ini to point to the FSAssets version of the HGAssetService known as HGFSAssetService.
To enable the HGFSAssetService, find the HGAssetService section of your Robust.ini, the LocalServiceModule should be changed to:
LocalServiceModule = "OpenSim.Services.HypergridService.dll:HGFSAssetService"
The following HGAssetService options are also used by HGFSAssetService:
UserAccountsService = "..." ;DisallowExport = "..." ;DisallowImport = "..."
HomeURI is also required for Hypergrid operation and is usually provided from Startup/Hypergrid/HGAssetService sections.