Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008859opensim[GRID] Asset Servicepublic2021-01-30 19:342021-02-02 00:48
ReporterFerd Frederix 
Assigned To 
PrioritynormalSeverityminorReproducibilitysometimes
StatusnewResolutionopen 
PlatformOperating SystemOperating System Version
Product Version 
Target VersionFixed in Version 
Summary0008859: FSAssets continues to fetch from fallback after import command
DescriptionThis may be hard for anyone to repro. When FSAssets is first started after a grid has run for a long time so it has a lot of assets, it will continue to fetch missing assets from fallback even after importing all assets. On multiple Windows Server 2019, Server 2016, and on Windows 10, I see bursts of these messages:

10:08:22 - [FSASSETS]: Added asset 60e52e2f-49e7-4457-97f7-14618d049b90 from fallback to local store

This happens even after multiple import commands have been issued. FSAssets should be reliably deduplicating and gzipping these files to make incremental backup of all assets possible. But there are some still going to assets in Mysql and are not found in fsassets.


I would like to be able to get automatically migrate users to fsassets, and get rid of these 100+ GB sized assets tables, but do not dare as these grids would lose assets. Large single files are prone to bit hits. The ibdata file is difficult to backup. Mysqldump is notoriously slow and hurts robust performance. Fsassets is easy to incrementally back up and more reliable. And Mysql can be dumped in seconds instead of hours.

Steps To ReproduceEnable Fsassets, including the section for LocalServiceModule. Then use the following import command in robust. It has a extra timeout added to it as it often fails on many machines without it. I have updated the wiki page to show this command timeout as an option.

import "Data Source=localhost;Port=3306;Database=Robust;User ID=SomeRobustuser;Password=Somerobustpassword;Old Guids=true;Command Timeout=300;" assets

See http://opensimulator.org/wiki/FSAssets_Service [^]

After running the import command, the fallback logging should stop. It may take some time to show up.
Additional Information
Config I use:

[AssetService]

;; Choose an asset service (Only one option should be enabled)
LocalServiceModule=OpenSim.Services.AssetService.dll:AssetService

;LocalServiceModule="OpenSim.Services.FSAssetService.dll:FSAssetConnector"
;; FSAssetDirectories. Base directory, where final asset files are stored and Spool directory for temp files
;; These directories must be on the same physical filesystem
BaseDirectory=./fsassets/data
SpoolDirectory=./fsassets/tmp

;; Original service can be checked if FSAssets can not find an asset
FallbackService=OpenSim.Services.AssetService.dll:AssetService

;; How many days since last updating the access time before its updated again by FSAssets when accessing an asset
;; Reduces DB calls if asset is requested often. Default value 0 will always update access time
DaysBetweenAccessTimeUpdates=0

;; Should FSAssets print read/write stats to the robust console, default is true
ShowConsoleStats=True

;; FSAssets Custom Database Config (Leave blank to use grids default database configuration)
;StorageProvider=
;ConnectionString=

Realm=fsassets

;; The following are common to both the default asset service and FSAsset service
;; Common asset service options
DefaultAssetLoader="OpenSim.Framework.AssetLoader.Filesystem.dll"
AssetLoaderArgs=./assets/AssetSets.xml

; Allow maptile assets to remotely deleted by remote calls to the asset service.
; There is no harm in having this as false - it just means that historical maptile assets are not deleted.
; This only applies to maptiles served via the version 1 viewer mechanisms
; Default is false
AllowRemoteDelete=true

; Allow all assets to be remotely deleted.
; Only set this to true if you are operating a grid where you control all calls to the asset service
; (where a necessary condition is that you control all simulators) and you need this for admin purposes.
; If set to true, AllowRemoteDelete = true is required as well.
; Default is false.
AllowRemoteDeleteAllTypes=false
TagsNo tags attached.
Git Revision or version numberVersion #70e00a00ec (fix creators user cache, 2021-01-07
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
Script EngineXEngine
Environment.NET / Windows64
Mono VersionNone
Viewer\
Attached Filespng file icon Image1.png [^] (472,388 bytes) 2021-01-30 19:39

- Relationships

-  Notes
(0037526)
Ferd Frederix (reporter)
2021-01-30 19:38

Attached a photo of it finishing the import, then immediately finding multiple assets not in fsassets.
(0037527)
tampa (reporter)
2021-01-31 03:45

The built-in importer is a bit... well odd. You can write your own though, the formats are not difficult to convert to file data once you know what they are. From there you just have to construct the directory structure and new database entries. Wrote a similar converter for SRAS-style assets a while back, so not that complex.

Why the built-in importer seems to not convert properly is beyond me, but it has been like that for a long time now. I think, especially on Windows, it may have something to do with case sensitivity somewhere in there. Of course there is also the problem of it being rather slow due to difference in file lock handling. Best to do the conversions on Linux with appropriate ulimits set.
(0037529)
UbitUmarov (administrator)
2021-02-01 12:25

on a quick look to code, only case i see that can keep doing that is a asset with a empty data array (there with len = 0)
can you check if the assets are (broken) like that?
(0037530)
Ferd Frederix (reporter)
2021-02-01 22:44

Thank you Ubit. I can check that. I have more data, but it is not what I expected at all. It may just be a bad database.

On one grid, it spits out the message, such as [FSASSETS]: Added asset 60e52e2f-49e7-4457-97f7-14618d049b90 from fallback to local store very often. Long lists of them.

I wrote a Perl program to scan the assets table, makes a SH256, and looks for it on disk. But it has just the usual small number of standard assets! So the message is misleading.

I then did a reverse lookup. I read fsassets->hash, break it down into a file path, and check to see if the hash folder exists on disk. Not even looking for the gzip yet, just the folder and finding missing assets.

my $fsasset = 'C:/DreamGrid/OutworldzFiles/Opensim/bin/fsassets/data';
my $RS = $schema->resultset('Fsasset');
foreach my $r ($RS->all) # read each row object
{
    $r->hash =~ /^(\S{2})(\S{2})(\S{2})(\S{4}).*/;
    my $file = $fsasset . "/$1/$2/$3/$4/";
    if (! -e $file) {
            Complain and log it...
    }
}

I have found 547 bad out of 698544 my first run. I have a log and will examine it to see if these are the assets it is looking for.

So I suspect the 547 folders that hold the hash that are missing from fsassets folders triggers the message, robust/fsassets is looking in fallback (assets).

Which is what it should be doing. I'll turn on DEBUG level and see if fsassets has more to say about this.
(0037531)
tampa (reporter)
2021-02-02 00:48

The original assets are still in the database somewhere, so see what size they are and if perhaps there is a data size bottleneck preventing proper conversion.

I do recall some issues, that were fixed though, regarding deduplication being somewhat iffy with some references existing already, I doubt it is that though.

Actually... there isn't really anything preventing assets from going bad on either so there is a possibility for the asset being bad in the first place. This does happen from time to time given how fragile such a complex system is. The original asset is proper, that's verified?

- Issue History
Date Modified Username Field Change
2021-01-30 19:34 Ferd Frederix New Issue
2021-01-30 19:38 Ferd Frederix Note Added: 0037526
2021-01-30 19:39 Ferd Frederix File Added: Image1.png
2021-01-31 03:45 tampa Note Added: 0037527
2021-02-01 12:25 UbitUmarov Note Added: 0037529
2021-02-01 22:44 Ferd Frederix Note Added: 0037530
2021-02-02 00:48 tampa Note Added: 0037531


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker