Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008247opensim[GRID] Asset Servicepublic2017-10-10 09:512017-10-12 03:53
Reporteraiaustin 
Assigned To 
PrioritynormalSeveritymajorReproducibilitysometimes
StatusnewResolutionopen 
PlatformPCOSWindowsOS Version10
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0008247: Possible difference in way load oar and load iar upload assets reliably
DescriptionIn testing a large upload of an OAR (1.5GB) for a new region and an IAR (2.3GB)for roughly the same content I see a very reliable save oar and save iar with no missing assets and no failures or warnings on my grid. Then moving the content to OSGrid, the load oar also goes up reliably with no warnings on the second grid. But the load iar starts to show a LOT of messages of the form

[Assets]: Upload failed ... will retry later

There are though the normal progress messages indicating the IAR contents are being loaded and I get a successful message when that is complete. But the Upload failed and retry message continue afterwards with the occasional pause.

In one test experiment these went on for over 12 hours after the message indicating the load iar had finished. And I am not yet sure if the assets are all in place on the second grid.

Is there a difference in the upload mechanism for load oar (which seems solid) and load iar (which does not) and can the load iar mechanism be altered to be like the load oar one if so?
TagsNo tags attached.
Git Revision or version number0.9.1 1684 59afeb6 (2017-10-06 01:43)
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
ViewerN/A
Attached Files? file icon Upload-3.iar [^] (2,420,936 bytes) 2017-10-12 02:17

- Relationships

-  Notes
(0032315)
watcher64 (reporter)
2017-10-10 12:16

I have loaded much larger oars and iars, but on linux .. It could be possible you are running into windows file locking issues, or even file limitations ...

Just my .02 as I can't speak to actually having this issue ..
(0032316)
UbitUmarov (administrator)
2017-10-11 00:25

just a "minor" difference...
OAR is basically a local operation, assets are stored on region dbs, etc.
IAR is a grid inventory/assets operation. Contents are stored on grid dbs
( of course minor means major ;) )
(0032317)
aiaustin (developer)
2017-10-11 00:26
edited on: 2017-10-11 00:27

Thanks for that suggestion @watcher64.. and I agree I also have uploaded much larger IARs in the past also. Something really is odd as the OpenSim.exe instance I am loading the IAR from is idle much of the time and goes idle between the retry bursts... but things never seem to complete the upload.

@Ubitmarov.. I assume that the load OAR though does talk to the asset server to upload any new assets that are not already on the target grid? But it could be that the assets are already in place from others or earlier uses of course.

I left things running overnight and its still re-trying every hour or so and reporting failures.

I am reporting this so anyone familiar with the load oar and load iar mechanism can check if there is some difference in the way the uploads work. the (pretty much the same) content in the OAR loaded absolutely solidly with no error or warning. But the load iar for a directory containing essentially the same assets fails... even though I would imagine almost all the assets are actually already in place on the target grid (OSGrid) due to the prior load of the OAR for a region with the assets on it.

(0032318)
aiaustin (developer)
2017-10-11 00:30

The upload never seems to complete and I am getting the retry warning every hour or so even after leaving things 12 hours. Still going as we speak for a load iar of just under 4000 assets (700 or so inventory entities). I am not sure if to quit from the OpenSim.exe or leave it to see if it ever completes.
(0032319)
UbitUmarov (administrator)
2017-10-11 00:43

the assets in queue for upload retry will be lost if you stop the instance, or after 48 hours of retries (or similar time )
(0032320)
aiaustin (developer)
2017-10-11 03:39
edited on: 2017-10-11 14:54

I thought so. That's why I left it running for over 12 hours but it never seemed to clear the backlog. Very odd it goes on so long. And something I have not noticed before. I will keep an eye open to see if this occurs again and try to better ascertain the circumstances.

(0032321)
aiaustin (developer)
2017-10-11 10:36
edited on: 2017-10-12 02:17

Could it be that the message is given in the situation where the asset is already in the (OSGrid in tis case) asset server with the identical asset and UUID?

Or even that the asset (or creator avatar) is "banned" to upload to the asset server as inappropriately copied in some way? Is that a features of some asset servers such as OSGrid?

(0032322)
aiaustin (developer)
2017-10-12 02:17
edited on: 2017-10-12 03:44

I have found that its not related to the size of the IAR file at all. I have attached here a simple single item IAR file that is only 1.7MB. It contains a ZHAO free AO which as far as I can see has full perms though some of the items have a creator that is a Simona.Stick variant as is common in OARs and IARs.

Yet when I do a load iar on this on OSGrid I get the never ending attempts to upload.. e.g.

10:19:18 - [Assets]: Upload failed: c541c47f-e0c0-058b-ad1a-d6ae3a4584d9 type 20 will retry later

(0032323)
aiaustin (developer)
2017-10-12 03:53
edited on: 2017-10-12 03:53

I tested the same simple single item load iar on my own AiLand grid and it uploads without failed asset upload issues or warnings there. I am beginning to suspect that OSGrid has some filter on upload content perhaps to protect against pirated items that bans certain avatar UUIDs (maybe all Simona Stick ones?)


- Issue History
Date Modified Username Field Change
2017-10-10 09:51 aiaustin New Issue
2017-10-10 12:16 watcher64 Note Added: 0032315
2017-10-11 00:25 UbitUmarov Note Added: 0032316
2017-10-11 00:26 aiaustin Note Added: 0032317
2017-10-11 00:27 aiaustin Note Edited: 0032317 View Revisions
2017-10-11 00:30 aiaustin Note Added: 0032318
2017-10-11 00:43 UbitUmarov Note Added: 0032319
2017-10-11 03:39 aiaustin Note Added: 0032320
2017-10-11 10:36 aiaustin Note Added: 0032321
2017-10-11 14:53 aiaustin Note Edited: 0032321 View Revisions
2017-10-11 14:54 aiaustin Note Edited: 0032320 View Revisions
2017-10-12 02:17 aiaustin Note Edited: 0032321 View Revisions
2017-10-12 02:17 aiaustin Note Added: 0032322
2017-10-12 02:17 aiaustin File Added: Upload-3.iar
2017-10-12 03:44 aiaustin Note Edited: 0032322 View Revisions
2017-10-12 03:53 aiaustin Note Added: 0032323
2017-10-12 03:53 aiaustin Note Edited: 0032323 View Revisions


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker