MantisBT - opensim
View Issue Details
0008247opensim[GRID] Asset Servicepublic2017-10-10 09:512017-10-12 03:53
aiaustin 
 
normalmajorsometimes
newopen 
PCWindows10
master (dev code) 
master (dev code) 
0.9.1 1684 59afeb6 (2017-10-06 01:43)
Grid (Multiple Regions per Sim)
BulletSim
.NET / Windows64
None
N/A
0008247: Possible difference in way load oar and load iar upload assets reliably
In testing a large upload of an OAR (1.5GB) for a new region and an IAR (2.3GB)for roughly the same content I see a very reliable save oar and save iar with no missing assets and no failures or warnings on my grid. Then moving the content to OSGrid, the load oar also goes up reliably with no warnings on the second grid. But the load iar starts to show a LOT of messages of the form

[Assets]: Upload failed ... will retry later

There are though the normal progress messages indicating the IAR contents are being loaded and I get a successful message when that is complete. But the Upload failed and retry message continue afterwards with the occasional pause.

In one test experiment these went on for over 12 hours after the message indicating the load iar had finished. And I am not yet sure if the assets are all in place on the second grid.

Is there a difference in the upload mechanism for load oar (which seems solid) and load iar (which does not) and can the load iar mechanism be altered to be like the load oar one if so?
No tags attached.
? Upload-3.iar (2,420,936) 2017-10-12 02:17
http://opensimulator.org/mantis/file_download.php?file_id=4686&type=bug
Issue History
2017-10-10 09:51aiaustinNew Issue
2017-10-10 12:16BillBlightNote Added: 0032315
2017-10-11 00:25UbitUmarovNote Added: 0032316
2017-10-11 00:26aiaustinNote Added: 0032317
2017-10-11 00:27aiaustinNote Edited: 0032317bug_revision_view_page.php?bugnote_id=32317#r6440
2017-10-11 00:30aiaustinNote Added: 0032318
2017-10-11 00:43UbitUmarovNote Added: 0032319
2017-10-11 03:39aiaustinNote Added: 0032320
2017-10-11 10:36aiaustinNote Added: 0032321
2017-10-11 14:53aiaustinNote Edited: 0032321bug_revision_view_page.php?bugnote_id=32321#r6442
2017-10-11 14:54aiaustinNote Edited: 0032320bug_revision_view_page.php?bugnote_id=32320#r6444
2017-10-12 02:17aiaustinNote Edited: 0032321bug_revision_view_page.php?bugnote_id=32321#r6445
2017-10-12 02:17aiaustinNote Added: 0032322
2017-10-12 02:17aiaustinFile Added: Upload-3.iar
2017-10-12 03:44aiaustinNote Edited: 0032322bug_revision_view_page.php?bugnote_id=32322#r6447
2017-10-12 03:53aiaustinNote Added: 0032323
2017-10-12 03:53aiaustinNote Edited: 0032323bug_revision_view_page.php?bugnote_id=32323#r6449

Notes
(0032315)
BillBlight   
2017-10-10 12:16   
I have loaded much larger oars and iars, but on linux .. It could be possible you are running into windows file locking issues, or even file limitations ...

Just my .02 as I can't speak to actually having this issue ..
(0032316)
UbitUmarov   
2017-10-11 00:25   
just a "minor" difference...
OAR is basically a local operation, assets are stored on region dbs, etc.
IAR is a grid inventory/assets operation. Contents are stored on grid dbs
( of course minor means major ;) )
(0032317)
aiaustin   
2017-10-11 00:26   
(edited on: 2017-10-11 00:27)
Thanks for that suggestion @watcher64.. and I agree I also have uploaded much larger IARs in the past also. Something really is odd as the OpenSim.exe instance I am loading the IAR from is idle much of the time and goes idle between the retry bursts... but things never seem to complete the upload.

@Ubitmarov.. I assume that the load OAR though does talk to the asset server to upload any new assets that are not already on the target grid? But it could be that the assets are already in place from others or earlier uses of course.

I left things running overnight and its still re-trying every hour or so and reporting failures.

I am reporting this so anyone familiar with the load oar and load iar mechanism can check if there is some difference in the way the uploads work. the (pretty much the same) content in the OAR loaded absolutely solidly with no error or warning. But the load iar for a directory containing essentially the same assets fails... even though I would imagine almost all the assets are actually already in place on the target grid (OSGrid) due to the prior load of the OAR for a region with the assets on it.

(0032318)
aiaustin   
2017-10-11 00:30   
The upload never seems to complete and I am getting the retry warning every hour or so even after leaving things 12 hours. Still going as we speak for a load iar of just under 4000 assets (700 or so inventory entities). I am not sure if to quit from the OpenSim.exe or leave it to see if it ever completes.
(0032319)
UbitUmarov   
2017-10-11 00:43   
the assets in queue for upload retry will be lost if you stop the instance, or after 48 hours of retries (or similar time )
(0032320)
aiaustin   
2017-10-11 03:39   
(edited on: 2017-10-11 14:54)
I thought so. That's why I left it running for over 12 hours but it never seemed to clear the backlog. Very odd it goes on so long. And something I have not noticed before. I will keep an eye open to see if this occurs again and try to better ascertain the circumstances.

(0032321)
aiaustin   
2017-10-11 10:36   
(edited on: 2017-10-12 02:17)
Could it be that the message is given in the situation where the asset is already in the (OSGrid in tis case) asset server with the identical asset and UUID?

Or even that the asset (or creator avatar) is "banned" to upload to the asset server as inappropriately copied in some way? Is that a features of some asset servers such as OSGrid?

(0032322)
aiaustin   
2017-10-12 02:17   
(edited on: 2017-10-12 03:44)
I have found that its not related to the size of the IAR file at all. I have attached here a simple single item IAR file that is only 1.7MB. It contains a ZHAO free AO which as far as I can see has full perms though some of the items have a creator that is a Simona.Stick variant as is common in OARs and IARs.

Yet when I do a load iar on this on OSGrid I get the never ending attempts to upload.. e.g.

10:19:18 - [Assets]: Upload failed: c541c47f-e0c0-058b-ad1a-d6ae3a4584d9 type 20 will retry later

(0032323)
aiaustin   
2017-10-12 03:53   
I tested the same simple single item load iar on my own AiLand grid and it uploads without failed asset upload issues or warnings there. I am beginning to suspect that OSGrid has some filter on upload content perhaps to protect against pirated items that bans certain avatar UUIDs (maybe all Simona Stick ones?)