MantisBT - opensim
View Issue Details
0008439opensim[REGION] OpenSim Corepublic2019-01-01 07:582020-08-10 01:55
aiaustin 
 
normaltweakalways
newopen 
PCWindows10
master (dev code) 
master (dev code) 
opensim-0.9.0.1-685-g9c043fe.zip
Grid (Multiple Regions per Sim)
BulletSim
.NET / Windows64
None
N/A
0008439: Region maptiles - can previous tiles be removed from database as they are replaced by later versions
Region maptiles are stored in the opensim.assets data table each time they are produced. But, as far as I understand, previous copies will never be used again. A good time to tidy the data base would seem to be at exactly the time that a new tile is stored, with older versions for that region UUID being deleted at that time.

Note that at present I occasionally do the following mysql console command while the grid is stopped and sometimes thousands of these are deleted... even on a simple grid.

DELETE FROM opensim.assets WHERE name LIKE "terrainImage_%";
No tags attached.
Issue History
2019-01-01 07:58aiaustinNew Issue
2019-01-01 07:59aiaustinDescription Updatedbug_revision_view_page.php?rev_id=7478#r7478
2019-01-01 10:51aiaustinDescription Updatedbug_revision_view_page.php?rev_id=7479#r7479
2019-02-05 13:37tampaNote Added: 0034325
2019-02-05 14:58UbitUmarovNote Added: 0034326
2019-02-05 14:59UbitUmarovNote Edited: 0034326bug_revision_view_page.php?bugnote_id=34326#r7766
2019-02-05 15:10tampaNote Added: 0034327
2019-02-05 15:12UbitUmarovNote Added: 0034328
2019-02-05 15:20tampaNote Added: 0034330
2019-02-05 15:20UbitUmarovNote Added: 0034331
2019-02-05 15:28tampaNote Added: 0034332
2019-02-06 02:06aiaustinNote Added: 0034334
2019-02-06 07:56paela argusNote Added: 0034337
2020-05-09 22:01Ferd FrederixNote Added: 0036457
2020-05-09 22:03Ferd FrederixNote Added: 0036458
2020-05-10 03:03aiaustinNote Added: 0036465
2020-05-10 04:15aiaustinNote Edited: 0036465bug_revision_view_page.php?bugnote_id=36465#r9023
2020-05-10 04:16aiaustinNote Edited: 0036465bug_revision_view_page.php?bugnote_id=36465#r9024
2020-08-09 22:21tampaNote Added: 0036678
2020-08-10 01:09aiaustinNote Added: 0036679
2020-08-10 01:10aiaustinNote Edited: 0036465bug_revision_view_page.php?bugnote_id=36465#r9136
2020-08-10 01:55tampaNote Added: 0036680

Notes
(0034325)
tampa   
2019-02-05 13:37   
Actually a very good question. The map tiles seem to be entered into the database as new asset even if their hash is identical. I believe they are in there for robust to be able to serve them directly, but why it creates new entries each time... strange.

I have fields for the creation date in my tables, perhaps a query to clear all but the newest tile would make more sense, then again ideally the entry in the database would be updated instead of new ones being created.
(0034326)
UbitUmarov   
2019-02-05 14:58   
(edited on: 2019-02-05 14:59)
commands to delete old terrain image and parcel image are actually issued on map generation.
but they are currently ignored unless you set AllowRemoteDelete = true on robust.ini.
AllowRemoteDeleteAllTypes should stay false

(0034327)
tampa   
2019-02-05 15:10   
Doesn't AllowRemoteDelete also enable deletion of other assets though or just the maptiles?

Why do we still store them that way if they are stored in maptiles folder, just for v1 viewers? Should those still be supported at this point?

I just issued a delete on all these entries I have on my assets table, so far nothing has caught fire, you sure these are actively used still?
(0034328)
UbitUmarov   
2019-02-05 15:12   
AllowRemoteDeleteAllTypes should stay false <--- or all types can be deleted
(0034330)
tampa   
2019-02-05 15:20   
What was the reasoning for setting the default to false?
(0034331)
UbitUmarov   
2019-02-05 15:20   
reason lost in time..
(0034332)
tampa   
2019-02-05 15:28   
So then why not change this and prevent blowing up asset tables with useless older versions of maptiles that are never even going to be used even if they happen to be requested by a v1 viewer, because I suppose that also only fetches the most current version?
(0034334)
aiaustin   
2019-02-06 02:06   
Thanks for pointing out this Robust[.HG].ini setting which I did not know about. I changed it to AllowRemoteDelete = true on the Openvue and Ailand grid.

Would it be worth setting this parameter to true by default now as these data base entries do build up.
(0034337)
paela argus   
2019-02-06 07:56   
because if you use fsassets is not needed AllowRemoteDelete = true
use old db think for assets at this years why ?
(0036457)
Ferd Frederix   
2020-05-09 22:01   
I had 7657 maptiles. I set AllowRemoteDelete = true in Robust.HG.ini. After clearing maptiles and restarting and making the highest version tiles, I still had 7654 maptiles in robust after all maps were regenned. AllowRemoteDelete = true had no effect.

mysql> select count(*) FROM robust.assets WHERE name LIKE "terrainImage_%";
+----------+
| count(*) |
+----------+
| 7657 |
+----------+
1 row in set (25.36 sec)
(0036458)
Ferd Frederix   
2020-05-09 22:03   
Also very time consuming:

mysql> delete FROM robust.assets WHERE name LIKE "terrainImage_%";
Query OK, 7657 rows affected (23 min 35.96 sec)
(0036465)
aiaustin   
2020-05-10 03:03   
(edited on: 2020-08-10 01:10)
Fred, I just checked my Openvue and AiLand grids and the following SQL gives exactly the number of tiles I would expect, one per region, so the deletes are working correctly in my 0.9.2.0 dev master setup and have been for some time.

SELECT * FROM opensim.assets WHERE name LIKE "terrainImage_%";

One difference I can see may be relevant. I run with a single data base "opensim" for each grid. Each grid does have several servers for the add on regions, so that is not a factor. Could it be that robust.assets old map tiles are not being cleared out and that the clear out only works if its in the same database as used for the region?

(0036678)
tampa   
2020-08-09 22:21   
I think austin is right, at least in regards to the database setup it seems only on standalone-like setups that setting actually deletes anything. I tried the other setting as well to delete all types, when purging inventory nothing happened. It seems those settings don't work for grid setups with connectors.

I had a look in my logs, I have about 3 months worth of logins from about 900 unique users, none of which used a v1 viewer. Probably going to patch v1 maptiles out to stop my database filling up. That's at least my opinion.

The system is not really thought out, as mentioned, it runs on region startup and has to if you want to keep maptile generation at all. That's hardlocked though and simply disallowing generation at startup while keeping the ability to manually issue a generate map is possible.

Not sure what direction is best for this, but the current setup is a bit of a mess, sorry to be so blunt.
(0036679)
aiaustin   
2020-08-10 01:09   
The phrase “standalone-like setups” might confuse some. as we use the term “standalone” for non-Robust installations. The old map tiles are properly deleted on Robust grids too... but my example grids where that applies use a single MySQL database for core service and assets, these are not split into separate data bases as allowed by OpenSim configuration options.
(0036680)
tampa   
2020-08-10 01:55   
I use a connector for assets rather than saving them to database, I don't see the deletions causing the error they should cause if they were sent, as the asset server itself has no delete it should produce a 404, but there is nothing. Guessing the connector does not support deletion only the localservice does?

I'll deal with this with some bash and do a deep clean manually, but that's not a long term option :S