MantisBT - opensim
View Issue Details
0006820opensim[GRID] Grid Servicepublic2013-10-28 05:482021-11-16 03:06
Richardus Raymaker 
normalminorhave not tried
master (dev code) 
Grid (1 Region per Sim)
.NET / Windows64
Singularity (5220)
0006820: Maptiles dont get deleted from robust grid server after clean region shutdown.
When you run a grid with robust as server, and then shutdown some region the clean way by typeing 'shutdown' on the opensim console. the maptiles from the region dont get removed from the robust maptile directory.

Solution, after deregistrating the region
on the robust grid server, delete the maptiles from the directory.
No tags attached.
related to 0008314new  Map tile generation from robust 
Issue History
2013-10-28 05:48Richardus RaymakerNew Issue
2013-10-29 11:03Richardus RaymakerNote Added: 0024583
2013-10-29 11:15Richardus RaymakerNote Added: 0024584
2013-10-29 11:16Richardus RaymakerNote Edited: 0024584bug_revision_view_page.php?bugnote_id=24584#r1684
2019-02-09 03:30tampaRelationship addedrelated to 0008314
2019-02-10 09:19Gavin HirdNote Added: 0034703
2021-10-28 12:21tampaNote Added: 0038188
2021-10-28 12:21tampaStatusnew => acknowledged
2021-11-12 15:35Ferd FrederixNote Added: 0038221
2021-11-15 08:07tampaNote Added: 0038228
2021-11-15 20:37Ferd FrederixNote Added: 0038229
2021-11-16 03:06tampaNote Added: 0038230

Richardus Raymaker   
2013-10-29 11:03   
Other idea would be to have option to clean automatic maptiles that dont exist anymore in the database. maptile directory compare with coordinates in database ? or nameing ?
Richardus Raymaker   
2013-10-29 11:15   
(edited on: 2013-10-29 11:16)
Note: this happens when i shutdown the region. rename and move the region to new location. the region.ini filenames are based on coordinates. so when the are moveing to new location the change.

Gavin Hird   
2019-02-10 09:19   
I don't think you would want to immediately delete the map tiles on a clean shutdown. Often regions are restarted shortly after, and having to regenerate the map-tiles slows down region startup.

A better approach would be to have an option on shutdown such as -delmaptile which would immediately delete it.
Otherwise it would be cached for a reasonable amount of time such as 48 hours before being purged if the region went awol.
2021-10-28 12:21   
The problem is there is no deletion going on, what would be possible is sending the "water" back over as new maptile just before shutdown.

Though this overall just relates to our simpleton Robust who cannot clean up after itself in regards to maptiles.
Ferd Frederix   
2021-11-12 15:35   
Maps are normally off due to the extensive RAM and CPU hit. Please do not delete them automatically.

"Shutdown" and "q" or "quit" are the same commands. They just stop the region.

Deregistering is done when the region exits if flags are set before the region boots, or by the command line. For example, marking a region as "persistent" will not deregister the region. This flag is used when regions and maps need to remain for long periods of time. OsGrid works this way, and Often some 4,000 other grids running DreamGrid will, with its Smart Boot capability that requires regions to remain online even if shut off.

The command line idea by Gavin Hird is useful if the region is still there. Often it will not be as they will type quit and now there is no way to delete the region maps.

External tools can easily do this, such as DreamGrid, which removes the INI, the maps, deregisters the region, and deletes the primitems when you click "Delete Region".
2021-11-15 08:07   
I had a look at this today and did end up modifying the shutdown procedure to upload an empty water tile for the region just before shutdown. This does bring the effect of Robust redoing the maptiles for that location. It still doesn't delete the unused tile from the database nor the files as that is async from the simulator uploading the maptile.

The next step I will try is to have Robust kick off a routine when it gets a deregistration to check whether it can delete the terrain image from the database so as to reduce the spam there. This is a bit more difficult based on how they are stored in regards to file based assets for example.

Also have not touched the maptiles locally to robust, not sure how that will work, for the most part only the local tile for the region should be removed, the other scales should have received the previously uploaded water tile. Main issue is the async nature of that whole processing and I haven't even looked at how the scale generation works. If it ends up using the actual files this might be a lot more difficult as deletion before a new scale is done would make the map inconsistent.

Now in regards to "persistent" regions I really have no sympathy for causing waste on the grid side of things. A grid doesn't exist to bend over backwards for those connecting regions to it and it certainly is the task of any good grid admin to maintain their systems, which ultimately includes database and storage maintenance. In the most core function there OpenSim should aid in the task and not produce even more garbage to be cleaned up manually. This entire idea of "reserving" spots and all that is a lot better handled externally and if uploading maptiles becomes a bandwidth issue then there are other concerns to be had. Provisions for regular interval maptile generation even exist still, surely with the idea in mind to have the map actually reflect what is on regions.

Making a mess on the map and abusing the nature of the maptile system to leave behind maptiles of dead regions is not by design at all. There exists the static maptile system to set a maptile manually and reduce the load on the region. This idea of reserving coordinates and all that is beyond silly. I really don't think that this should be in any form consider as a feature. The mess that all this creates would have the entire system being thrown out and rewritten elsewhere, it is not in an acceptable state at the moment.

Having Robust go through the regions every hour or so see if they still respond and booting them off the grid replacing the maptile as well would actually be more consistent with the entire concept of garbage collection. As such deletion doesn't need to be issued from the simulator side, but this does involve accepting that a region running on a dial up modem unable to reply to an xml request within ten seconds just cannot be considered "connected" and seemingly some object to that idea.

The detrimental effects of slow regions and various waits and timeouts, as well as filling up queues and whatnot cache seems to have entirely slipped here I feel. I went quite deep down that rabbit hole recently and I have to agree with some of the things Ubit likes to remind me of in this regard. OpenSim was not designed with some of these scales in mind and even though they work they most certainly won't work well. Fixing the latter is down to accepting that within a networked distributed structure such as a grid certain boundaries must exist. After all when you setup microservice structures you kick out containers that have crashed or are slow to respond as well. Sitting around on a failing structure that keeps deteriorating and viewing that as feature is asking for trouble in the long term.

Cleaning this mess is in the intend of the system given the residual code hinting at such. Whether this ultimately would include logic to prevent overly aggressive cleaning remains to be seen. The nature of the project means those that elect to do something out of spec and with the severe consequences that arise from disabling such cleaning routines are ultimately free to remove these things or disable them, but out of the box the rightful expectation of any piece of software is to not be a source of a continuous headache from manually cleaning the mess it leaves behind each time it is run. That sort of thing would get it kicked off any linux repo or definitely not well graded for your masters degree in software engineering.

I mean this is equivalent to the issue with the mediametadata service downloading and creating thousands of music album thumbnails filling up the temp folder, MS considered that a bug too. Can't believe I have to draw that comparison, but even MS used to have standards in this regard... used to.
Ferd Frederix   
2021-11-15 20:37   
IMHO, any region that is registered, or not, should never have a map removed automatically without human involvement, or an external controller that understands the implications. Otherwise Opensim would be "out of spec".

An unregistered region does not mean it is deleted, or that its map needs to be deleted. It means offline, which is perfectly normal. The default INI is set this way. The map then shows the grid owner where it was and will be when booted. But its deactivated - clicking it does not work. Unregistered may also mean another region is taking that spot soon. In this case, deleting the map is unnecessary as it will be replaced.

Grids use the "persistent" flag to keep the region registered when it stops, so region places cannot be stolen away. LBSA plaza in OsGrid could be replaced in an instant otherwise. The thousands of regions there would be chaos without this flag keeping regions registered.

In addition, about 96% of all grids that were online this year can be set to use "persistent". With it set, the region is booted, once, to set the flag, and get it registered. The region is then powered off when no one in in it. Registration remains as it was marked "persistent". Anyone can then click on the map, The viewer and Opensim initiate a Teleport, and the controller initiates a power up and then a teleport. This is standard operating behavior in Opensim and it uses standard modules such as Region Ready and a MIT licensed function by Ubit and I that allows Opensim to redirect teleport to the controller.

Lastly, making maps should be rare. No PCs has the resources to make maps every time a regions boot. This requires about 2X the normal RAM to hold the map making process, plus the CPU's and disk to read every single item in a region to make Yet Another Identical Map, while still never deleting the old map from the db. DreamGrid's auto memory and CPU throttles can handle this even with 300 regions on an 8 GB Windows PC (or a super fast Mac M1), it still greatly slows the time to get the grid ready the first time.

I would think it is okay to delete the maps of "dead" regions, given some proper definition of "dead' and a method to do this. Deregistered regions do not meet that definition. I would hate to make maps for my 200+ regions just because I moved them slightly, especially when an external tool such as Dreamgrid is capable of moving the map just as easily as it can move a region.

What could work on Robust, would be to modify the command "delete-region <name> - Delete a region from disk.". This command is already "out of spec", i.e., broken as it does not support the INI file options that move the region INI to a different place. Neither does create region.

A fixed command or a new one 'delete-region-all <name>' could certainly delete the map. It could also delete the database entries which are the really massive bloat in the system, and then delete the INI file. Maptile deletion is also easy. My code to delete the map looks like this:

    Public Sub DeleteMaps(RegionUUID As String)

        Dim path = IO.Path.Combine(Settings.OpensimBinPath(), "maptiles\00000000-0000-0000-0000-000000000000")

        For i = 1 To 8
            For XX = 0 To SizeX(RegionUUID) / 256
                For YY = 0 To SizeY(RegionUUID) / 256
                    Dim x1 = CoordX(RegionUUID) + XX
                    Dim y1 = CoordY(RegionUUID) + YY
                    Dim file = IO.Path.Combine(path, $"map-{i}-{x1}-{y1}-objects.jpg")
                    Debug.Print($"Delete map-{i}-{x1}-{y1}-objects.jpg")
    End Sub

DreamGrid has a "Delete Region" button that calls this function, and it also has a temp-region function that deletes temp regions when you leave them. These types of regions fill in areas that you can fly, drive or boat around in when you use Endless Land and Sea settings to make land and plants procedurally. Tens of thousands of miles of land are possible, with almost no disk and Mysql impact.

These functions can also delete data from various region tables, where massive bloat otherwise occurs. For Standalone you would have to also delete the corresponding robust tables. For this next example my parameters are the UUID, RegionDB.table_name, and of column for the uuid of the region. Opensim tables are a hot mess of names like these. A DBIX::Class structured delete would be more like this, assuming there was a schema with 'has', 'has_many', and 'might_have' defined.

$schema->resultset('Opensim')->search({uuid => $param,})->delete_all;

But without Perl, we need this next code, where DeleteContent is a parameterized Mysql Delete. Or a 2-strided list and one call passing it as a param.

        DeleteContent(regionUUID, "primshapes", "uuid")
        DeleteContent(regionUUID, "bakedterrain", "regionuuid")
        DeleteContent(regionUUID, "estate_map", "regionid")
        DeleteContent(regionUUID, "land", "regionuuid")
        DeleteContent(regionUUID, "prims", "uuid")
        DeleteContent(regionUUID, "primitems", "primid")
        DeleteContent(regionUUID, "regionenvironment", "region_id")
        DeleteContent(regionUUID, "regionextra", "regionid")
        DeleteContent(regionUUID, "regionsettings", "regionuuid")
        DeleteContent(regionUUID, "regionwindlight", "region_id")
        DeleteContent(regionUUID, "spawn_points", "regionid")
        DeleteContent(regionUUID, "terrain", "regionuuid")

And you need these:

        DeleteMaps(regionUUID) //A tiny piece of this, see above
        DeregisterRegionUUID(regionUUID) //just what you expect
        Delete_Region_Map(regionUUID) // DreamGrid specific maps

The last line deletes a full size region map which is a single image, such as at, [^] which is also in DreamGrid. Take a peek at Alexandria which is about 4 million sq feet in one region. Now make that map. Without maps, an old $300 server can get me there in less than 45 seconds from cold boot. With maps is more like 5 minutes.
2021-11-16 03:06   
Had a lengthy response, but I feel like I am barking up the wrong tree. So in shorter terms, osgrid or third party projects don't dictate OpenSim spec, even SL spec has limits on what can reasonably be implement or constitutes bad design(bake on server pfft). OpenSim cleaning up after itself is the clear intention within the system, residual code shows as much and making a huge mess everywhere has never been specification either. Constructs of reserved grid coordinates and other "persistence" is down to operators and their approach, OpenSim does not dictate how that should be handled or what the conditions are for it. That's a design discussion if anything and not really within the scope of this bug report.