Automatic Region Backup

Basic Info

• Summary: A new module for periodically saving per-region backup OARs according to rules defined in Regions.ini.
• Developer Sponsor(s): allquixotic
• Start Date: February 18, 2011
• Branches Targeted: git master
• Commercial: No
• Code Repository: https://github.com/allquixotic/opensim-autobackup

High-Level Design

Idea

Currently, saving OARs in an organized manner requires significant hackery in the way of scripts or other external wrappers. This project is an attempt to get automatic OAR backup as a core capability of OpenSim.

• On a Per-Region basis, we want to give the user the opportunity:
  • To enable/disable automatic periodic backups of their region to an OAR.
  • To specify the length of time between automatic backups.
  • To invoke an external shell script or binary when a backup is taken (e.g. to transport to an offsite backup over FTP or S3).
  • To choose between overwriting a single file over and over, or saving a series of files whose name includes the timestamp of the OAR.
  • To enable or disable some simple logic that attempts to determine whether the sim is "busy", in which case no backup will be taken (because starting a backup can cause performance degradation).

Just to emphasize, all of these choices will be available completely separately for each region, regardless of whether the regions run in multiple instances or in one instance.

Ideally, users will be able to customize these settings in Regions.ini to conveniently specify when to take backups, and for which regions.

Criteria

• Scope: The primary focus of the feature will be a new region module that keeps track of when regions need to be OARed, and other various state as gleaned from Regions.ini.
• Software Requirements: No additional third-party dependencies or version bumps will be required.
• Impact: The only existing modules that could conceivably be affected are the IRegionArchiverModule, and any files dealing with the options in the Regions.ini config file.
• Blockers: No blockers yet, but I'm sure as time goes on...

Implementation Overview

• Proposed Module Name: IRegionAutoBackupModule
• Functionality Pseudocode:
  • Parse out the options from Regions.ini for each region
  • Set timers to wake up when the elapsed time for each region has expired. Optimization: Merge timers into one with a list of regions to process if the time is the same. Cuts down on the number of timers.
  • In a timer handler:
    • If the user wants us to check for busy conditions, then go through the conditions and make sure they all pass. Haven't determined what those conditions should be yet, but an obvious start is to set a reasonable avatar threshold, and examine the sim time dilation. If there are either "a lot" of avatars, or "a very low" time dilation, we probably don't want to kill the sim with an oar backup right now. But the user can bypass our logic if they want. Maybe we could add yet more options for the thresholds themselves, if we settle on e.g. time dilation and avatar count.
    • If we checked some conditions and one or more of them failed, put off the oar backup for (interval / 2) seconds and then try again, where "interval" is the original interval specified in Regions.ini, not the interval we were just asleep for.
    • Else: 1. Get the Scene object for the region(s) associated with this timer; get the IRegionArchiverModule interface; call HandleSaveOarConsoleCommand (or one of the other functions in the module?). The file name will either be the name of the region + ".oar", or the name of the region + a friendly-formatted timestamp + ".oar".
    • 2. Invoke the shell script associated with the region's backup event, as given in Regions.ini. Possible security issue, but then, allowing sysadmins to set threat level severe is not exactly secure...
    • 3. Reset the timer.

The only functions that other modules need to call into this one will be the initialization stuff: creating the implementation object and setting it up to do its work. Then it basically runs in an infinite loop, periodically doing stuff. I anticipate being able to do everything in a single thread, but I'm not sure how threading is worked into the other modules. In particular, I need to know if IRegionArchiverModule is thread-safe, or if I need to enter into a specific thread or take some locks to use it.

User Experience

This shouldn't be visible to end-users at all, but it will be very visible to OpenSim sysadmins. Sysadmins will edit Regions.ini to manage the config settings on a per-region basis. Maybe, for user-friendliness, we can have global config options in OpenSim.ini that serve as "default" options for the ones in Regions.ini, if they are not specified. The default will be to disable auto-backup altogether, but if a user enables auto-backup without specifying any other auto-backup config options, it'd be nice if they could set the rest of the options globally in OpenSim.ini. This would reduce the size of Regions.ini for very large numbers of regions, and make it trivial to merge the timers into one.

Low-Level Design

TBD! I am still learning about the OpenSim architecture, so pardon any layering violations / false assumptions I've made in my write-up :)