User:JeffKelley/Automating grid management

From OpenSimulator

Jump to: navigation, search

THIS IS A DRAFT


Here is one of many solutions to manage a Linux grid from one unique bin folder.

Contents

OpenSim configuration

First, we parametrize our configuration files to use shell environment variables. We will use only one variable named SIMULATOR. The expression ${Environment|SIMULATOR} can then be used in various parts of OpenSim.ini and GridCommon.ini. 

[Environment]
    SIMULATOR=""
 
[Startup]
    ConsolePrompt = "Simulator ${Environment|SIMULATOR} (\R) "
    regionload_regionsdir = "path_to_regions/simul${Environment|SIMULATOR}"
    PIDFile = "/tmp/grid/simu${Environment|SIMULATOR}.pid"
 
[Network]
    http_listener_port = 900${Environment|SIMULATOR}

With such a configuration, the listening port for simulator N will be 900N. [ Note: We assume a one-digit simulator number. If you need more than 10 simulators, you may use 90${Environment|SIMULATOR} and pass a two-digit SIMULATOR variable. ] The regions file for simulator N is located inside path_to_regions/simulN, so you should have this structure on disk : 

path_to_regions
   simul1
       Regions.ini
   simul2
       Regions.ini
   ......

If you are using separate databases for each simulator (this is a design choice), then you can parametrize the ConnectionString in GridCommon.ini this way : 

[DatabaseService]
    ConnectionString = "Data Source=localhost;Database=sim${Environment|SIMULATOR};User ID=xxx;Password=xxx;"


Using GNU screen

At the hearth of the management system is the launch script. We take care to label each session with the -S option. 

#!/usr/bin/perl
 
# Starting a grid process
# Usage : launch [-d] task_name (robust, simulN)
 
use strict;
use Getopt::Std;
use POSIX qw(strftime);
 
my %OPTS; getopts ('d', \%OPTS);
my $dflag = '-d' if %OPTS->{d};
my ($proc, $dir, $exe, $arg);
 
$proc = $ARGV[0];
$bin = '/home/me/grid'; # Where is the bin folder
 
$exe = 'Robust.exe'	if $proc =~ m/robust/;
$exe = 'OpenSim.exe'	if $proc =~ m/simul(.*)/;
 
$arg = 'Robust.HG.ini'	if $proc =~ m/robust/;
$arg = 'OpenSim.ini'	if $proc =~ m/simul(.*)/;
 
die "Invalid proccess\n" unless defined $exe;
 
my $now = strftime "%F %H:%M:%S", localtime;
printf  "$now Launching $proc\n";
 
chdir $bin or die "Cannot chdir $bin";
 
$ENV{SIMULATOR} = $1 if $proc =~ m/simul(.*)/;
$ENV{PATH} .= ':/usr/local/bin'; # MONO is here
$ENV{LANG}='C';	# Pour le point décimal
 
my $exe = "mono $exe -inifile=$arg";
my $cmd = "screen -S $proc -t $proc $dflag -m $exe";
print "$cmd\n"; exec $cmd;

Launching the grid is a matter of issuing this sequence of commands : 

launch robust
launch simul1
launch simul2
.............

Then: 

$ screen -ls
There are screens on:
	26039.simul6	(Multi, attached)
	24972.simul4	(Multi, attached)
	24676.simul1	(Multi, attached)
	24652.robust	(Multi, attached)
	24750.simul2	(Multi, attached)
	29220.simul3	(Multi, attached)
	25481.simul5	(Multi, attached)
8 Sockets in /var/run/screen/S-jyb.

This is not very friendly, so we write the procs script 

#!/usr/bin/perl
 
# List running grid processes
# Only detached if -d
 
use Getopt::Std;
 
getopts ('d', \%OPTS);
$detached = $OPTS{d};
 
@screens = split "\n", `screen -ls`;
@wanted;
 
for (@screens) {
 	next unless $_ =~ /(\S*)\.(\S*).*\((.*)\)/;
	($pid,$name,$attr) = ($1,$2,$3);
	$is_detached = ($attr =~ /detached/i);
	push @wanted,$name unless $detached && !$is_detached;
}
 
for (sort @wanted) {
	print "$_\n";
}

Then: 

$ procs
robust
simul1
simul2
simul3
simul4
simul5
simul6

Automating

Create a file named proclist: 

# processes to launch
 
robust
simul1
simul2
simul3
simul4
simul5
simul6


It is easy now to write a couple of scripts that launch the next wanted process, or re-attach the next detached process.


Script s : 

#!/usr/bin/perl
 
# Launch first missing process from proclist
 
use strict;
 
my @running = `procs`; # List of running screens
 
# Get wanted processes list
 
my $interactive = -t STDIN && -t STDOUT;
my $dflag = '-d' unless $interactive;
my $proclist = "$ENV{HOME}/proclist";
 
open IN, "<$proclist" or die "Cannot open $proclist\n";
 
my @procs = <IN>;
 
# Find first not running process
 
for my $proc (@procs) {
	chomp $proc;
	next if $proc eq '';
	next if $proc =~ /^#/;
 
	my $running = grep /$proc/,@running;
	exec "bin/launch $dflag $proc" unless $running;
}
 
print "Nothing to start\n" if $interactive;


Script r : 

#!/usr/bin/perl
 
# Reattach first detached screen process
 
@detached = `procs -d`; # List detached screens
die "Nothing to reattach\n" unless @detached;
 
$process = shift @detached;
 
printf "Reattaching $process\n";
sleep 1; exec "screen -r $process";

Starting your grid is now as simple as opening as many terminals as you have grid processes and typing s in each. This can be further automated by having a cron job calling s every minute. The job will launch any missing process from proclist. As an added benefit, it will relaunch crashed simulators.

Adding a simulator

Add a line to proclist 

# processes to launch
 
robust
simul1
simul2
simul3
simul4
simul5
simul6
simul7

Create a folder path_to_regions/simul7 and put the Regions.ini inside.

That's all.

Retrieved from "http://opensimulator.org/index.php?title=User:JeffKelley/Automating_grid_management&oldid=41188"
Personal tools
General
About This Wiki