[Opensim-users] How to get ROBUST to notify that it has finished setup?, (Fred Beckhusen)

[Fred Beckhusen]

I do much of this in DreamGrid.   MySQL, Robust and Opensim each have 
different ways that you can detect being "up".

You are correct in that Linux spawns processes (not services, which are 
a Windows construct) and detecting the process is running is not what 
you need.    You need to know when the process is ready to use.

MySQL:   MySQL is ready if you get a response back from a Select 
Version();  query.   Socket connectivity to port 3306, or even a login 
to it is not enough. Success is determined in my app if the 
Response.length  > 0 . I don't check the version as I don't care and 
it's NaN anyway. There may be no Robust or Region database at first 
boot. They will make the databases they need.

Robust: I used to just probe the robust HTTP port for a 200 OK  HTTP 
response.  But when robust is first building the database, this is not 
correct.   A slow PC could take too long to make the robust database and 
a launched Opensim would fail to connect.  Ubit added a HTTP response 
code to my code which is mostlycore Opensim.  Robust is MIT licensed so 
feel free to use it.  The source is in the Dreamworld repo at 
github.com/Outworldz/Dreamworld.  gitk can help locate the spot of any 
changes to core. Ubit may be able to point you in the correct place.

DreamGrid uses a URL in my file Robust.vb in function IsRobustRunning() 
As Boolean, which uses a parameter to ask for robust readiness.

Up = Client.DownloadString("http://" & Settings.PublicIP & ":" & 
Settings.HttpPort & "/index.php?version")

Typically, this would translate to http://127.0.0.1:8002/index.php?version

If the Up string contains "Opensim", I mark Robust as up and launch all 
Opensims sequentially (not in parallel, though you could fork and do 
so).  I want to control the boot more closely.  As soon as Opensim is is 
spawned I try to back a Process ID, and then I launch another after 
checking out the CPU and RAM.   A PID can take anywhere from 
milliseconds to 10-15 seconds ( possibly much longer) to get so I have 
to wait for it.  I need the PID to locate the instances later.  You 
could use the PID file that Opensim makes on disk, but then you must 
know all the regions by parsing files,  and delete the .pid file before 
the launch, and that brings in more headaches in detecting a running 
Opensim.  They may be still shutting down, or they restarted,  or never 
shut down, or was left running deliberately by the Grid owner.  All 
those edge cases are where I spend a lot of my time. This requires a 
complicated state machine to track it.

Opensim is a hot mess.  I spawn them as a process, and so can detect an 
exit by using  withevents, and if desired, restart them ( typically 
initiated by a use request, as the reboot a region should not be used. 
You need to exit and respawn the instance.

And detecting "Up" may be impossible.  The Region Ready module exists to 
report that Opensim is up.   But it's not always correct.  If scripts 
are off, you get nothing from the module. Also, if Logins are disabled, 
there will never be a  Region Ready.  Worse, after Login Enabled is 
reported on the console, and Region Ready says the Region scripts are 
done, as long as a minute or more, you can still get a Teleport Denied 
because the region is offline.     And that even when Robust has it 
marked as Online!  Its something deep in Opensim I still need to fix.

Opensim will be CPU bound at boot as it is extremely thread happy.  
MySQL will rarely go above a few percent CPU even with 100% of the RAM 
in use with many regions launched at once.  The CPU will get swamped, 
especially if maps are on, Maps can double RAM use and extend the CPU 
use for many minutes, versus the more normal few seconds it takes.

DreamGrid gets the average CPU and RAM over 3 second periods, and stops 
down spawning Opensim processes when CPU > 90% or memory > 90%.  It 
begins to spawn again when the average drops below these threshholds.  
This leaves a bit of both left over which makes the system much more 
responsive. I also support Core Affinity for Opensim so you can select 
the cores to use,  and am experimenting with using just once core during 
boot and then using more as it becomes stable.   Lots to play with 
here.  I will be adding Core Affinity as a choice for Robust and Mysql soon.


--ooo------/\/\/\-----|(------ooo------/\/\/\-----|(------ooo----

Fred K. Beckhusen