OpenSim Services and Service Connectors

Motivation

The OpenSim framework is becoming so many things to so many people that it has reached the point where the original architecture of the software itself, largely inspired by what the SL Client dictates and by the Linden Lab grid, is hampering progress on all those fronts. We need to rethink the architecture of the software in order to be able to support the variety of things that people want to do with OpenSim without having to go through lengthy design discussions and negotiations for it to do "the right thing", which obviously doesn't exist. For some, "the right thing" is a server infrastructure that is a 100% reproduction of Second Life; for others, "the right thing" is a 3D server infrastructure that is 100% compatible with the Web; for others, "the right thing" is a desktop 3D simulator; and for many others "the right thing" is many points in between.

This proposal comes in the sequence of an email thread on the -dev mailing list: https://lists.berlios.de/pipermail/opensim-dev/2009-April/006325.html A large part of what's described here is already in place in trunk, but the only connector in place is the one for the Asset Service.

Proposal Summary

We propose a new software architecture that can easily accommodate all the system architectures that people want to build. The basic concepts of this new software architecture are (1) Service Interfaces; (2) Service Connectors; (3) Service Implementations; and (4) Server Shells. Service implementations are pieces of code that can be loaded into a server shell, along with their "in" service connectors that receive service requests from clients. Clients access services through service interfaces, and by loading the corresponding "out" service connectors that send requests to the service implementations. The figure above illustrates the general concept.

A Service is the collection of its implementation and its connectors, and obeys a certain service interface. The people that implement the service also implement its connectors. Therefore, the details of the protocol between the client-side and the server-side are invisible to the service clients, which only know about the service interface. As long as the interface is preserved, switching service implementations is a trivial matter of replacing the out connector in the client. The specification of all of these elements is done at initialisation time, by loading the specified DLLs and/or by using a generic plugin infrastructure.

Benefits

The figure below shows 4 different system architectures that can be trivially configured if the proposed software architecture is in place.

Note that these are only 4 out of a variety of combinations.

The benefits of this software architecture are the following:

• We move away from the current boolean architectural choices given by the variable "gridmode=true|false", and into a model where simulators can be configured in a variety of hybrid modes.
• The dynamic loading of service connectors into the simulator allows a variety of protocols "on-the-wire" between the simulator and the services without having to change the simulator code. (As long as that service implementation can honor the established service interface for it)
• [Corollary of the previous] All discussions about simulator-service protocols (HTTP vs XMLRPC vs ...) become irrelevant to the OpenSim framework itself. Opensim will continue to provide a few service implementations and their connectors out-of-the-box, but anyone can replace those with their own without having to make any changes to the source code of the Simulator, by making the corresponding connector DLL available to the simulators.
• All OpenSim servers become server shells of the same type. That is, they are all configured through the .ini mechanism, and they can run any of the services and service connectors. Therefore, it also becomes trivially possible to have system architectures where some resource service uses another resource service directly. For example, the Inventory server may use the Asset server in order to provide a combined IInventoryService+IAssetService interface to viewers.

Implementation of the Proposed Software Architecture

The implementation is illustrated using the Asset Service as an example. Here is the overview of the new organization:

OpenSim/

  Region/

    CoreModules/
      ServiceConnectors/
       (place to put all OUT service connectors from the simulator)
        Asset/
          HGAssetBroker.cs
          LocalssetServiceConnector.cs
          RemoteAssetServiceConnector.cs
        Grid/
          ...
        Interregion/
          ...
        Inventory/
          ...
        Messaging/
          ...
        User/ 
          ...

    Server/

      ServerMain.cs (the server shell, produces OpemSim.Services.exe)

      Base/
        (Base and basic classes for the HTTP server, one DLL)
        HttpServerBase.cs
        ServerUtils.cs
        ServicesServerBase.cs

      Handlers/
         (place to put all IN connectors and network service handlers)
         Asset/
           AssetServerConnector.cs
           AssetServerDeleteHandler.cs
           AssetServerGetHandler.cs
           AssetServerPostHandler.cs
         ...
         Base/
           (Base and basic classes for IN connectors)
           ServerConnector.cs

    Services/
      (service implementations, each on its own dll)

       AssetService/

       Base/

       GridService/

       InventoryService/

       MessagingService/

       UserService/

       (meat of the OUT connectors, one DLL)
       Connectors/
         ...

       (service interfaces, one DLL)
       Interfaces/
         ...

Service Interfaces

OpenSim.Services.Interfaces: This DLL contains all the service interfaces known to OpenSim.

Here is how IAssetService looks like:

public interface IAssetService
    {
        // Three different ways to retrieve an asset
        //
        AssetBase Get(string id);
        AssetMetadata GetMetadata(string id);
        byte[] GetData(string id);
 
        bool Get(string id, Object sender, AssetRetrieved handler);
 
        // Creates a new asset
        // Returns a random ID if none is passed into it
        //
        string Store(AssetBase asset);
 
        // Attachments and bare scripts need this!!
        //
        bool UpdateContent(string id, byte[] data);
 
        // Kill an asset
        //
        bool Delete(string id);
    }

Connectors

The direction of a connector is determined by the direction of the connection being made, not by the direction of the data flow. Therefore, an OUT connector is always a client, while an IN connector is always a server. Conversely, an OUT connector will provide an interface, while an IN connector will consume an interface. Interfaces are consistent across the entire chain of connectors, therefore IN connectors can be directly connected to OUT connectors, which would create a proxy server.

Out Connectors

OpenSim.Region.CoreModules.ServiceConnectors: this is where all concrete OUT connectors used by the simulator hang out.

OpenSim.Services.Connectors: this is where all the base classes of the OUT connectors hang out. These ones can be reused by servers other than the simulator.

An OUT connector primarily exposes its interfaces, it may have a secondary interface to be able to load as a region module. Here is a snippet of the RemoteAssetServiceConnector, which, as the name says, connects to a remote asset server:

public class RemoteAssetServicesConnector :
            AssetServicesConnector, ISharedRegionModule, IAssetService
    {
 
        private bool m_Enabled = false;
        private IImprovedAssetCache m_Cache;
 
        public string Name
        {
            get { return "RemoteAssetServicesConnector"; }
        }
 
        public override void Initialise(IConfigSource source)
        {
            IConfig moduleConfig = source.Configs["Modules"];
            if (moduleConfig != null)
            {
                string name = moduleConfig.GetString("AssetServices", "");
                if (name == Name)
                {
                    IConfig assetConfig = source.Configs["AssetService"];
                    if (assetConfig == null)
                    {
                        m_log.Error("[ASSET CONNECTOR]: AssetService missing from OpenSim.ini");
                        return;
                    }
                    m_Enabled = true;
                    base.Initialise(source);
                    m_log.Info("[ASSET CONNECTOR]: Remote assets enabled");
                }
            }
        }
 
        public void AddRegion(Scene scene)
        {
            if (!m_Enabled)
                return;
            scene.RegisterModuleInterface<IAssetService>(this);
        }
        ...
     }

The above code is just the region module, which is enabled or not depending on configuration variables (described further down). Two things are noteworthy:

• This module registers with Scene as the simulator's IAssetService. In other words, this modules is the direct provider of that interface within the simulator. All asset operations go through it first.
• This class extends AssetServicesConnector, which is the class that has the 'meat' of the service. A small part of it is presented below:

public class AssetServicesConnector : IAssetService
    {
        private string m_ServerURI = String.Empty;
        private IImprovedAssetCache m_Cache = null;
 
        public AssetServicesConnector(string serverURI)
        {
            m_ServerURI = serverURI;
        }
 
        public AssetServicesConnector(IConfigSource source)
        {
            Initialise(source);
        }
 
        public virtual void Initialise(IConfigSource source)
        {
            IConfig assetConfig = source.Configs["AssetService"];
            if (assetConfig == null)
            {
                m_log.Error("[ASSET CONNECTOR]: AssetService missing from OpenSim.ini");
                throw new Exception("Asset connector init error");
            }
 
            string serviceURI = assetConfig.GetString("AssetServerURI",
                    String.Empty);
 
            if (serviceURI == String.Empty)
            {
                m_log.Error("[ASSET CONNECTOR]: No Server URI named in section AssetService");
                throw new Exception("Asset connector init error");
            }
            m_ServerURI = serviceURI;
        }
 
        public AssetBase Get(string id)
        {
            string uri = m_ServerURI + "/assets/" + id;
 
            AssetBase asset = null;
            if (m_Cache != null)
                asset = m_Cache.Get(id);
 
            if (asset == null)
            {
                asset = SynchronousRestObjectRequester.
                        MakeRequest<int, AssetBase>("GET", uri, 0);
 
                if (m_Cache != null)
                    m_Cache.Cache(asset);
            }
            return asset;
        }
        ...
    }

The class above is the one that actually holds the reference to the remote server, as an URL, and that performs the remote calls to it.

Besides this connector, there are currently 2 other ones that can be used: LocalAssetServiceConnector and HGAssetBroker. The former is meant to be used when the service runs within the simulator's process; the latter is meant to be used for Hypergrided simulators.

In Connectors

OpenSim.Server.Handlers: this DLL is where all IN connectors hang out.

IN connectors listen for requests and decode them for processing. They are typically bound to a HTTP server, although other transports are possible. An IN connector will acquire an instance of a class exposing the target interface and send all received and decoded data to that class, which may be a consumer (database, etc.), a forwarder/broker (HGBroker) or an OUT connector, which would create a proxy server. IN connectors can share the same HTTP server, so all services can be contained in a single server process, or split among several processes or hosts. The proxy configuration is of particular interest, since it allows load balancing and write request routing. For instance, in SQL clusters it is possible to route write requests to different hosts than read requests.

Here is how the IN connector for an asset server looks like:

public class AssetServiceConnector : ServiceConnector
    {
        private IAssetService m_AssetService;
 
        public AssetServiceConnector(IConfigSource config, IHttpServer server) :
                base(config, server)
        {
            IConfig serverConfig = config.Configs["AssetService"];
            if (serverConfig == null)
                throw new Exception("No section 'Server' in config file");
 
            string assetService = serverConfig.GetString("LocalServiceModule",
                    String.Empty);
 
            if (assetService == String.Empty)
                throw new Exception("No AssetService in config file");
 
            Object[] args = new Object[] { config };
            m_AssetService =
                    ServerUtils.LoadPlugin<IAssetService>(assetService, args);
 
            server.AddStreamHandler(new AssetServerGetHandler(m_AssetService));
            server.AddStreamHandler(new AssetServerPostHandler(m_AssetService));
            server.AddStreamHandler(new AssetServerDeleteHandler(m_AssetService));
        }
    }

Two noteworthy things about this connector:

• It establishes the service handlers on the server
• It loads a DLL containing the service implementation (finally!).

Service Implementations

OpenSim.Services.AssetService: the default OpenSim implementation of the asset service. Here is a snippet:

public class AssetService : AssetServiceBase, IAssetService
    {
        public AssetService(IConfigSource config) : base(config)
        {
            if (m_AssetLoader != null)
            {
                IConfig assetConfig = config.Configs["AssetService"];
                if (assetConfig == null)
                    throw new Exception("No AssetService configuration");
 
                string loaderArgs = assetConfig.GetString("AssetLoaderArgs",
                        String.Empty);
 
                m_log.InfoFormat("[ASSET]: Loading default asset set from {0}", loaderArgs);
                m_AssetLoader.ForEachDefaultXmlAsset(loaderArgs,
                        delegate(AssetBase a)
                        {
                            Store(a);
                        });
 
                m_log.Info("[ASSET CONNECTOR]: Local asset service enabled");
            }
        }
 
        public AssetBase Get(string id)
        {
            m_log.DebugFormat("[ASSET SERVICE]: Get asset {0}", id);
            UUID assetID;
 
            if (!UUID.TryParse(id, out assetID))
                return null;
 
            return m_Database.FetchAsset(assetID);
        }
        ...
     }

The above class is loaded by the IN connector shown before.

Server Shells

OpenSim.Services.exe

There is now a generic server shell that can load any IN service connectors, along with their specified service implementations. This server shell is configured with a .ini file, using exactly the same configuration variables that pertain to the asset service configuration in OpenSin.ini.

Here is the new server shell:

public class OpenSimServer
    {
        protected static HttpServerBase m_Server = null;
 
        protected static List<IServiceConnector> m_ServiceConnectors =
                new List<IServiceConnector>();
 
        static int Main(string[] args)
        {
            m_Server = new HttpServerBase("Asset", args);
 
            IConfig serverConfig = m_Server.Config.Configs["Startup"];
            if (serverConfig == null)
            {
                System.Console.WriteLine("Startup config section missing in .ini file");
                throw new Exception("Configuration error");
            }
 
            string connList = serverConfig.GetString("ServiceConnectors", String.Empty);
            string[] conns = connList.Split(new char[] {',', ' '});
 
            foreach (string conn in conns)
            {
                if (conn == String.Empty)
                    continue;
 
                string[] parts = conn.Split(new char[] {':'});
                string friendlyName = parts[0];
                if (parts.Length > 1)
                    friendlyName = parts[1];
 
                m_log.InfoFormat("[SERVER]: Loading {0}", friendlyName);
 
                Object[] modargs = new Object[] { m_Server.Config, m_Server.HttpServer };
                IServiceConnector connector =
                        ServerUtils.LoadPlugin<IServiceConnector>(conn,
                        modargs);
 
                if (connector != null)
                {
                    m_ServiceConnectors.Add(connector);
                    m_log.InfoFormat("[SERVER]: {0} loaded successfully", friendlyName);
                }
                else
                {
                    m_log.InfoFormat("[SERVER]: Failed to load {0}", conn);
                }
            }
            return m_Server.Run();
        }
    }

Essentially, this generic server loads all DLLs for the specificed ServiceConnectors configuration variable. Therefore it's trivial to combine several services in a single server shell, or to have several server shells each one running a service.

Configuration

This architecture splits all services and service connectors into individual region modules and DLLs that are then specified in the server's configuration files. Each server -- OpenSim.exe and all OpenSim.Services.exe -- has its own .ini file. There are two ways of configuring the servers, described next.

Direct Configuration

The familiar .ini file has now several new configuration variable for every service connector. See Services and Service Connectors Configuration for the extra variables pertaining to configuring the asset service in OpenSim.ini.

As for configuring a resource server, here is an example of an asset server in the new style of server shells (these variables are stored in OpenSim.Services.ini)

[Startup]
; These are also available as command line options

; console = "local" ; Use "basic" to use this on a pipe
; inifile = "OpenSim.Servers.AssetServer.ini"
; logfile = "AssetServer.log" ; Also read from application config file

; Connectors, comma separated
ServiceConnectors = "OpenSim.Server.Handlers.dll:AssetServiceConnector"

[Network]
port = 8003

[AssetService]
LocalServiceModule = "OpenSim.Services.AssetService.dll:AssetService"
StorageProvider = "OpenSim.Data.MySQL.dll"
ConnectionString = "Data Source=localhost;Database=opensim;User ID=opensim;Password=opensim;"
DefaultAssetLoader = "OpenSim.Framework.AssetLoader.Filesystem.dll"
AssetLoaderArgs = "assets/AssetSets.xml"

Prepackaged System Architectures

While the split in small design elements provides all the flexibility one can get from the framework, the number of configuration variables increases considerably, and can quickly become overwhelming. To address this problem we are working on configuration "includes", which will allow us to provide pre-packaged system architectures, having all the new variables properly set up so that they can produce architectures such as the ones in the picture above.

There will be three prepackaged system architectures: StandloneGrid, StandardGrid and HyperGrid.