[Opensim-dev] Technical assessment of Cable Beach asset server

Mike Mazur mmazur at gmail.com
Thu Jan 15 05:10:13 UTC 2009 

Hello,

I'd like to present my analysis of the Cable Beach[1] design & code,
with the aim to eventually use it as the official OpenSim asset and
inventory server.

In this email I'll give a brief summary describing why this project was
developed, describe how it works and highlight a few concerns.

For more information, please refer to the Asset Server Proposal[2],
Client Documentation[3] and Developer Documentation[4] prepared by John
Hurliman on the OpenSim wiki.


Short Summary of Motivation
===========================

Currently assets can only be accessed by logging into a region on the
particular grid the assets exist on. The main motivation for this new
asset server is to empower the user to access their assets in any way.

With the built-in authentication capabilities in Cable Beach, regions in
other grids, or HyperGrid regions, could access assets and inventory
without the assets or inventory being openly available to anyone on the
Internet. External applications, such as texture or script editors,
capable of pushing to the asset server directly become feasible.

Future viewer clients can access assets directly from the asset server.
Removing asset traffic from the region server results in improved
region server performance.


How It Works
============

The main asset server executable[5]:

1. defines a set of interfaces,
2. reads a config file,
3. loads plugins, specified in the config file, which implement the
   interfaces,
4. starts an HTTP server, and
5. starts loaded plugins.

Interfaces
----------

The interfaces[6] are:

- IStorageProvider
- IInventoryProvider
- IAuthenticationProvider
- IAuthorizationProvider
- IMetricsProvider

IStorageProvider declares methods for accessing and creating
(modifying?) asset data and metadata. How the data is stored depends on
the storage provider. Current storage provider plugins include:

- SimpleStorage[7], stores assets to disk
- OpenSimMySQLStorage[8], compatible with the current core OpenSim
  MySQL asset storage
- AmazonS3Storage[9], stores assets in Amazon's S3 cloud solution

IInventoryProvider declares methods for interacting with inventory.
Current inventory plugins include:

- SimpleInventory[10], stores inventory information to disk
- OpenSimMySQLInventory[11], compatible with the current core OpenSim
  MySQL inventory storage

IAuthenticationProvider declares methods for managing authenticated
users/sessions in the asset server.

IAuthorizationProvider declares methods for determining access to
assets and inventory. Currently only the AuthorizeAll[12] plugin exists,
which allows all access.

IMetricsProvider declares methods to collect access statistics.

Config File
-----------

The asset server uses Nini to read an INI config file[13]. The config
file allows control over basic things like the port the asset server
should listen on or the DB connection string for the OpenSimMySQL*
plugins. Plugins implementing the various interfaces are listed as well.

Loading Plugins
---------------

The asset server uses ExtensionLoader[14] for loading the plugins at
runtime. Currently the plugins to load are specified in the config
file. If any of the interfaces are not implemented, an exception is
thrown and the server does not start.

Starting HTTP Server and Plugins
--------------------------------

The HTTP server in the LibOMV codebase is used[15]. After the HTTP
server has started, each of the plugins is started. 

The frontend plugins will register handlers with the HTTP server
according to the services they provide. For example, the
OpenSimFrontend[16] registers a GET and POST handler at /assets/, which
provide asset retrieval and creation, respectively.

The backend plugins also perform setup actions. The SimpleStorage[7]
plugin, for instance, will load asset metadata from disk into memory, or
create the storage directory if it doesn't already exist. The MySQL
plugins will first check that a valid connection to the DB can be
established.

On start, the plugins are passed a reference to the main asset server
object through which they communicate. For example, when a request for
an asset arrives at the /assets/<UUID>?texture handler of the
HttpServer, the OpenSimFrontend[16] will access the asset through the
StoragerProvider, whichever it may be, as follows:

  Metadata metadata;
  byte[] assetData;
  server.StorageProvider.TryFetchDataMetadata(assetID,
                                              out metadata,
                                              out assetData);


Concerns and Thoughts
=====================

After going through the code and testing the asset server in my
(admittedly non-production) development environment, I like where this
thing is going. I would like to see the concepts included in OpenSim.

Not everything currently in the Cable Beach SVN needs to be brought in.
For starters, the main asset server along with the OpenSim-specific
plugins and dummy authentication/authorization plugins are all that's
required. Additional plugins can be available on Forge. This subset of
the Cable Beach codebase shouldn't duplicate any existing code that
isn't meant to be replaced.

Storage Backends
----------------

Currently only MySQL is supported. Additional storage plugins will have
to be written to support SQLite, MSSQL, NHibernate and Null storage
backends. Another alternative could be writing one plugin that uses
OpenSim's IAssetProviderPlugin interface to support the various
backends.

ExtensionLoader and Mono Addins
-------------------------------

ExtensionLoader, used for loading plugins at runtime, duplicates Mono
Addins functionality and adds another dependency. The path of least
resistance is probably to have the asset server use Mono Addins
instead, though I'm not sure what the workload would be like to achieve
this.

Alternatively, if ExtensionLoader is superior to Mono Addins, it may be
beneficial to have both in the tree for the time being, and slowly
convert the plugin architecture to standardize on ExtensionLoader.
Further investigation is necessary here.

Config Files
------------

The asset server configuration will change from the familiar
AssetServer_Config.xml file identical in format to all the other UGAI
server config files to an INI file. This can increase support requests
by users which are unaware that there is a new asset server.

Support issues can be minimized with good documentation that's widely
publicized and with a long transition period where both asset servers
are available in the tree. Once most users are comfortable with the new
asset server, the old one can be removed.


Next Steps
==========

At this stage, let's discuss. Please ask your questions, raise
concerns, and we'll work through this.

Thank you for reading,
Mike


References
==========

[1] http://forge.opensimulator.org/gf/project/assetserver/
[2] http://opensimulator.org/wiki/AssetServerProposal
[3] http://opensimulator.org/wiki/AssetServerProposal/ClientDocs
[4] http://opensimulator.org/wiki/AssetServerProposal/DeveloperDocs
[5]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FAssetServer.cs&view=markup
[6]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FInterfaces.cs&view=markup
[7]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FSimple%2FSimpleStorage.cs&view=markup
[8]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FOpenSimMySQL%2FOpenSimMySQLStorage.cs&view=markup
[9]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FAmazonS3%2FAmazonS3Storage.cs&view=markup
[10]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FSimple%2FSimpleInventory.cs&view=markup
[11]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FOpenSimMySQL%2FOpenSimMySQLInventory.cs&view=markup
[12]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FAuthorizeAll.cs&view=markup
[13] An example INI file, well documented:
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2Fbin%2FAssetServer.ini&view=markup
[14] http://code.google.com/p/extensionloader/
[15] http://openmetaverse.org/svn/misc/HttpServer/
[16]
http://forge.opensimulator.org/gf/project/assetserver/scmsvn/?action=browse&path=%2Ftrunk%2FAssetServer%2FExtensions%2FOpenSimFrontend.cs&view=markup

More information about the Opensim-dev mailing list