REST

The following is taken from Alan M Webb's OpenSim REST Support Draft document dated 8/18/08. The information here and specification are works in progress but provide the basis for useful OpenSim integrations including content input and retrieval without direct DB access. --MoHax 00:10, 10 October 2008 (UTC)

There are a number of REST interfaces implemented within OpenSim. This document describes the work done to support specific interactions required in support of external application integration. XML is used to encode the payloads associated with these requests. The specific XML used is documented as an appendix, this is an interim specification.

What is REST?

The notion of Representational State Transfer (REST) was introduced by Roy Fielding in his doctoral dissertation in 2000. It is essentially an architectural concept rather than a protocol, and this has led to a variety of implementations, all of which might claim to be compliant REST implementations.

A REST interface is built upon the HTTP protocol, making use of the four principle methods defined therein: GET, PUT, POST, and DELETE. A fifth method HEAD is also supported, but that is really just a semantic variation of GET. The TRACE and OPTIONS methods have a less fundamental role. No other protocol is involved.

Note that REST does not specify the nature of the payload either sent or received. The interface associated with this document uses XML meta-data to manage an arbitrary payload. The need for a schema document to give form to this XML will be dealt with as a separate discussion.

Of particular significance is the understood meaning of the methods themselves, which is (and must be) as defined by the HTTP specification:

• GET retrieves the information identified by the associated URI and returns it to the requester. GET is idempotent. If the URI identifies a process, then it is the output of that process which is returned, and not the source text of that process. GET is conditional when the headers include any of the If headers. GET is partial if a range related header is included. The results of GET may be able to be cached.

• HEAD is functionally equivalent to GET except that the payload is not included in the response. The result of a HEAD request may be able to be cached.

• POST requests that the server accept the payload as a new entity, subordinate to the existing entity identified by the URI. POST places requirements upon session capabilities in the server insofar as it requires persistent connection and flow control capabilities.

• PUT indicates that the payload should be stored on the server under1 the resource named by the URI. If the named resource already exists, then the payload is considered to be a modified version of that resource. If the named resource does not exist, then it should be created by the server.

• DELETE requests that the server delete the resource identified by the URI.

Other methods (TRACE, OPTIONS, CONNECT) may be supported, but are not fundamental to the principles of REST. Use of these methods also implies specific requirements in terms of how the response codes should be used.

REST Inventory Support

The REST support in OpenSim is separated into a number of functional components, all of which live in the OpenSim/ApplicationPlugins/Rest hierarchy.

1. Plug-in management. E.g. RestPlugin.cs and Inventory/RestHandler.cs
2. Service management: Inventory/RestHandler.cs
3. REST implementation: RequestData.cs, Rest.cs
4. Service implementations: Inventory/RestInventoryServices.cs, Inventory/RestAssetServices.cs, etc.

This separation serves a number of purposes:

• it limits the sensitivity of each component to implementation changes in the others
• it simplifies the task of service implementation by “hiding” infrastructure
• it helps make services consistent by sharing a single implementation of REST related tasks

The design intention is that each subdirectory in the REST hierarchy, I.e. Inventory, corresponds to a distinct REST handler assembly consisting of service management and one or more service implementations. The Inventory directory is one example of such an assembly.

Configuration Parameters

In order for the REST handlers to be loaded and effective, statements must be added to the OpenSim.ini file. OpenSim reads the initialization file during start-up. This file is partitioned into various functional domains, e.g.

[Startup]
gridmode = false
physics = basicphysics

[RestPlugins]
enabled = false

[RestHandler]
enabled = false

The RestPlugins partition affects REST handlers in general and supports the following entries:

[RestPlugins]
enabled = true|false || false
god_key = <string> || “”
prefix  = <string>   || “/admin”

Only enabled=true is necessary to enable REST processing.

Additionally, REST inventory processing currently requires that the following partition be defined:

[RestHandler]
enabled = true|false || false
authenticate = true|false || true
extended-escape = true|false || true
realm = <string> || “OpenSim REST”
dump-asset = true|false || false
dump-line-size = int%8 || 32
path-fill = true|false || false

The partition name corresponds to the ConfigName value specified in the plug-in subclass, I.e. RestHandler.cs.

Again, only the enabled=true entry must be provided to enable the inventory handler; all other defaults are acceptable for normal operation. The meanings of these values will become apparent in the rest of the document.

When the server starts, the REST handler will publish several messages indicating which configuration values are in effect.

Effective URI

From a REST perspective, the path component of the URI is viewed as a resource identification, the precise significance of which depends upon the HTTP method being used. There are subtleties here, in some cases a prefix of the path may be viewed as a functional qualification of the authority, with the remainder of the URI being a resource identification within the domain of that authority. In this document, the notion of a service corresponds to the functional qualification mechanism just described.

The prefix value in the configuration file is used to construct the server's base prefix for REST services, as illustrated in the description of the inventory services prefix shown below:

http://<host>:<port>/admin/inventory
                     ^^^^^

In this URI /admin is the base prefix, and /inventory is a prefix extension that identifies the inventory service handler.

All requests that have an /admin/inventory prefix will be passed to the inventory handler. The prefix must be followed by the name of the avatar that owns the inventory in question. For example:

http://localhost:9000/admin/inventory/Alan Webb

Assuming the GET method, then this will return Alan Webb's inventory catalog, as XML data in the response payload, for example:

<Inventory>
<Folder name = “My Inventory” uuid=”11111111-...-111111111111” ....
        <Folder name=”Objects” uuid=....
        ....
        </Folder>
    </Folder>
</Inventory>

In this case /admin/inventory identiifes the service, and /Alan Webb is the resource specification component of the path.

Implementation Overview

The REST implementation being described is manifest in a subset of the files contained in the ApplicationPlugins/Rest file hierarchy. It is logically partitioned into three domains:

• Plugin and configuration management
• REST implementation services
• REST service providers

The objective here being to separate functionality so that the task of the service provider is made as straightforward as possible while ensuring that all services implemented using the interface are consistent in both approach and behavior.

Illustration 1: Overall Data Flow Model

As shown in Illustration 1: Overall Data Flow Model all requests originate, and are eventually returned to, the OpenSim HTTP server. The handler is isolated from this implementation by the IHttpAgentHandler interface and the OSHTTP data-types. After matching, the request is passed to the appropriate handler; this relationship is defined by the IRest interface and the RequestData data-type.

Changes to BaseHttpServer

The existing OpenSim HTTP server class was modified to support plug-in modules that conform to the IHttpAgentHandler interface. Three new subroutines were added: AddHttpAgentHandler, RemoveHttpAgentHandler, and TryHttpAgentHandler; this is consistent with the way in which existing handlers are managed.

Additionally, the HandleRequest method was extended to include support for the IHttpAgentHandler interface.

The processing of existing handlers is not affected by the new support.

Plugin and Configuration Management

IHttpAgentHandler describes the interface between OpenSim's base HTTP server and the REST implementation, a plug-in must implement this interface. Any class that exports this interface, and that registers its handlers using the appropriate (not defined by an interface1) methods in the base HTTP server can act as a handler. Note that while the usage described here is intended to support REST, the interface itself is agnostic and may be used as the basis for any HTTP request handler.

RestPlugin.cs provides a basic agent-handling superclass. An implementation of this class represents a handler from the point of view of the base HTTP server. This class implements the IHttpAgentHandler interface. A given implementation class need only be differentiated from the others in terms of the agent name it uses when it registers. At present, a duplicate agent (based upon supplied agent name) will not be registered; how such a failure is handled depends upon the implementation class. Aside from the name used, a given handler should be differentiated in terms of the set of request patterns that it will match; overlap is undesirable as the first match will win, and at present the order in which OpenSim loads modules is non-deterministic. The RestHandler class in the Inventory sub-directory is an example of an implementation class for RestPlugin.cs.

Inventory/RestHandler.cs implements the plug-in interface described above, and also implements a local model of integrating a set of services within a single handler instance. It does this by scanning the classes included in the assembly for those that implement the IRest interface; each such class is then initialized. This provides implementation independence from a development standpoint (adding a new service is as simple as adding a single class) whilst avoiding unnecessary overt modularization in OpenSim. This class also takes care of global configuration and initialization of the basic REST environment, i.e. It sets basic references for the OpenSim communications manager, user services, inventory services, and asset management.

For performance reasons, and as an obligation to the IHttpAgentHandler definition, this class also implements the Match function, using pattern information registered by the individual services when they were initialized1.

This class contains no REST specific, or service specific functionality beyond some basic initialization. This class can be used in multiple assemblies without change.

Rest.cs provides both state and functionality in support of the REST implementation. Most of the state is static and constant, a small set of static values are initialized by the RestHandler when the plug-in is loaded. This class will only change as the REST implementation evolves and becomes more complete. It must not contain any service sensitive elements. This class may be sub-classed by a given assembly to provide global additions required by a specific set of services.

RequestData.cs serves as a carrier for the state associated with a request. It may be sub-classed by a given service provider in order to add service specific state and functionality, for example the inventory services currently utilize an XML extension to handle inventory specific payload interpretation and generation.

The remaining classes in the Inventory directory provide different classes of service.

Adding A New REST Service To An Existing Handler

The process for adding a new service is straightforward. The file RestTestServices.cs serves as an example1 of what a new services class must minimally include. Key points to note are:

1. Each service class should have an associated value for qPrefix. This should be unique for each service, as it represents the relative extension of the URI that will be matched by this service. The value supplied here may be absolute, but this is discouraged. The constructor will automatically update a relative value to reflect the absolute prefix set globally during initialization.
2. In the case of test services the constructor loads a set of classes from a sub-directory, each of which represents a specific service implementation. For example, these may be method-related, or based upon other values contained in the supplied URI.
3. The services constructor must register itself with the RestHandler. It provides three values at registration time: a reference to the service method, the URI prefix that this service will match, and a reference to an RequestData allocation method that will return an appropriately sub-classed instance of the RequestData class.
4. Implementation of the service and allocation methods described above must be provided.

Other methods definitions are simply part of the overall infrastructure and would be unique to a given service implementation.

Note that a given services class may implement zero or more services, according to its intended function. A class providing shared functionality to other services may be loaded but not register any actual handling methods. Similarly, a single services class could register multiple handling methods.

REST Implementation Classes

This section provides additional documentation of the state and services provided by Rest.cs and RequestData.cs files.

Rest.cs

The eponymous class provides a collection of request independent services and state definitions. Any functions directly dependent upon the HTTP request and response elements are provided in RequestData.cs.

State

The Log declaration provides local access to the log4net facility. Though apparently complex it simply resolves to a static, read-only reference that can be used to pass messages to the logging facility.

The value of DEBUG reflects whether or not debug messages have been enabled in the OpenSim log. If they have, then DEBUG will be enabled, if not, then they will not. Being a static declaration it is anticipated that the JIT compiler will be able to make efficient code generation assumptions. This is only of real significance where some substantial operation is performed solely in support of debugging.

The Plugin reference is set by RestHandler during initialization. It eliminates the need for a services class to have direct awareness of how it was activated. For example, this reference is used to locate the AddPathHandler method needed to register its interface.

The main reference is set by the RestHandler during initialization. Its value is a reference to the simulator's base class. It is provided to the plug-in interface and is stored for that reason. It is not currently used except by RestHandler to locate the communications manager during initialization.

Comms provides a globally accessible reference to the simulator's communications manager. It is initialized by RestHandler during initialization and is currently only used by RestHandler to locate other, commonly needed, simulator services.

InventoryServices, UserServices, and AssetServices are references to the corresponding simulator service interfaces. This list may grow as common service requirements expand.

Prefix is the common URI prefix associated with all handlers within a given assembly. In the case of inventory services it is obtained from the global REST configuration. It is currently /admin. This is a configurable attribute. Typical this should be set to whatever is considered to be the standard prefix for REST services in a given URI, it will be combined with scheme and authority information to construct a valid URL.

Config is a saved reference to the configuration file used by this assembly. The configuration partition is identified by the ConfigName string supplied by the RestHandler implementation. This is currently only used by the RestHandler, but it could be used by a service implementation to access attributes unique to the service.

GodKey is a string obtained from the configuration file which contains the password for the God user. This user has unfettered access to all administrative functions. This is a temporary mechanism and will eventually be replaced by something more secure. This is a configurable attribute.

Authenticate indicates whether or not user authentication is required by this handler. This is a configurable attribute, true by default.

Secure indicates whether or not the HTTP session should be secured. Secure sessions are currently not available to OpenSim. This is a configurable attribute, true by default.

ExtendedEscape controls whether or not the URI is processed for parsing extensions such as “+” in place of spaces. This is a configurable attribute, true by default.

DumpAsset controls whether or not the content of an incoming asset should be dumped to the trace file after decoding has completed. This is only intended to be used when there is some indication of a problem with the decoding process. It may produce a significant amount of trace output depending upon the quantity of asset data being uploaded. This is configurable and is normally false.

Fill controls whether or not the server will automatically generate intermediate folder names when handling a POST request; it is not currently considered appropriate for PUT. This is not presently implemented.

FlushEnabled indicates that the server will completely flush an inbound data stream before responding. This accommodates clients that can not handle the case where a socket error is generated as a result of an early termination of the inbound data stream (due to an authentication failure for example). This mechanism is vulnerable to DOS attacks and can be selectively enabled/disabled.

Realm is a string that may be used to distinguish the authentication realm associated with the handler, in place of the URI prefix. This corresponds directly to the realm token included in the WWW-Authenticate response header. This is a configurable option, and is set to 'OpenSim REST' by default. Clients may use this value to determine if additional authentication is necessary, or to select appropriate cached credentials.

DumpLineSize indicates the line length to use for any data dumps (e.g. DumpAsset) during runtime. The default is 132. This is a configurable option.

statusHead contains the standard HTML header used for reporting request status. Similarly, statusTail is used to provide a standard suffix. These are both used by the standard status reporting mechanisms.

HttpStatusDesc is an array which, when indexed by an appropriate HTTP status code, will return a corresponding status description.

CreationDate (property) provides a single consistent reference for creation data value needed when introducing new items into the inventory.

MsgId (property) provides a consistent message prefix for log messages generated by the handler. Although static, the value is thread specific, so it is actually unique per request.

RequestId (property) provides a unique, monotonically incrementing request counter for processed requests.

Encoding provides a standard encoding reference (UTF8) for all data handled by the handler.

Version and Name are intended to uniquely identify the REST implementation represented by this particular class. Version information should be updated if any significant changes to the REST interface are implemented. This is not currently exploited, but should be. We should have an extended header that indicates the REST version that the client is expecting, and the Match process should be sensitive to this. In this way we can provide concurrent support for multiple REST versions.

The extensive list of constant values are all held to be self-explanatory and should be used wheresoever the value they represent is required in a service implementation. This ensures consistency between the REST implementation and the REST service implementations.

Functions

StringToBase64 provides a standard Base64 encoding mechanism for service implementations.

Base64ToString provides a standard Base64 decoding mechanism for service implementations.

Hex2Int provides a standard mechanism for converting a hexadecimal string of arbitrary length into an integer.

Int2hex8 converts an integer into an 8 character hexadecimal number. This is not currently used, and as such should not be trusted.

NonceGenerator is a simple nonce producer that ensures uniqueness by simply converting a newly generated UUID into a Base64 string. This function is called each time the user is called upon to authenticate, which is not necessarily that often, provided the client caches credentials based upon realm.

Dump provides a general purpose facility for generating an interpreted hexadecimal data dump to the log file. It is currently used by the DumpAsset debug facility. But it may be used by anybody requiring such data dumping services.

RequestException

This class provides an extended exception that can be distinguished from regular, system generated, exceptions. A RequestException does not indicate a programming failure, rather it indicates that the received request could not be validly processed, for example, the URI was invalid. This exception should only be used in those cases where request processing is to be immediately terminated in conjunction with normal response processing. Principally it allows the handler to differentiate between a failure in processing, which should be logged, and a normal response to an invalid request, which should not.

RequestData.cs

This file, and its eponymous class provides consistent support for request specific REST processing. When the RestHandler has successfully matched a handler to a request, the registered allocate method for that service interface is called to allocate an instance of this class either directly, or as a super class of some service specific extension. This description deals only with the base implementation.

With a single instance of a service class per handler, it is necessary to encapsulate request/response specific data in this way. Those functions that operate over that data are included here for convenience and represent part of the previously mentioned REST implementation encapsulation.

Members

When an instance of this class is created, the constructor always does the following:

1. The parameters are not validated. Their validity is considered to be a precondition and as such the natural exception mechanism is an efficient and effective way to deal with invalid argument values.
2. The supplied request and response object references are cached, as is the supplied request prefix.
3. The effective encoding is obtained from the supplied request if possible, or the Rest default (UTF8) is applied.
4. A lower case copy of the request method is stored.
5. InitUrl is called to decompose the supplied URI into its basic components: query, path, host name, and port.
6. InitParameters is called to create an array of strings, each of which represents a separated portion of whatever URI remains after the handler's selection prefix is removed. This list may be empty.

In the interests of laziness, constructor processing is kept to a minimum.

The IsAuthenticated property checks to see if the user is currently authenticated, and if not, then invokes that authentication mechanism. This allows a service request handler so simply test this boolean and reject the request if the result is false. This is an example of a failure that should not normally be logged, as the client is expected to respond to the challenge with an appropriate set of credentials.

The PathNodes property is an array that contains a normalized list of elements in the absolute path provided in the URI. It does not differentiate the elements.

The Parameters property is an array that contains only those URI path elements that are not a part of the service selection prefix (i.e. /admin/inventory). This simplifies service processing by providing a prefix-independent base for request parameter indeces.

Complete allows a service provider to indicate the manner in which a request has completed. Called with no arguments, this corresponds to a status code of 200 indicating a normal completion. In many cases the service provider must necessarily indicate a more specific code. The message associated with this call may be arbitrarily complex, it is used, as-is, as the message body in the response.

Redirect allows a service provider to indicate that the requester should be redirected to an alternative URL. Redirection may be permanent or temporary as indicated by the boolean argument.

Variations on Fail allow a service to abandon request processing. This call constructs an appropriate HTTP failure response, including any specified message body, generates the response, adds request information to an instance of RestException and finally throws the exception. It is important that a service provider allow RestException events to pass through transparently, (i.e. re-throw any that are inadvertently trapped).

Reject encapsulates whatever processing is appropriate for requests that attempted processing that is not implemented.

Respond allows a service provider to generate a response and complete processing with respect to the current HTTP request and response. This call is mandatory for any request that is considered to have been processed successfully.

AddHeader allows a service provider to add a header to the set of response headers that will be included when the response is finally generated. The Rest constants should be used when specifying the header name. The header value is treated as an arbitrary string.

RemoveHeader allows a service provider to remove a header that was previously added to the response header set.

DumpHeaders causes a list of the currently effective headers to be sent to the log. This is a DEBUG option.

RestInventoryServices

This file, and the classes it contains, provide a monolithic implementation of all of the services provided in support of inventory processing. If required, authentication is established before any processing of inventory data. The following abnormal completion codes may be returned:

• 301 indicating that a permanently applicable alternative URI for a specified resource has been provided.
• 400 indicating a bad request. Information about the nature of the request error is included in the status information returned to the client. Currently this is textual, but when a proper schema document for the interface is established, a more quantitative status will be returned.
• 401 indicating that the client must authenticate.
• 409 indicating that an ambiguous or conflicting request was received.

Inventory Resource Identification

An avatar's inventory is a catalog of the assets 'owned' by the avatar; the inventory contains only of meta-data and depends upon the asset data base for whatever actual data represents the perceived content. References to inventory in this document normally refer to the meta-data rather than the content. An inventory consists of two types of element: folders and items. In both cases, a given element is uniquely identified by an assigned UUID. An associated name is also provided for human convenience, but such names are not required to be unique and are insufficient to ensure an exact identification. The URI used in inventory related requests allows either UUID or name to specified for each element in a resource identification path, if any ambiguity arises then the request will be failed.

The avatar's inventory can contain a private catalog, and one or more shared, or public, catalogs. Currently only the private catalog is processed. When support for the shared catalogs is introduced it will do so by adding by prefixing any path as follows:

/admin/inventory/shared/[<libname>]/

This allows all global inventory, or a specific library to be referenced. For all requests, specifying an avatar name implies the private inventory. In all cases, if additional path information is provided, then the root directory name may be inferred (but accepted if specified explicitly). So

/admin/inventory/Alan Webb/My Inventory/Clothing

and

/admin/inventory/Alan Webb/Clothing

are semantically equivalent. This is a convenient and harmless special case. Where no additional path information is provided, as in

/admin/inventory/Alan Webb

then the root directory may still be inferred for GET and POST, but can not be allowed for PUT. PUT requires the URI to identify the element that is being updated, so inference in this case could be extremely damaging. Explicit specification of the root inventory name is therefore required for all PUT requests. Given that this is an interface normally used only by programmed clients, the additional verbosity is not expected to be an issue. If this proves to be contentious, then the only viable approach would be to drop the inference altogether.

Supported Services

GET

GET allows the client to retrieve all, or some sub-tree of, the inventory associated with a specified avatar's inventory. Inventory information is returned in an XML payload. This request is guaranteed not to affect the state of the avatar's inventory. Possible normal completions are:

• 200 indicating normal completion of the request. The payload will contain an XML representation of the avatar's inventory.

PUT

PUT allows a resource, explicitly nominated in the URI, to be created or replaced. I.e. the URI identifies an instance rather than a context. Possible normal completions are

• 200 if the nominated resource was successfully updated
• 201 if the nominated resource was successfully created.
• 204 if no content change (but the resource was recognized and valid)

POST

POST allows a new resource to be created in the context identified by the URI. It follows from this that the resource identified by a POST request should always be a folder. Normal completions are:

• 200 if some portion of the context was modified.
• 201 if the new data was entirely new.

DELETE

DELETE allows a named resource to be deleted. If the resource is a folder, then the entire sub-tree will be removed. Normal completion codes are:

• 200 if the specified resource was deleted.

Both PUT and POST allow any assets upon which the inventory catalog depends, to be included in-line with the XML catalog data.