AssetServer/ClientDocs

From OpenSimulator

(Difference between revisions)
Jump to: navigation, search
(New page: <pre>{ "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97", "name":"Default Avatar", "description":"", "creation_date":"date::2008-11-26T00:49:48.83Z", "type":"image/jp2", ...)
 
m (Robot: Replacing 'OpenSim' to 'OpenSimulator', which is the precise name)
 
(24 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
__NOTOC__
 +
{{Quicklinks}}
 +
{{proposal}}
 +
{{content}}
 +
 +
= Client Documentation =
 +
 +
Through the variety of extensions that ship with the asset server, several wire protocol formats are supported. Connectors for existing OpenSimulator grid protocols have been added wherever possible, along with new frontends that support web browsers and extensible metadata through JSON.
 +
 +
== Reference Asset Protocol ==
 +
 +
In the previous OpenSimulator asset model, asset requests and asset data proxied through simulators to a backend asset service. This required sending asset data over UDP and adding unnecessary bandwidth overhead to simulators.
 +
 +
[[File:Client-sim-assetserver-original.png]]
 +
Previous OpenSimulator asset flow diagram
 +
 +
[[File:Client-sim-assetserver-distributed.png]]
 +
Distributed Asset Service flow diagram
 +
 +
=== Wire Format ===
 +
 +
To support the new metadata features and extensible content methods in the asset server, a new wire protocol is being designed. This is referred to in the documentation as the Reference Asset Protocol, and is currently using JSON as the transport format. JSON provides a lightweight markup and is loosely typed, allowing extended metadata to be transferred without breaking compatibility with existing or lightweight clients. Although the markup is pure JSON, some additional "types" are implemented by prefixing strings with type identifiers. This allows data to easily be mapped between [[OpenStructuredData]] and JSON. The types and their prefixes are as follows:
 +
 +
{|
 +
|+ '''New JSON Types'''
 +
|-
 +
|'''uuid::''' || UUID, 128-bit unique identifier
 +
|-
 +
|'''uri::''' || Uniform resource identifier
 +
|-
 +
|'''date::''' || ISO-8601 numeric encoding in UTC
 +
|-
 +
|'''b64::''' || Base64-encoded binary data
 +
|}
 +
 +
=== Fetching Asset Metadata ===
 +
 +
Asset data may be located on the asset server, or scattered all over the world at various URLs. To find the location of the data for each asset, you must first fetch the asset metadata which can be retrieved at a fixed location. The URL for metadata is http://[someserver.com]/[assetid]/metadata. An example request:
 +
 +
<pre>GET /c228d1cf-4b5d-4ba8-84f4-899a0796aa97/metadata HTTP/1.1
 +
Host: assets.virtualworld.com
 +
Date: Mon, 22 Sep&nbsp; 2008 12:00:00 GMT
 +
Authorization: OpenGrid ce9a6995-a737-4206-bae8-910cae310e94</pre>
 +
 +
The Authorization header is used to pass an authentication token (a login cookie) to the server. This is covered in detail in the Authentication section.
 +
 +
An example response:
 +
 
<pre>{
 
<pre>{
 
   "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97",
 
   "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97",
Line 8: Line 56:
 
   "temporary":false,
 
   "temporary":false,
 
   "methods":{
 
   "methods":{
       "data":"uri::http://localhost:9001/c228d1cf-4b5d-4ba8-84f4-899a0796aa97/data"
+
       "data":"uri::http://assets.virtualworld.com/c228d1cf-4b5d-4ba8-84f4-899a0796aa97/data"
 
   },
 
   },
 
   "extra_data":{
 
   "extra_data":{
Line 21: Line 69:
 
   }
 
   }
 
}</pre>
 
}</pre>
 +
 +
In this case the metadata is for a texture (type of "image/jp2", JPEG2000 compressed image). The methods map contains a single key/value pair for "data", the method used to fetch the actual asset data. The extra_data structure is also used in this example, to carry the number of components in the JPEG2000 data and the byte offsets of each quality layer in the texture.
 +
 +
=== Fetching Asset Data ===
 +
 +
Once the location of the asset data has been retrieved with a metadata request, a simple HTTP GET will download the content. The Authorization header is still required to fetch content that is not publicly accessible. In a future release, the asset server will support partial downloads with the standard HTTP Range header.
 +
 +
=== Creating an Asset ===
 +
 +
To create an asset, a simple JSON structure containing the asset metadata and data is POSTed to /createasset. Creating an asset and replacing an existing asset are done through the same interface. The "id" field in the POST data is optional. Specifying a UUID for the new asset may impose additional security restrictions, as this is generally reserved for system users who have been granted permission to replace existing assets. An example POST of a text file called note (specifying a UUID in the request) is shown here:
 +
 +
<pre>POST /createasset HTTP/1.1
 +
Host: assets.virtualworld.com
 +
Date: Mon, 22 Sep&nbsp; 2008 12:00:00 GMT
 +
Authorization: OpenGrid http://users.virtualworld.com/users/jhurliman ce9a6995-a737-4206-bae8-910cae310e94
 +
Content-Type: application/json
 +
Content-Length: 225
 +
 +
{
 +
  "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97",
 +
  "name":"note",
 +
  "description":"A small note containing the word testing",
 +
  "type":"text/plain",
 +
  "temporary":false,
 +
  "data":"b64::dGVzdGluZw=="
 +
}</pre>
 +
 +
An HTTP status code 201 (Created) indicates success; all other status code indicate a failure to create the asset. On success, the return data will be a JSON map containing the key "id" with a value of the new asset's ID. If a failure occurred, the HTTP status description may contain more information. An example of a successful response:
 +
 +
<pre>{ "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97" }</pre>
 +
 +
==== Content-Types ====
 +
 +
The asset server treats all stored content as binary blobs with content types. Some content types can have special handlers that expose additional methods or metadata, such as image/jp2 returning extended metadata for the number of image components and byte offsets of the quality layers. However, no special handling is needed for storing and serving up data. To store ShockWave Flash files, use the content-type application/x-shockwave-flash and upload a .swf file.
 +
 +
To maintain compatibility with existing virtual worlds, MIME types have been defined for the existing OpenSimulator asset types. It is important that the correct content-type is used for uploads. The following table shows the mapping between OpenSimulator assets, commonly used file extensions, and content-types. Note that some of the formats used already have recognized MIME types. Anything starting with application/x-metaverse is ad hoc.
 +
 +
{|
 +
! OpenSimulator Type
 +
! File Extensions
 +
! Content-Type
 +
! Notes
 +
|-
 +
| 0 || jp2,j2c,texture || image/jp2 || JPEG2000 texture. Not actually JP2 files, only the codestream. LRCP ordering
 +
|-
 +
| 1 || ogg,sound || application/ogg || OGG Vorbis audio
 +
|-
 +
| 2 || callingcard || application/x-metaverse-callingcard || OpenSimulator calling card. Custom/undocumented text format
 +
|-
 +
| 3 || landmark || application/x-metaverse-landmark || OpenSimulator landmark. Custom/undocumented text format
 +
|-
 +
| 5 || clothing || application/x-metaverse-clothing || OpenSimulator clothing. Custom/undocumented text format
 +
|-
 +
| 6 || primitive || application/x-metaverse-primitive || OpenSimulator primitive. Custom/undocumented XML format
 +
|-
 +
| 7 || notecard || application/x-metaverse-notecard || OpenSimulator notecard. Custom/undocumented text format
 +
|-
 +
| 10 || lsl || application/x-metaverse-lsl || OpenSimulator LSL script
 +
|-
 +
| 11 || lso || application/x-metaverse-lso || OpenSimulator compiled script binary. Currently unused
 +
|-
 +
| 12 || tga || image/tga || Targa image. Currently unused
 +
|-
 +
| 13 || bodypart || application/x-metaverse-bodypart || OpenSimulator bodypart. Custom/undocumented text format
 +
|-
 +
| 17 || wav || audio/x-wav || WAV audio file
 +
|-
 +
| 19 || jpg,jpeg || image/jpeg || JPEG texture
 +
|-
 +
| 20 || animation || application/x-metaverse-animation || OpenSimulator animation. Custom/undocumented binary format
 +
|-
 +
| 21 || gesture || application/x-metaverse-gesture || OpenSimulator gesture. Custom/undocumented format
 +
|-
 +
| 22 || simstate || application/x-metaverse-simstate || OpenSimulator simulator state. Currently unused
 +
|-
 +
| -1 || (none,unknown) || application/octet-stream || Arbitrary or unrecognized binary data
 +
|}
 +
 +
== Reference Inventory Protocol ==
 +
 +
The inventory reference protocol is a work in progress. Leave notes on the discussion page for this document or start a thread on the [https://lists.berlios.de/mailman/listinfo/opensim-dev opensim-dev] mailing list if you'd like to be a part of the process.
 +
 +
== OpenSimulator Asset Protocol ==
 +
 +
The OpenSim.Grid.AssetServer.exe protocol uses a REST interface for retrieving and storing assets. Assets are fetched with a GET to /assets/[assetid] or /assets/[assetid]?texture. The additional ?texture parameter is optional and does not change the response in any way. Asset data and metadata are returned in an XML format, with the asset data base64 encoded as an XML element. To upload, the same XML data structure that is received from a GET is POSTed to /assets/. The XML data is serialized from a class in OpenSimulator code, therefore the structure is subject to change. The current XML serialization is undocumented outside of source code.
 +
 +
== OpenSimulator Inventory Protocol ==
 +
 +
OpenSim.Grid.InventoryServer.exe supports both a REST-like interface and an XML-RPC interface. Although the former interface shares some similar properties with a REST interface, it defines an additional messaging layer using HTTP POSTs of XML data for all requests. The REST-like interface is actually somewhere between SOAP and XML-RPC.
 +
 +
An example request, passing a single UUID argument in with session information:
 +
 +
<pre>&lt;?xml version="1.0"?&gt;
 +
&lt;RestSessionObjectOfGuid xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"&gt;
 +
  &lt;SessionID&gt;070b4811-0a57-4fcd-bb3a-a03fbe9106d2&lt;/SessionID&gt;
 +
  &lt;AvatarID&gt;634ada51-3151-4e6d-a989-309f89f740e0&lt;/AvatarID&gt;
 +
  &lt;Body&gt;3207551c-da01-49ec-bb6e-f24f9ab36dcb&lt;/Body&gt;
 +
&lt;/RestSessionObjectOfGuid&gt;</pre>
 +
 +
Another example, passing a single UUID argument with no session information:
 +
 +
<pre>&lt;?xml version="1.0"?&gt;
 +
&lt;guid&gt;3207551c-da01-49ec-bb6e-f24f9ab36dcb&lt;/guid&gt;</pre>
 +
 +
== Authentication ==
 +
 +
If an asset is protected, an identity must be proven by a client (and that identity authorized to access the asset) before access is granted. Internally the asset server stores identities as URLs. A number of different methods of proving client identity can be used depending on the setup of the asset server. The reference asset server has OpenID login support at http://[yourserver.com]/authenticate. This is a simple OpenID login process that can happen over the reference JSON protocol, or through web forms. If the OpenID provider being used is setup with a well formed login method (such as the current OpenSimulator user server), the entire process can happen without user intervention. Once the OpenID authentication has completed, a UUID token will be returned to the client. Optionally, a cookie will be returned as well containing the same token. This token is a temporary proof of identity that can be used in the Authorization header when requesting assets.
 +
 +
Instead of directly logging into the asset server, a common grid setup may be that the login server contacts an asset server that it knows about to inform it of a successful login and create a temporary session token. This token could be passed back to the client to avoid having to login to both the login server and asset server. Another setup may involve the client using an SSL client certificate signed by an authority that the asset server trusts. The different methods of authenticating allow flexibility between creating a loose, decentralized set of grids and services and a tightly coupled closed or semi-closed grid.
 +
 +
= See Also =
 +
 +
* [[:Category:Getting Started|Getting Started]]
 +
* [[OpenSim:Introduction_and_Definitions|What are UGAIM Services?]]
 +
* [[UserServer|User Server]]
 +
* [[UserServer/ClientDocs|UserServer Client Documentation]]
 +
* [[UserServer/DeveloperDocs|UserServer Developer Documentation]]
 +
* [[GridServer|Grid Server]]
 +
* [[GridServer/ClientDocs|GridServer Client Documentation]]
 +
* [[GridServer/DeveloperDocs|GridServer Developer Documentation]]
 +
* [[AssetServer|Asset Server]]
 +
* [[AssetServer/ClientDocs|AssetServer Client Documentation]]
 +
* [[AssetServer/DeveloperDocs|AssetServer Developer Documentation]]
 +
* [[InventoryServer|Inventory Server]]
 +
* [[InventoryServer/ClientDocs|InventoryServer Client Documentation]]
 +
* [[InventoryServer/DeveloperDocs|InventoryServer Developer Documentation]]
 +
* [[MessagingServer|Messaging Server]]
 +
* [[MessagingServer/ClientDocs|MessagingServer Client Documentation]]
 +
* [[MessagingServer/DeveloperDocs|MessagingServer Developer Documentation]]
 +
* [[:Category:Tech Reference|Technical Reference Pages]]
 +
 +
[[Category:Support]]
 +
[[Category:Tech Reference]]
 +
[[Category:Help]]
 +
[[Category:Configuration]]
 +
[[Category:Getting Started]]
 +
[[Category:Development]]
 +
[[Category:Todo]]
 +
[[Category:Proposal]]

Latest revision as of 22:13, 3 March 2012


[edit] Client Documentation

Through the variety of extensions that ship with the asset server, several wire protocol formats are supported. Connectors for existing OpenSimulator grid protocols have been added wherever possible, along with new frontends that support web browsers and extensible metadata through JSON.

[edit] Reference Asset Protocol

In the previous OpenSimulator asset model, asset requests and asset data proxied through simulators to a backend asset service. This required sending asset data over UDP and adding unnecessary bandwidth overhead to simulators.

Client-sim-assetserver-original.png Previous OpenSimulator asset flow diagram

Client-sim-assetserver-distributed.png Distributed Asset Service flow diagram

[edit] Wire Format

To support the new metadata features and extensible content methods in the asset server, a new wire protocol is being designed. This is referred to in the documentation as the Reference Asset Protocol, and is currently using JSON as the transport format. JSON provides a lightweight markup and is loosely typed, allowing extended metadata to be transferred without breaking compatibility with existing or lightweight clients. Although the markup is pure JSON, some additional "types" are implemented by prefixing strings with type identifiers. This allows data to easily be mapped between OpenStructuredData and JSON. The types and their prefixes are as follows:

New JSON Types
uuid:: UUID, 128-bit unique identifier
uri:: Uniform resource identifier
date:: ISO-8601 numeric encoding in UTC
b64:: Base64-encoded binary data

[edit] Fetching Asset Metadata

Asset data may be located on the asset server, or scattered all over the world at various URLs. To find the location of the data for each asset, you must first fetch the asset metadata which can be retrieved at a fixed location. The URL for metadata is http://[someserver.com]/[assetid]/metadata. An example request:

GET /c228d1cf-4b5d-4ba8-84f4-899a0796aa97/metadata HTTP/1.1
Host: assets.virtualworld.com
Date: Mon, 22 Sep  2008 12:00:00 GMT
Authorization: OpenGrid ce9a6995-a737-4206-bae8-910cae310e94

The Authorization header is used to pass an authentication token (a login cookie) to the server. This is covered in detail in the Authentication section.

An example response:

{
   "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97",
   "name":"Default Avatar",
   "description":"",
   "creation_date":"date::2008-11-26T00:49:48.83Z",
   "type":"image/jp2",
   "sha1":"b64::3ftHyRoIfVYSeumfxRcSk4LZVVU=",
   "temporary":false,
   "methods":{
      "data":"uri::http://assets.virtualworld.com/c228d1cf-4b5d-4ba8-84f4-899a0796aa97/data"
   },
   "extra_data":{
      "components":3,
      "layer_ends":[
         580,
         1987,
         7983,
         31969,
         36041
      ]
   }
}

In this case the metadata is for a texture (type of "image/jp2", JPEG2000 compressed image). The methods map contains a single key/value pair for "data", the method used to fetch the actual asset data. The extra_data structure is also used in this example, to carry the number of components in the JPEG2000 data and the byte offsets of each quality layer in the texture.

[edit] Fetching Asset Data

Once the location of the asset data has been retrieved with a metadata request, a simple HTTP GET will download the content. The Authorization header is still required to fetch content that is not publicly accessible. In a future release, the asset server will support partial downloads with the standard HTTP Range header.

[edit] Creating an Asset

To create an asset, a simple JSON structure containing the asset metadata and data is POSTed to /createasset. Creating an asset and replacing an existing asset are done through the same interface. The "id" field in the POST data is optional. Specifying a UUID for the new asset may impose additional security restrictions, as this is generally reserved for system users who have been granted permission to replace existing assets. An example POST of a text file called note (specifying a UUID in the request) is shown here:

POST /createasset HTTP/1.1
Host: assets.virtualworld.com
Date: Mon, 22 Sep  2008 12:00:00 GMT
Authorization: OpenGrid http://users.virtualworld.com/users/jhurliman ce9a6995-a737-4206-bae8-910cae310e94
Content-Type: application/json
Content-Length: 225

{
   "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97",
   "name":"note",
   "description":"A small note containing the word testing",
   "type":"text/plain",
   "temporary":false,
   "data":"b64::dGVzdGluZw=="
}

An HTTP status code 201 (Created) indicates success; all other status code indicate a failure to create the asset. On success, the return data will be a JSON map containing the key "id" with a value of the new asset's ID. If a failure occurred, the HTTP status description may contain more information. An example of a successful response:

{ "id":"uuid::c228d1cf-4b5d-4ba8-84f4-899a0796aa97" }

[edit] Content-Types

The asset server treats all stored content as binary blobs with content types. Some content types can have special handlers that expose additional methods or metadata, such as image/jp2 returning extended metadata for the number of image components and byte offsets of the quality layers. However, no special handling is needed for storing and serving up data. To store ShockWave Flash files, use the content-type application/x-shockwave-flash and upload a .swf file.

To maintain compatibility with existing virtual worlds, MIME types have been defined for the existing OpenSimulator asset types. It is important that the correct content-type is used for uploads. The following table shows the mapping between OpenSimulator assets, commonly used file extensions, and content-types. Note that some of the formats used already have recognized MIME types. Anything starting with application/x-metaverse is ad hoc.

OpenSimulator Type File Extensions Content-Type Notes
0 jp2,j2c,texture image/jp2 JPEG2000 texture. Not actually JP2 files, only the codestream. LRCP ordering
1 ogg,sound application/ogg OGG Vorbis audio
2 callingcard application/x-metaverse-callingcard OpenSimulator calling card. Custom/undocumented text format
3 landmark application/x-metaverse-landmark OpenSimulator landmark. Custom/undocumented text format
5 clothing application/x-metaverse-clothing OpenSimulator clothing. Custom/undocumented text format
6 primitive application/x-metaverse-primitive OpenSimulator primitive. Custom/undocumented XML format
7 notecard application/x-metaverse-notecard OpenSimulator notecard. Custom/undocumented text format
10 lsl application/x-metaverse-lsl OpenSimulator LSL script
11 lso application/x-metaverse-lso OpenSimulator compiled script binary. Currently unused
12 tga image/tga Targa image. Currently unused
13 bodypart application/x-metaverse-bodypart OpenSimulator bodypart. Custom/undocumented text format
17 wav audio/x-wav WAV audio file
19 jpg,jpeg image/jpeg JPEG texture
20 animation application/x-metaverse-animation OpenSimulator animation. Custom/undocumented binary format
21 gesture application/x-metaverse-gesture OpenSimulator gesture. Custom/undocumented format
22 simstate application/x-metaverse-simstate OpenSimulator simulator state. Currently unused
-1 (none,unknown) application/octet-stream Arbitrary or unrecognized binary data

[edit] Reference Inventory Protocol

The inventory reference protocol is a work in progress. Leave notes on the discussion page for this document or start a thread on the opensim-dev mailing list if you'd like to be a part of the process.

[edit] OpenSimulator Asset Protocol

The OpenSim.Grid.AssetServer.exe protocol uses a REST interface for retrieving and storing assets. Assets are fetched with a GET to /assets/[assetid] or /assets/[assetid]?texture. The additional ?texture parameter is optional and does not change the response in any way. Asset data and metadata are returned in an XML format, with the asset data base64 encoded as an XML element. To upload, the same XML data structure that is received from a GET is POSTed to /assets/. The XML data is serialized from a class in OpenSimulator code, therefore the structure is subject to change. The current XML serialization is undocumented outside of source code.

[edit] OpenSimulator Inventory Protocol

OpenSim.Grid.InventoryServer.exe supports both a REST-like interface and an XML-RPC interface. Although the former interface shares some similar properties with a REST interface, it defines an additional messaging layer using HTTP POSTs of XML data for all requests. The REST-like interface is actually somewhere between SOAP and XML-RPC.

An example request, passing a single UUID argument in with session information:

<?xml version="1.0"?>
<RestSessionObjectOfGuid xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <SessionID>070b4811-0a57-4fcd-bb3a-a03fbe9106d2</SessionID>
  <AvatarID>634ada51-3151-4e6d-a989-309f89f740e0</AvatarID>
  <Body>3207551c-da01-49ec-bb6e-f24f9ab36dcb</Body>
</RestSessionObjectOfGuid>

Another example, passing a single UUID argument with no session information:

<?xml version="1.0"?>
<guid>3207551c-da01-49ec-bb6e-f24f9ab36dcb</guid>

[edit] Authentication

If an asset is protected, an identity must be proven by a client (and that identity authorized to access the asset) before access is granted. Internally the asset server stores identities as URLs. A number of different methods of proving client identity can be used depending on the setup of the asset server. The reference asset server has OpenID login support at http://[yourserver.com]/authenticate. This is a simple OpenID login process that can happen over the reference JSON protocol, or through web forms. If the OpenID provider being used is setup with a well formed login method (such as the current OpenSimulator user server), the entire process can happen without user intervention. Once the OpenID authentication has completed, a UUID token will be returned to the client. Optionally, a cookie will be returned as well containing the same token. This token is a temporary proof of identity that can be used in the Authorization header when requesting assets.

Instead of directly logging into the asset server, a common grid setup may be that the login server contacts an asset server that it knows about to inform it of a successful login and create a temporary session token. This token could be passed back to the client to avoid having to login to both the login server and asset server. Another setup may involve the client using an SSL client certificate signed by an authority that the asset server trusts. The different methods of authenticating allow flexibility between creating a loose, decentralized set of grids and services and a tightly coupled closed or semi-closed grid.

[edit] See Also

Personal tools
General
About This Wiki