[Opensim-dev] Packet Pooling - Should it work?
Diva Canto
diva at metaverseink.com
Sat Aug 17 14:03:20 UTC 2019
The core implementation of the GetTexture/GetMesh Handlers doesn't use a
cache -- because the default configuration assumes those caps are being
served by the simulators, and the simulators already have a cache which
is accessed transparently by GetTexture/GetMesh. But it's really easy to
code up another version that includes a cache. I did it for that grid I
was running. Super trivial.
If you are going to have both the simulator and its GetTexture service
on the same machine, make sure to share the cache.
In my architecture, I had groups of regions (estates) on the same
machine sharing a GetTexture instance also on the same machine, and all
of them sharing the file system cache. So, one cache per machine.
On 8/17/2019 6:38 AM, Mike Dickson wrote:
> One hopefully quick question…
>
> Ideally I’d like to leverage the Flotsam Cache (the architecture I have in mind is one service feeding texture and mesh per region server instance). There is a shared Flotsam Cache on the server already and I’d like to make the GetMesh Robust instance use it. Since Flotsam is a region module I’m guessing that means custom code. But I thought I’d ask quickly for a pointer if there is a simple easy way I’m missing.
>
> Mike
>
> Sent from Mail for Windows 10
>
> From: Diva Canto
> Sent: Friday, August 16, 2019 6:31 PM
> To: opensim-dev at opensimulator.org
> Subject: Re: [Opensim-dev] Packet Pooling - Should it work?
>
> Both simulators and viewers need to access the grid's assets, so they
> all need to know where to get them from. For the simulators, it's
> simple: the addresses of all backend services are given as configuration
> variables in things like this, in GridCommon, or wherever:
>
> [AssetService]
> ;
> ; Change this to your grid-wide asset server. Do not add a slash
> to the end of any of these addresses.
> ;
> AssetServerURI = "${Const|BaseURL}:${Const|PrivatePort}"
>
>
> But what about the viewers? Where do viewers get the information about
> where to download textures/meshes/etc from?
>
> This information is not hardcoded grid-wide anywhere. It's not even
> hardcoded upon login. Viewers get all this information from the
> simulators they connect to, as capability URLs. This is interesting
> because it allows for dynamic associations -- your viewer first connects
> to simulator1, it gets assets from wherever simulator1 tells it to; but
> if later the viewer connects to simulator2, it gets the assets from
> wherever that simulator tells it to; it may be different. The Linden
> viewer is engineered to receive these capability URLs and they are,
> indeed, quite flexible. (this is what allows the hypergrid to work with
> the Linden viewer in the first place)
>
> So where do we define the capability URLs that the simulators send to
> the viewers? They are all in OpenSim.ini, under the section
> [ClientStack.LindenCaps]. Here is that section as it currently stands in
> OpenSim.ini.example:
>
> [ClientStack.LindenCaps]
> ;; For the long list of capabilities, see OpenSimDefaults.ini
> ;; Here are the few ones you may want to change. Possible values
> ;; are:
> ;; "" -- empty, capability disabled
> ;; "localhost" -- capability enabled and served by the simulator
> ;; "<url>" -- capability enabled and served by some other server
> ;;
> ; These are enabled by default to localhost. Change if you see fit.
> Cap_GetTexture = "localhost"
> Cap_GetMesh = "localhost"
> Cap_AvatarPickerSearch = "localhost"
> Cap_GetDisplayNames = "localhost"
>
> GetTexture and GetMesh are the two potentially most impactful services,
> because they serve relatively large data. But, as the comment says,
> there's many others, and they can all be redirected elsewhere. These
> variables are all defaulted to "localhost" meaning that the service is
> done by the simulator. If you want it to be some other server, you just
> need to set it differently. For example:
>
> Cap_GetTexture = "http://mygrid.com:8206/CAPS/textures"
>
> The viewer will get this new URL to fetch textures from, away from the
> simulator.
>
> Now, having set that, you need to make sure that the service actually
> exists at that other URL. This can be done by placing a Robust service
> at that end point serving this capability URL. Here's the configuration
> of a simple Robust server that serves only the textures and nothing
> else, and that is simply a texture-only interface on top of the central
> asset service:
>
> [Network]
> port = 8206 ; make sure to open it in the firewall
>
> [ServiceList]
> ;; GetTexture
> GetTextureConnector =
> "OpenSim.Capabilities.Handlers.dll:GetTextureServerConnector"
>
> [CapsService]
> AssetService = "OpenSim.Services.AssetService.dll:AssetService"
>
> [DatabaseService]
> StorageProvider = "OpenSim.Data.MySQL.dll"
> ConnectionString = "..."
>
> [AssetService]
> LocalServiceModule = "OpenSim.Services.AssetService.dll:AssetService"
> AssetLoaderEnabled = false
>
> There's a lot more to it, but this will be enough to configure something
> that pulls texture-servicing away from the simulator. (Similar for Mesh)
>
> I'm explaining the basic mechanism. But you need to think about the data
> replication architecture that best fits your needs. There are many
> options, some more complicated than others. In fact, the default
> configuration (sim serves textures) is one point in the data replication
> design space, where textures are replicated in every simulator that ever
> served them. It increases the load in the simulators, but decreases the
> load of serving textures in the central server. But, as I said, there
> are many more options. This is not something that has one single / best
> configuration; it depends on many things.
>
> Nothing of this is documented, as usual. There is enough information in
> this email that a capable person can trace it in the code -- search for
> some of the configuration sections I mention above and trace it from there.
>
> I won't give customized solutions, and I don't have time to write
> documentation, but I'll be happy to answer more concrete questions that
> benefit everyone.
>
> Good luck!
>
> On 8/16/2019 11:14 AM, Mike Dickson wrote:
>> Thanks Diva, that would be much appreciated and I think helpful to a great many people.
>>
>> Sent from Mail for Windows 10
>>
>> From: Diva Canto
>> Sent: Friday, August 16, 2019 9:53 AM
>> To: opensim-dev at opensimulator.org
>> Subject: Re: [Opensim-dev] Packet Pooling - Should it work?
>>
>> [I meant to send this to the list]
>> I'll send some info on where to look later today.
>>
>> -------- Forwarded Message --------
>> Subject: Re: [Opensim-dev] Packet Pooling - Should it work?
>> Date: Thu, 15 Aug 2019 17:27:38 -0700
>> From: Diva Canto <diva at metaverseink.com>
>> To: Mike Dickson <mike.dickson at utopiaskye.com>
>>
>> Hi,
>>
>> You can already remove almost everything out of the simulator with the
>> right configurations.
>>
>> I don't recommend serving the large assets, like textures and mesh, from
>> one single server because that one quickly becomes the bottleneck. But
>> there are may ways of serving replicating the data.
>>
>> I ran a large grid that had a completely different arrangement than the
>> default: each group of regions that were related (i.e. an "estate") had
>> its own set of asset services. Users' assets in inventory were served by
>> yet a separate server. All this has been possible for a very long time.
>>
>> Diva
>>
>> On 8/15/2019 1:07 PM, Mike Dickson wrote:
>>> I'm actually not so much trying to optimize network traffic (that is
>>> another great topic and I personally think what core should focus on is
>>> protocols rather than code). I'm concerned there is an issue with garbage
>>> collection limiting scaling of the simulator. Basically I'm of the opinion
>>> that really the only thing that should be running in the simulator is...
>>> the simulation. So as much as possible pulling things out (Moving to AISv3
>>> for inventory out of process, SSB out of process, etc would all be good
>>> additional steps). It's true if you're just running a standalone thats
>>> probably overkill but I'm trying to support large regions and scale. Hence
>>> yes the whole stack is really the focus.
>>>
>>> Mike
>>>
>>> On Wed, Aug 14, 2019 at 10:54 PM Mister Blue <misterblue at misterblue.com>
>>> wrote:
>>>
>>>> There have been many attempts at optimizing the network traffic from the
>>>> simulator to the clients. GP optimization confuses low level networking
>>>> (queuing, ..) with application level (object updates before wind updates).
>>>>
>>>> Make sure you're thinking of the whole virtual world stack.
>>>>
>>>> On Wed, Aug 14, 2019 at 4:07 PM Mike Dickson <mike.dickson at utopiaskye.com>
>>>> wrote:
>>>>
>>>>> So after doing some research I think my fix to this is to get as many of
>>>>> the big allocations out of the region server as possible and secondarily
>>>> to
>>>>> get the rest coming from a pool where I can. I think that translates to
>>>> the
>>>>> follow projects:
>>>>>
>>>>> 1) Move GetMesh and GetTexture out of process and into a separate server
>>>>> 2) Get the rest of the UDP allocations coming from a pool.
>>>>>
>>>>> For #1 there was code originally done for InWorldz/Halcyon to do that. I
>>>>> can try and ressurect it and interface it to the asset service (ideally
>>>>> through the local asset cache) but I think alternatively a redo makes
>>>> more
>>>>> sense.
>>>>> For #2 there is buffer pooling code in LibOMV originally in Halcyon that
>>>> I
>>>>> believe Cinder got upstream via Latif a while back.
>>>>>
>>>>> And yes I did run with -desktop mode in Mono for some time. It sort of
>>>>> bandaids things a bit but when a GC pass does happen UDP stalls until the
>>>>> GC completes and the protocol recovers (for reliable messages). If you
>>>>> extend that to a busy region with 20, 30 or more avatars it falls apart
>>>>> quickly. Especially with everyone wearing mesh. Still probably better
>>>> than
>>>>> the standard GC but the real fix is to stop making garbage to collect.
>>>>>
>>>>> Mike
>>>>>
>>>>> Sent from Mail for Windows 10
>>>>>
>>>>> From: Leal Duarte
>>>>> Sent: Tuesday, August 13, 2019 1:43 PM
>>>>> To: opensim-dev at opensimulator.org
>>>>> Subject: Re: [Opensim-dev] Packet Pooling - Should it work?
>>>>>
>>>>> PacketPool.cs code has been in usage, and still is in same packets, but
>>>> of
>>>>> limited usefulness
>>>>> in same cases it could even be GC induced pseudo memory leak.
>>>>>
>>>>> It was replaced by a simpler pool of memory buffers (actually libomv
>>>>> UDPPacketBuffers, but not its objectpool) on most sent packets and
>>>> receive
>>>>> buffers
>>>>> also used as temporary work buffers on a few other places.
>>>>> Most send packets have nothing to reuse but the buffer.
>>>>>
>>>>> Try running opensim in Workstation (desktop on mono) mode.
>>>>> Server mode heuristics don't seem to match opensim needs that well.
>>>>>
>>>>> Ubit
>>>>>
>>>>> On 13-Aug-19 16:04, Mike Dickson wrote:
>>>>>> I've been investigating UDP stalls for a while now and at least in some
>>>>>> cases I'm fairly convinced some cases occur due to GC pauses. There is
>>>>>> some packet pooling code in the underlying LibOMV probably originally
>>>>>> derived from work done on Halcyon to address this case. I don't see
>>>> any
>>>>>> attempt in the UDP comms to make use of these buffer pools.
>>>>>>
>>>>>> There is seperate code in PacketPool.cs to, I think reuse packet
>>>> buffers
>>>>>> based on a couple of buffer sizes and it looks like this should be on
>>>> by
>>>>>> default but I can't find any evidence by looking at status of any
>>>> packet
>>>>>> reuse occuring. That is it looks like there is code there but it's
>>>>> either
>>>>>> switched off somewhere else or just doesn't work (or the stats are
>>>> wrong
>>>>> :).
>>>>>>
>>>>>> Should this PacketPooling be functional? Alternatively has any attempt
>>>>>> been made to wire in the PacketBuffer support thats already in LibOMV?
>>>>>> I'm going to dig through all this as I have time but I figured a
>>>> little
>>>>>> information might help short circuit some paths and direct my search.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Mike
>>>>>> _______________________________________________
>>>>>> Opensim-dev mailing list
>>>>>> Opensim-dev at opensimulator.org
>>>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>>>>
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at opensimulator.org
>>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at opensimulator.org
>>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at opensimulator.org
>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at opensimulator.org
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>>
>>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>>
>>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
More information about the Opensim-dev
mailing list