[Opensim-dev] Memory cache
diva at metaverseink.com
diva at metaverseink.com
Thu May 21 21:04:42 UTC 2009
You're right, Mic! There seems to be have been not one but *two*
different rats in the closet that no one noticed before. One may be that
the cache was/is simply not caching anything -- hence no difference in
asset server hits. This needs to be checked.
But the other may have to do with a critical change that Melanie did in
the way assets are now fetched from the server, and which hadn't
register in my mind until you pointed this out. Before, there was this
one thread asset fetcher object which had a request queue and went
through it one by one. Melanie did away with that completely. There is
no one-thread fetcher. We now have an asynchronous REST GET handler,
whose concurrency is now dealt at a lower thread pool level. Because of
this, we now send out many more asset requests than before!
So, there was, indeed, a major change and it's having noticeable
performance gains.
Mic Bowman wrote:
> so removing the asset cache from a "reasonable" number of regions has
> not changed the number of requests to the asset server. it sure sounds
> like asset cache was not absorbing many of the asset requests.
>
> regarding the performance... nothing concrete.. but the way the
> process queue function in the cache code looks to be written right
> now, it would process at most five asset requests then sleep for half
> a second before it processed any others. if the cache was not
> absorbing any requests... then the best you're going to get is 10
> assets per second. which for a complex scene might take awhile.
>
> --mic
>
>
> On Thu, May 21, 2009 at 11:57 AM, Nebadon Izumi <nebadon2025 at gmail.com> wrote:
>> osgrid has seen 0% increase in traffic to the asset server.
>>
>> http://assets.osgrid.org/stats
>>
>> have a look for yourself.
>>
>>
>> On Thu, May 21, 2009 at 11:55 AM, Melanie <melanie at t-data.com> wrote:
>>> Indeed, it might be it's not really working....
>>>
>>> Melanie
>>>
>>>
>>> Mic Bowman wrote:
>>>> is there any instrumentation in the asset cache about hit rate? the
>>>> fact that there is no measurable increase reported on the osgrid asset
>>>> server with the caches turned off suggests that the cache isn't really
>>>> absorbing many hits (either because its not working the way its
>>>> expected to or because the re-use isn't significant).
>>>>
>>>> --mic
>>>>
>>>>
>>>> On Thu, May 21, 2009 at 11:43 AM, <diva at metaverseink.com> wrote:
>>>>> The cache, as configured, has 32K asset slots, which is a lot. So it
>>>>> takes a while until the cache releases some objects for the GC to
>>>>> collect. I think there's something else going on with that cache --
>>>>> probably too many locking conflicts between threads, or something like
>>>>> that. Whatever it is, it's much heavier than fetching the assets from
>>>>> the remote asset server, which is amazing. And another amazing thing is
>>>>> that remote fetching of so many assets at the same time (whenever a
>>>>> client connects) has such good observable performance for the user.
>>>>>
>>>>> If the purpose of the cache is to go easy on the asset server (not
>>>>> necessarily improve the observable performance, which is already very
>>>>> good without the mem cache) it might make more sense to use a file
>>>>> system / local DB instead of a memory cache.
>>>>>
>>>>> Stefan Andersson wrote:
>>>>>> If the cache is releasing objects, I think we might well be seeing
>>>>>> LOH fragmentation.
>>>>>>
>>>>>> http://msdn.microsoft.com/en-us/magazine/cc534993.aspx
>>>>>>
>>>>>> The solution would then be to break the binaries apart into byte cucks
>>>>>> smaller than the LOH trshold.
>>>>>>
>>>>>> Best regards,
>>>>>> Stefan Andersson
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> > From: adam at deepthink.com.au
>>>>>> > To: opensim-dev at lists.berlios.de
>>>>>> > Date: Wed, 20 May 2009 17:21:51 -0400
>>>>>> > Subject: Re: [Opensim-dev] Memory cache
>>>>>> >
>>>>>> > We are using an external caching library I believe, it could be
>>>>>> sitting in there.
>>>>>> >
>>>>>> > Adam
>>>>>> >
>>>>>> > > -----Original Message-----
>>>>>> > > From: opensim-dev-bounces at lists.berlios.de [mailto:opensim-dev-
>>>>>> > > bounces at lists.berlios.de] On Behalf Of Melanie
>>>>>> > > Sent: Wednesday, 20 May 2009 1:56 PM
>>>>>> > > To: opensim-dev at lists.berlios.de
>>>>>> > > Subject: Re: [Opensim-dev] Memory cache
>>>>>> > >
>>>>>> > > The surprise really is that the cache would be a performance loss
>>>>>> > > instead of a performance gain. What is there that makes it so
>>>>>> slow?
>>>>>> > >
>>>>>> > > Melanie
>>>>>> > >
>>>>>> > > Frisby, Adam wrote:
>>>>>> > > > As chief engineer aboard the USS OSgrid, I might want to
>>>>>> recommend
>>>>>> > > against this.
>>>>>> > > >
>>>>>> > > > Not having the sims cache assets will mean that every asset
>>>>>> request
>>>>>> > > will hit the core asset server which in turn will result in
>>>>>> higher
>>>>>> > > bandwidth requirements for it in the long run. It doesn't appear
>>>>>> to
>>>>>> > > have made a huge impact on our bandwidth charts for this week yet
>>>>>> -
>>>>>> > > however I wouldn't be surprised if it did if everyone did this.
>>>>>> > > >
>>>>>> > > > Adam
>>>>>> > > >
>>>>>> > > > From: opensim-dev-bounces at lists.berlios.de [mailto:opensim-dev-
>>>>>> > > bounces at lists.berlios.de] On Behalf Of Nebadon Izumi
>>>>>> > > > Sent: Wednesday, 20 May 2009 10:19 AM
>>>>>> > > > To: diva at metaverseink.com; opensim-dev at lists.berlios.de
>>>>>> > > > Subject: Re: [Opensim-dev] Memory cache
>>>>>> > > >
>>>>>> > > > Yay for mistakes that end in gains!!! of course if anyone asks
>>>>>> i
>>>>>> > > planned this ;P anyway i thought people would like to see my ini
>>>>>> > > changes.. this is what lead to the discovery:
>>>>>> > > >
>>>>>> > > > [OpenSim.ini]
>>>>>> > > > ; The following is the configuration section for the new style
>>>>>> > > grid servers
>>>>>> > > > ; If you don't know what this is, don't enable it. It will eat
>>>>>> > > your data,
>>>>>> > > > ; format your hard drive and make all meat in your fridge
>>>>>> spoil.
>>>>>> > > > ; You have been warned.
>>>>>> > > > ; Some of this is starting to work!
>>>>>> > > > [Modules]
>>>>>> > > > ; Choose one
>>>>>> > > > ;AssetServices = "LocalAssetServicesConnector"
>>>>>> > > > AssetServices = "RemoteAssetServicesConnector"
>>>>>> > > > ;AssetServices = "HGAssetBroker"
>>>>>> > > >
>>>>>> > > > ; If you don't want asset caching in the regions, comment this
>>>>>> > > > AssetCaching = "CoreAssetCache" ;<<<---*** SEE ERROR HERE
>>>>>> ***---
>>>>>> > > ->>>
>>>>>> > > >
>>>>>> > > > ; Choose one
>>>>>> > > > ;UserServices = "LocalUserServicesConnector"
>>>>>> > > > UserServices = "RemoteUserServicesConnector"
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > [AssetService]
>>>>>> > > > ; Parameters for local assets, formerly known as "standalone"
>>>>>> > > > LocalServiceModule =
>>>>>> > > "OpenSim.Services.AssetService.dll:AssetService"
>>>>>> > > > StorageProvider = "OpenSim.Data.SQLite.dll"
>>>>>> > > > ;StorageProvider = "OpenSim.Data.MySQL.dll"
>>>>>> > > > ;ConnectionString = "Data
>>>>>> Source=localhost;Database=opensim;User
>>>>>> > > ID=opensim;Password=opensim;"
>>>>>> > > > DefaultAssetLoader =
>>>>>> > > "OpenSim.Framework.AssetLoader.Filesystem.dll"
>>>>>> > > > AssetLoaderArgs = "assets/AssetSets.xml"
>>>>>> > > >
>>>>>> > > > ; Parameters for remote assets, formerly known as "grid"
>>>>>> > > > AssetServerURI = "http://assets.osgrid.org:8003/"
>>>>>> > > >
>>>>>> > > > ; Paremeters for the Hypergrid connector
>>>>>> > > >
>>>>>> > > > ;; Parameters for the HG Broker
>>>>>> > > > ; Use this one if you have a standalone grid
>>>>>> > > > ;LocalGridAssetService =
>>>>>> > > "OpenSim.Services.AssetService.dll:AssetService"
>>>>>> > > > ; Use this one if this sim is connected to a grid-wide asset
>>>>>> > > server
>>>>>> > > > ;LocalGridAssetService =
>>>>>> > > "OpenSim.Services.Connectors.dll:AssetServiceConnector"
>>>>>> > > > ;HypergridAssetService =
>>>>>> > > "OpenSim.Services.AssetService.dll:HGAssetService"
>>>>>> > > >
>>>>>> > > > [AssetCache]
>>>>>> > > > ; Number of buckets for assets
>>>>>> > > > CacheBuckets = 32768
>>>>>> > > >
>>>>>> > > > [UserService]
>>>>>> > > > ;LocalServiceModule =
>>>>>> > > "OpenSim.Services.UserService.dll:UserService"
>>>>>> > > >
>>>>>> > > > [/OpenSim.ini]
>>>>>> > > >
>>>>>> > > > Please test this out and see if you see gains too and let us
>>>>>> know.
>>>>>> > > >
>>>>>> > > > Neb
>>>>>> > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> ---------------------------------------------------------------------
>>>>>> > > ---
>>>>>> > > >
>>>>>> > > > _______________________________________________
>>>>>> > > > Opensim-dev mailing list
>>>>>> > > > Opensim-dev at lists.berlios.de
>>>>>> > > > https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>> > > _______________________________________________
>>>>>> > > Opensim-dev mailing list
>>>>>> > > Opensim-dev at lists.berlios.de
>>>>>> > > https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>> > _______________________________________________
>>>>>> > Opensim-dev mailing list
>>>>>> > Opensim-dev at lists.berlios.de
>>>>>> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> Opensim-dev mailing list
>>>>>> Opensim-dev at lists.berlios.de
>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at lists.berlios.de
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>> --
>> Michael Emory Cerquoni - Nebadon Izumi @ http://osgrid.org
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
More information about the Opensim-dev
mailing list