[Opensim-users] What is going on with the latest release ofopensim for OSgrid

Wed Jun 28 14:33:38 UTC 2017

What kind of connection do you have between the asset server and the
simulators. We have some very large regions that are pack to the brim, they
rarely take 30 min from cold start, a warm start is in minutes before the
login are enabled. Out asset servers are on the same lan as the simulator
servers  It takes about 60 min to soft restart the server.  And this will
delay start up of simulators keeps simulators from locking out the core
code from each other.  We figured .8 to .9 its taks about double the memory
and maybe 25% longer start times. Main due to physic engine

Was looking at a few of our big regions, they take maybe 5 min to load
assets at max, and maybe the same with scripts. This is with 0 cache hits.
and that at teh largest regions we have right now with 30k objects/8k
scripts

We use an older version of .9 that 2 months old, and been waiting for the
release version to upgrade, we have a  bit of work to incorporate our grid
mods into the code.   were in a production environment and have to stay a
step behind the bleeding edge or it will cut deep :)

even now we have to restart the region about 1 to 2 days because of memory
leak and the "creep". The memory leak is when the region grow from 1.4gb
used to 20gb use. The creep is something a little different,  it apears the
region disconnects from the grid, and you can TP to it or out of it.  A
soft restart and it working like new. All I have come with is it being
disconnected from the Asset server and not reconnecting.

Wish you luck, I might be making a new OSGrid region soon and maybe enable
hypergrid who knows our little garden works for us.

David.

On Tue, Jun 27, 2017 at 9:01 PM, Chris <mewtwo0641 at gmail.com> wrote:

> I have been seeing similar issues myself between the past week and a half
> - two weeks on master
>
> Though the previous responses in this thread mention map and asset cache
> population as being a possible issue I'm not so sure that this is the case
> in this example because there is very little rezzed out in my case; 2 of
> the regions are completely empty and the other 2 are sparsely populated
> with random cube prims here and there mostly for script testing.
>
> This is being seen on a test install independent from my main grid (not
> running both at the same time), clean database, 4 regions, grid mode (Not
> HG, 1 OpenSim.exe instance), Windows 7 x64 under .NET, and on ubODE. I'm
> also using the default map tile module (That is, the old one, and not
> Warp3D)
>
> Log File Examples:
>
> 17:05:47 - [REGION LOADER FILE SYSTEM]: Loaded config for region Test
> Sandbox
> 17:05:47 - [LOAD REGIONS PLUGIN]: Loading specific shared modules...
> 17:05:47 - [LOAD REGIONS PLUGIN]: Done.
> 17:05:47 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID:
> 1)
> 17:05:47 - [SCENE]: Default script engine XEngine
> 17:05:47 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0
> seconds
>
> 17:05:47 - [SCENE]: Using the BestAvatarResponsiveness prioritization
> scheme
> 17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
> 17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>.
> Region s
> ize=<256,256>
> 17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from
> Variable2D
> Gzip serialization. Size=<256,256>
> 17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
> 17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>.
> Region s
> ize=<256,256>
> 17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from
> Variable2D
> Gzip serialization. Size=<256,256>
> 17:05:47 - [MODULES]: Loading Region's modules (old style)
> 17:05:47 - [REGIONMODULES]: Loading Region's modules (new style)
> 17:05:47 - [CHAT]: Initialized for Monte Cristo w:10 s:20 S:100
> 17:05:47 - [MESSAGE TRANSFER]: Message transfer module active
> 17:05:47 - [REMOTE GRID USER CONNECTOR]: Enabled remote grid user for
> region Mon
> te Cristo
>
> (A lot of times it gets stuck for a long time here)
>
>
>
> Another log from a different OS session:
>
> 17:07:55 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID:
> 1)
> 17:07:55 - [SCENE]: Default script engine XEngine
> 17:07:55 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0
> seconds
>
> 17:07:55 - [SCENE]: Using the BestAvatarResponsiveness prioritization
> scheme
> 17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
> 17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>.
> Region s
> ize=<256,256>
> 17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from
> Variable2D
> Gzip serialization. Size=<256,256>
> 17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
> 17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>.
> Region s
> ize=<256,256>
> 17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from
> Variable2D
> Gzip serialization. Size=<256,256>
> 17:07:55 - [MODULES]: Loading Region's modules (old style)
> 17:07:55 - [REGIONMODULES]: Loading Region's modules (new style)
>
> (Sometimes it gets stuck for a long time here but not as often as the
> first example)
>
>
>
>
> On 6/25/2017 7:40 PM, AJLDuarte wrote:
>
> Hi
>
>                 Good to read things are not so bad after all
>
>                 About reboots, do them as often as your use case allows
> (and backups of course).
>
>                 Regions memory use keeps piling up.. etc, etc
>
> Ubit
>
>
>
>
>
> *From:* opensim-users-bounces at opensimulator.org [
> mailto:opensim-users-bounces at opensimulator.org
> <opensim-users-bounces at opensimulator.org>] *On Behalf Of *
> tringate at gmail.com
> *Sent:* Monday, June 26, 2017 01:28
> *To:* opensim-users at opensimulator.org
> *Subject:* Re: [Opensim-users] What is going on with the latest release
> ofopensim for OSgrid
>
>
>
> Ubit,
>
>
>
> After checking everything I decided to do a “warm” start, one where it is
> not a total upgrade with all the compiled stuff deleted.  This time the
> restart took 14 minutes which is what I normally observe.  I guess I never
> watched a cold restart before so I guess maybe it always took that long.
> Usually I let the updates happen at the middle of the night restart I have
> scheduled.  I always found everything working just fine in the mornings.
>
>
>
> Sorry for the alarm, but the 70 minute restart time took me by surprise.
>
>
>
> One good thing I did learn from all of this was that it may not be
> necessary to do a reboot every night to prevent lag.  One of my servers ran
> 12 days with no reboot because I had forgot to turn the nightly reboot back
> on during a routine manual maintenance I did earlier this month.
>
>
>
> I may go to a once a week reboot on one server and see how that goes.  If
> all goes well, I’ll change them all to once a week.
>
>
>
> Tom
>
>
>
>
>
> *From:* AJLDuarte
>
> *Sent:* Sunday, June 25, 2017 12:55 PM
>
> *To:* opensim-users at opensimulator.org
>
> *Subject:* Re: [Opensim-users] What is going on with the latest release
> ofopensim for OSgrid
>
>
>
> Hi
>
>                 What was the state of folder bin/AssetsCache on those 2
> tests?
>
>                 According to options, map generation my require full
> assets  fetch from grid and that can take ages if not cached.
>
>                 Of course, make sure FloatSam cache is proper configure.
> It should no longer be optional, since it is needed and others don’t do the
> job now.
>
>
>
> Also seems your t2 is not doing map and not using ubOde ?
>
>                 Even with all assets present in cache map is heavy, can
> take a bit
>
>                 ubOde does operations before scene Ready so that ready
> does mean it a bit more
>
> Other engines do it later so you don’t see it and region Ready really
> doesn’t mean much.
>
> Ubit
>
>
>
>
>
> *From:* opensim-users-bounces at opensimulator.org [
> mailto:opensim-users-bounces at opensimulator.org
> <opensim-users-bounces at opensimulator.org>] *On Behalf Of *
> tringate at gmail.com
> *Sent:* Sunday, June 25, 2017 17:32
> *To:* OS-Opensim Users
> *Subject:* [Opensim-users] What is going on with the latest release of
> opensim for OSgrid
>
>
>
> I just did a update to the latest release of opensim released by OSgrid
> and it took 1 hour and 12 minutes from the time the command was give to
> load before it enabled logins.
>
>
>
> This is on a 4 by 4 var region that previously took 17 minutes to load.
> While that is nothing to write home about, an hour and 12 minutes is
> totally insane.
>
>
>
> This region runs on a dedicated server with nothing else running on it at
> all.  It is a 6 core AMD with 16GB of ram.
>
>
>
> Using “top” to view cpu usage the most I ever noticed it going to as 16%
> total with only one core hitting 100% for a few MS every now and then.
>
>
>
> It looked like it too most og that time generating the 16 tile textures
> for the grid display, but scripts did take about 11 minutes to get
> started.  That part did not appear to be significantly different than
> before.
>
>
>
> Here are some key log entries so you can see how long each thing is taking.
>
>
>
> 2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using
> default OpenSim.exe.config  <—first log entry
>
> 2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381
> prims
>
> 2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563
> objects
>
> 2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the
> datastore
>
> 2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1
>
>
>
>                                 <—big delay
>
> 2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image
> c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1
>
> 2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in
> TSim 1
>
> 2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances
> in TSim 1
>
> 2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor
> operations
>
>
>
>                                 <—big delay
>
> 2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1
>
>
>
>                                 <—big delay but not much different than
> previous version
>
> 2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts
> on TSim 1
>
> 2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready:
> "server_startup,1,21," on channel -800
>
>
>
> This is a very limited log entry list as there are hundreds of entries
> many of them listing bad textures or other things indicating corrupt assets
> coming from the asset server.
>
>
>
> This was a cold start where it does have to generate and compile stuff,
> but it has never taken any where near this long for a region to come online
> before.
>
>
>
> No one else has noticed this huge increase in start-up time?
>
>
>
> I’m running 4 servers, so I tried this same region on my test server, it
> only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on
> that server.
>
>
>
> My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is
> not the cold start time, which does take longer.
>
>
>
> 2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using
> default OpenSim.exe.config
>
> 2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622
> prims
>
> 2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region
> TSim 2
>
> 2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2
>
> 2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts
> on TSim 2
>
> 2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready:
> "server_startup,1,27," on channel -800
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
>
> _______________________________________________
> Opensim-users mailing listOpensim-users at opensimulator.orghttp://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
>
>
> --
> OpenSim: 10 Region Standalone on 0.8.1 Dev
> Physics: Open Dynamics Engine
> OS: Windows 7 (x64)
> CPU: AMD FX 8320 8-Core 3.5 GHz
> Memory: 16 GB DDR3
> Database: MySQL 5.1.63 (x64)
>
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-users/attachments/20170628/e0d8d0b8/attachment-0001.html>