0008461: Powerup slows down after a while

(0033892)
BillBlight (developer)
2019-01-21 12:48

Are you using Warp3d map?

Ubit has said that the new Warp3d is MUCH slower on map generation than the older one, but makes much nicer maps ..

(0033901)
contributor (reporter)
2019-01-21 23:49
edited on: 2019-01-21 23:51

@BillBlight I'm not using Warp3d. I use the default MapImageModule.

(0033919)
paela argus (reporter)
2019-01-23 08:58

Os Used ?

(0033920)
BillBlight (developer)
2019-01-23 09:02

Would also be curious as to your installed .NET version since you say .NET / Windows64

(0033921)
contributor (reporter)
2019-01-23 09:16

I'm currently using Net 4.7.2 under Windows64.

(0033922)
BillBlight (developer)
2019-01-23 09:35

What windows 64, there are many ...

(0033927)
contributor (reporter)
2019-01-23 23:19

I forgot to mention it, it is Windows 7 x64.

(0033928)
tampa (reporter)
2019-01-24 00:43

It is normal for the startup to slow down once it gets to starting scripts, loading objects, informing neighbors, during which logins are disabled. There is a switch to enable logins regardless of the startup state, though you may find scripts not yet started when entering such a region. The only thing I can think of why startup time would triple is if the scripts were being recompiled each time, which should not occur if the script cache is in use. You may want to test adjusting the script engine settings in the OpenSim.ini, this may result in better behavior(there used to be a recommended setting outside of the default one, but I believe that is no longer the case).

Could you perhaps attach a log of your startup with some markers inserted of where exactly the "slowdown" occurs? This way we may be able to determine more closely if this is an actual bug or just misconfiguration.

(0033929)
contributor (reporter)
2019-01-24 01:55

I noticed that the slowdown starts when the Watchdog procedures become active. This takes one or two minutes depending on the number of regions.

I would suggest that the Watchdog procedures should be activated at the end of the load session.

The configuration I use is the same of the configuration I had with previous opensim versions and scripts are not recompiled every time.

(0033930)
BillBlight (developer)
2019-01-24 01:57
edited on: 2019-01-24 01:58

The watchdog can be slowed by load, and unresponsive threads , so if the watchdog is kicking in that often there is some type of load causing it ...

(0033931)
contributor (reporter)
2019-01-24 02:37

I uploaded a clean log file.

Line 5672 is were the slowdown begins.
Instead of multiple lines per second, we see clearly thereafter one or two lines per second only.
Notice that the delay starts when WATCHDOG Heartbeat begins for the first time.

(0033932)
danbanner (manager)
2019-01-24 02:56

60 regions in a single simulator?

(0033933)
contributor (reporter)
2019-01-24 03:07

@danbanner Yes, the machine is Ryzen with 64GB RAM and no avatars except myself.

(0033934)
danbanner (manager)
2019-01-24 03:18

so you have a powerful machine that could handle several simulators and distribute the load better but you're attempting to run all the regions in one instance?
one instance of opensimulator has limitations.

(0033935)
Monamusa Kaliopov (reporter)
2019-01-24 03:21

http://opensimulator.org/wiki/Performance [^]

(0033936)
contributor (reporter)
2019-01-24 03:35

@danbanner I am fully aware that I could split the regions into multiple simulators. In fact I have already done it. Each of the multiple instances have its own critical point were delay starts.
I have sent you the worst case scenario, where all the regions were in one simulator in order to stress the problem.

(0033939)
paela argus (reporter)
2019-01-24 04:43

to many simulator for a basic amd ryzen

(0033940)
paela argus (reporter)
2019-01-24 04:48

isn't really a problem of Xengine your mysql are very busy for start that all sim in same tables please see what dan said

(0033942)
UbitUmarov (administrator)
2019-01-24 07:04

watch memory usage during load with task manager

(0033943)
contributor (reporter)
2019-01-24 07:29

Memory use is only 30% of the available memory and no windows pagefile exists.

In my opinion, it has to do with Watchdog Heartbeat activation.

Notes
(0033892) BillBlight (developer) 2019-01-21 12:48	Are you using Warp3d map? Ubit has said that the new Warp3d is MUCH slower on map generation than the older one, but makes much nicer maps ..

(0033901) contributor (reporter) 2019-01-21 23:49 edited on: 2019-01-21 23:51	@BillBlight I'm not using Warp3d. I use the default MapImageModule.

(0033919) paela argus (reporter) 2019-01-23 08:58	Os Used ?

(0033920) BillBlight (developer) 2019-01-23 09:02	Would also be curious as to your installed .NET version since you say .NET / Windows64

(0033921) contributor (reporter) 2019-01-23 09:16	I'm currently using Net 4.7.2 under Windows64.

(0033922) BillBlight (developer) 2019-01-23 09:35	What windows 64, there are many ...

(0033927) contributor (reporter) 2019-01-23 23:19	I forgot to mention it, it is Windows 7 x64.

(0033928) tampa (reporter) 2019-01-24 00:43	It is normal for the startup to slow down once it gets to starting scripts, loading objects, informing neighbors, during which logins are disabled. There is a switch to enable logins regardless of the startup state, though you may find scripts not yet started when entering such a region. The only thing I can think of why startup time would triple is if the scripts were being recompiled each time, which should not occur if the script cache is in use. You may want to test adjusting the script engine settings in the OpenSim.ini, this may result in better behavior(there used to be a recommended setting outside of the default one, but I believe that is no longer the case). Could you perhaps attach a log of your startup with some markers inserted of where exactly the "slowdown" occurs? This way we may be able to determine more closely if this is an actual bug or just misconfiguration.

(0033929) contributor (reporter) 2019-01-24 01:55	I noticed that the slowdown starts when the Watchdog procedures become active. This takes one or two minutes depending on the number of regions. I would suggest that the Watchdog procedures should be activated at the end of the load session. The configuration I use is the same of the configuration I had with previous opensim versions and scripts are not recompiled every time.

(0033930) BillBlight (developer) 2019-01-24 01:57 edited on: 2019-01-24 01:58	The watchdog can be slowed by load, and unresponsive threads , so if the watchdog is kicking in that often there is some type of load causing it ...

(0033931) contributor (reporter) 2019-01-24 02:37	I uploaded a clean log file. Line 5672 is were the slowdown begins. Instead of multiple lines per second, we see clearly thereafter one or two lines per second only. Notice that the delay starts when WATCHDOG Heartbeat begins for the first time.

(0033932) danbanner (manager) 2019-01-24 02:56	60 regions in a single simulator?

(0033933) contributor (reporter) 2019-01-24 03:07	@danbanner Yes, the machine is Ryzen with 64GB RAM and no avatars except myself.

(0033934) danbanner (manager) 2019-01-24 03:18	so you have a powerful machine that could handle several simulators and distribute the load better but you're attempting to run all the regions in one instance? one instance of opensimulator has limitations.

(0033935) Monamusa Kaliopov (reporter) 2019-01-24 03:21	http://opensimulator.org/wiki/Performance [^]

(0033936) contributor (reporter) 2019-01-24 03:35	@danbanner I am fully aware that I could split the regions into multiple simulators. In fact I have already done it. Each of the multiple instances have its own critical point were delay starts. I have sent you the worst case scenario, where all the regions were in one simulator in order to stress the problem.

(0033939) paela argus (reporter) 2019-01-24 04:43	to many simulator for a basic amd ryzen

(0033940) paela argus (reporter) 2019-01-24 04:48	isn't really a problem of Xengine your mysql are very busy for start that all sim in same tables please see what dan said

(0033942) UbitUmarov (administrator) 2019-01-24 07:04	watch memory usage during load with task manager

(0033943) contributor (reporter) 2019-01-24 07:29	Memory use is only 30% of the available memory and no windows pagefile exists. In my opinion, it has to do with Watchdog Heartbeat activation.

Issue History
Date Modified	Username	Field	Change
2019-01-21 10:37	contributor	New Issue
2019-01-21 12:48	BillBlight	Note Added: 0033892
2019-01-21 23:49	contributor	Note Added: 0033901
2019-01-21 23:51	contributor	Note Edited: 0033901	View Revisions
2019-01-23 08:58	paela argus	Note Added: 0033919
2019-01-23 09:02	BillBlight	Note Added: 0033920
2019-01-23 09:16	contributor	Note Added: 0033921
2019-01-23 09:35	BillBlight	Note Added: 0033922
2019-01-23 23:19	contributor	Note Added: 0033927
2019-01-24 00:43	tampa	Note Added: 0033928
2019-01-24 01:55	contributor	Note Added: 0033929
2019-01-24 01:57	BillBlight	Note Added: 0033930
2019-01-24 01:58	BillBlight	Note Edited: 0033930	View Revisions
2019-01-24 02:36	contributor	File Added: OpenSim.log
2019-01-24 02:37	contributor	Note Added: 0033931
2019-01-24 02:56	danbanner	Note Added: 0033932
2019-01-24 03:07	contributor	Note Added: 0033933
2019-01-24 03:18	danbanner	Note Added: 0033934
2019-01-24 03:21	Monamusa Kaliopov	Note Added: 0033935
2019-01-24 03:35	contributor	Note Added: 0033936
2019-01-24 04:43	paela argus	Note Added: 0033939
2019-01-24 04:48	paela argus	Note Added: 0033940
2019-01-24 07:04	UbitUmarov	Note Added: 0033942
2019-01-24 07:29	contributor	Note Added: 0033943

Relationships