0008307: Unhandled exception and then sim crash

0008307

opensim

[REGION] OpenSim Core

public

2018-03-27 06:39

2018-03-27 08:07

Reporter

kenvc

Assigned To

Priority

normal

Severity

crash

Reproducibility

sometimes

Status

new

Resolution

open

Platform

Dual xeon CPUs 32GB ram

Operating System

Windows

Operating System Version

Server 2012

Product Version

master (dev code)

Git Revision or version number

Run Mode

Grid (Multiple Regions per Sim)

Physics Engine

ubODE

Script Engine

Environment

.NET / Windows64

Mono Version

None

Viewer

N/A

Summary

0008307: Unhandled exception and then sim crash

Description

Sometimes the sim has this unhandled exception and terminates. Log info is included below

Steps To Reproduce

Not sure how to reproduce this. It just happens sometimes with no warning. See the applicable part of the log file included below.

Additional Information

Time now is 03/26/2018 6:19:40 PM
Server has been running since Tuesday, 03/20/2018 11:19:48 PM
That is an elapsed time of 5.18:59:51.7402521

CONNECTION STATISTICS
Client logouts due to no data receive timeout: 5

SAMPLE FRAME STATISTICS
Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr ScrLPS
  1.00 55 55.1 0.0 1 0 7740 0 148 29

PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt AgntFt ImgsFt
     1 2 0 0 0 18.2 0.0 0.5 0.0 0.0 0.0

MEMORY STATISTICS
Heap allocated to OpenSim : 580 MB
Last heap allocation rate : 0.364 MB/s
Average heap allocation rate: 1.463 MB/s
Process memory: Physical 1597 MB Paged 1910 MB Virtual 3119 MB
Peak process memory: Physical 3243 MB Paged 3232 MB Virtual 4076 MB

44 threads are being tracked:
    ID NAME LAST UPDATE (MS) LIFETIME (MS) PRIORITY STATE
    11 PollServiceWorkerThread 0:20000 2824 500401553 Normal Background, WaitSleepJoin
    12 PollServiceWorkerThread 1:20000 2715 500401553 Normal Background, WaitSleepJoin
    13 PollServiceWatcherThread:20000 94 500401553 Normal Background, WaitSleepJoin
    18 GetTextureWorker0 2449 500400258 Normal Background, WaitSleepJoin
    19 GetTextureWorker1 2980 500400258 Normal Background, WaitSleepJoin
    20 GetTextureWorker0 2699 500400226 Normal Background, WaitSleepJoin
    21 GetTextureWorker1 2980 500400226 Normal Background, WaitSleepJoin
    27 MapItemRequestThread (Vintage Lace) 1919 500398776 BelowNormal Background, WaitSleepJoin
    28 MapBlockSendThread (Vintage Lace) 4696 500398776 BelowNormal Background, WaitSleepJoin
    30 Incoming Packets (Vintage Lace) 203 500398635 Highest WaitSleepJoin
    31 Outgoing Packets (Vintage Lace) 0 500398635 Highest WaitSleepJoin
    32 GetMeshWorker0 3510 500398620 Normal Background, WaitSleepJoin
    33 GetMeshWorker1 3510 500398604 Normal Background, WaitSleepJoin
    34 GetTextureWorker0 1607 500398604 Normal Background, WaitSleepJoin
    35 GetTextureWorker1 1404 500398604 Normal Background, WaitSleepJoin
    36 InventoryWorkerThread0 3588 500398495 Normal Background, WaitSleepJoin
    37 InventoryWorkerThread1 2699 500398495 Normal Background, WaitSleepJoin
     6 MapItemRequestThread (Jazzys Island) 1373 500386233 BelowNormal Background, WaitSleepJoin
    45 MapBlockSendThread (Jazzys Island) 1950 500386233 BelowNormal Background, WaitSleepJoin
    47 Incoming Packets (Jazzys Island) 156 500386218 Highest WaitSleepJoin
    48 Outgoing Packets (Jazzys Island) 94 500386218 Highest WaitSleepJoin
    51 MapItemRequestThread (Schnargleflatz) 1747 500375235 BelowNormal Background, WaitSleepJoin
    52 MapBlockSendThread (Schnargleflatz) 31 500375235 BelowNormal Background, WaitSleepJoin
    54 Incoming Packets (Schnargleflatz) 94 500375235 Highest WaitSleepJoin
    55 Outgoing Packets (Schnargleflatz) 94 500375235 Highest WaitSleepJoin
    56 MapItemRequestThread (Vicky Dreams) 375 500372131 BelowNormal Background, WaitSleepJoin
    57 MapBlockSendThread (Vicky Dreams) 3744 500372131 BelowNormal Background, WaitSleepJoin
    59 Incoming Packets (Vicky Dreams) 219 500372131 Highest WaitSleepJoin
    60 Outgoing Packets (Vicky Dreams) 0 500372115 Highest WaitSleepJoin
    63 MapItemRequestThread (ARAUCANA) 2699 500361788 BelowNormal Background, WaitSleepJoin
    64 MapBlockSendThread (ARAUCANA) 1997 500361788 BelowNormal Background, WaitSleepJoin
    66 Incoming Packets (ARAUCANA) 156 500361788 Highest WaitSleepJoin
    67 Outgoing Packets (ARAUCANA) 94 500361788 Highest WaitSleepJoin
    78 MapItemRequestThread (Rissys Island) 2730 500350899 BelowNormal Background, WaitSleepJoin
    79 MapBlockSendThread (Rissys Island) 4696 500350899 BelowNormal Background, WaitSleepJoin
    81 Incoming Packets (Rissys Island) 156 500350899 Highest WaitSleepJoin
    82 Outgoing Packets (Rissys Island) 94 500350899 Highest WaitSleepJoin
    83 Heartbeat-(Vintage_Lace) 78 500342366 Normal WaitSleepJoin
    88 AsyncLSLCmdHandlerThread 0 500326407 Normal Background, WaitSleepJoin
    97 Heartbeat-(Jazzys_Island) 31 500304348 Normal WaitSleepJoin
   103 Heartbeat-(Schnargleflatz) 16 500301041 Normal WaitSleepJoin
    50 Heartbeat-(Vicky_Dreams) 63 500231168 Normal WaitSleepJoin
   140 Heartbeat-(ARAUCANA) 47 500205225 Normal WaitSleepJoin
   112 Heartbeat-(Rissys_Island) 63 500177597 Normal WaitSleepJoin

Framework main threadpool
workers: 9 (1000 / 8)
Completion: 0 (1000 / 8)

Thread pool used: Framework main threadpool

Total process threads active: 354

2018-03-26 18:19:57,771 INFO [WEB UTIL]: Slow ServiceOSD request 41319 QUERYACCESS http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] took 21014ms, 0ms writing(0 at Json; 0 at comp), 445 bytes (445 uncomp): {"position":"<255.4192, 153.041, 21.34339>","my_version":"SIMULATION/0.3","simulation_service_supported_min":0.300000011920929,"simulation_service_supported_max":0.600000023841858,"simulation_service_
2018-03-26 18:19:57,771 DEBUG [LOGHTTP]: JSON-RPC request 41319 QUERYACCESS to http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] FAILED: Unable to connect to the remote server
2018-03-26 18:20:27,817 WARN [LLUDPSERVER]: No packets received from root agent of Henry Prix for 60000ms in Vicky Dreams. Disconnecting.
2018-03-26 18:20:28,332 DEBUG [CLIENT]: Close has been called for Henry Prix attached to scene Vicky Dreams
2018-03-26 18:20:28,862 DEBUG [JobEngine] Stopping AsyncInUDP-8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7
2018-03-26 18:20:28,862 DEBUG [SCENE]: Removing root agent Henry Prix 8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7 from Vicky Dreams
2018-03-26 18:20:28,862 DEBUG [CAPS]: Remove caps for agent 8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7 in region Vicky Dreams
2018-03-26 18:20:28,878 DEBUG [HG MAP]: Resetting 404 blocks
2018-03-26 18:20:28,878 DEBUG [Scene] The avatar has left the building
2018-03-26 18:20:29,985 DEBUG [HG STATUS NOTIFIER]: Notifying 1 friends in http://world.narasnook.com:8900/ [^]
2018-03-26 18:20:31,795 DEBUG [HG STATUS NOTIFIER]: Notifying 2 friends in http://hg.zetaworlds.com:80/ [^]
2018-03-26 18:20:33,870 INFO [WEB UTIL]: Slow ServiceOSD request 41321 QUERYACCESS http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] took 21029ms, 0ms writing(0 at Json; 0 at comp), 445 bytes (445 uncomp): {"position":"<255.506, 153.0092, 21.35902>","my_version":"SIMULATION/0.3","simulation_service_supported_min":0.300000011920929,"simulation_service_supported_max":0.600000023841858,"simulation_service_
2018-03-26 18:20:33,870 DEBUG [LOGHTTP]: JSON-RPC request 41321 QUERYACCESS to http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] FAILED: Unable to connect to the remote server
2018-03-26 18:20:34,681 ERROR [APPLICATION]:
APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs

Exception: System.NullReferenceException: Object reference not set to an instance of an object.

Server stack trace:
   at OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule.CrossAsync(ScenePresence agent, Boolean isFlying) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\CoreModules\Framework\EntityTransfer\EntityTransferModule.cs:line 1580
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
   at OpenSim.Region.Framework.Interfaces.CrossAsyncDelegate.EndInvoke(IAsyncResult result)
   at OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule.CrossCompleted(IAsyncResult iar) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\CoreModules\Framework\EntityTransfer\EntityTransferModule.cs:line 1550
   at System.Runtime.Remoting.Messaging.AsyncResult.SyncProcessMessage(IMessage msg)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Application is terminating: True

2018-03-26 18:20:35,523 INFO [FORMS]: Slow request 41333 POST http://hg.zetaworlds.com:80/hgfriends [^] took 3307ms, 0ms writing, 3151
2018-03-26 18:20:35,523 DEBUG [HG STATUS NOTIFIER]: Notifying 1 friends in http://lfgrid.com:8002/ [^]

MantisBT - opensim
View Issue Details

ID	Project	Category	View Status	Date Submitted	Last Update
0008307	opensim	[REGION] OpenSim Core	public	2018-03-27 06:39	2018-03-27 08:07

Reporter	kenvc
Assigned To
Priority	normal	Severity	crash	Reproducibility	sometimes
Status	new	Resolution	open
Platform	Dual xeon CPUs 32GB ram	Operating System	Windows	Operating System Version	Server 2012
Product Version	master (dev code)
Target Version		Fixed in Version
Git Revision or version number
Run Mode	Grid (Multiple Regions per Sim)
Physics Engine	ubODE
Script Engine
Environment	.NET / Windows64
Mono Version	None
Viewer	N/A

Summary	0008307: Unhandled exception and then sim crash
Description	Sometimes the sim has this unhandled exception and terminates. Log info is included below
Steps To Reproduce	Not sure how to reproduce this. It just happens sometimes with no warning. See the applicable part of the log file included below.
Additional Information	Time now is 03/26/2018 6:19:40 PM Server has been running since Tuesday, 03/20/2018 11:19:48 PM That is an elapsed time of 5.18:59:51.7402521 CONNECTION STATISTICS Client logouts due to no data receive timeout: 5 SAMPLE FRAME STATISTICS Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr ScrLPS 1.00 55 55.1 0.0 1 0 7740 0 148 29 PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt AgntFt ImgsFt 1 2 0 0 0 18.2 0.0 0.5 0.0 0.0 0.0 MEMORY STATISTICS Heap allocated to OpenSim : 580 MB Last heap allocation rate : 0.364 MB/s Average heap allocation rate: 1.463 MB/s Process memory: Physical 1597 MB Paged 1910 MB Virtual 3119 MB Peak process memory: Physical 3243 MB Paged 3232 MB Virtual 4076 MB 44 threads are being tracked: ID NAME LAST UPDATE (MS) LIFETIME (MS) PRIORITY STATE 11 PollServiceWorkerThread 0:20000 2824 500401553 Normal Background, WaitSleepJoin 12 PollServiceWorkerThread 1:20000 2715 500401553 Normal Background, WaitSleepJoin 13 PollServiceWatcherThread:20000 94 500401553 Normal Background, WaitSleepJoin 18 GetTextureWorker0 2449 500400258 Normal Background, WaitSleepJoin 19 GetTextureWorker1 2980 500400258 Normal Background, WaitSleepJoin 20 GetTextureWorker0 2699 500400226 Normal Background, WaitSleepJoin 21 GetTextureWorker1 2980 500400226 Normal Background, WaitSleepJoin 27 MapItemRequestThread (Vintage Lace) 1919 500398776 BelowNormal Background, WaitSleepJoin 28 MapBlockSendThread (Vintage Lace) 4696 500398776 BelowNormal Background, WaitSleepJoin 30 Incoming Packets (Vintage Lace) 203 500398635 Highest WaitSleepJoin 31 Outgoing Packets (Vintage Lace) 0 500398635 Highest WaitSleepJoin 32 GetMeshWorker0 3510 500398620 Normal Background, WaitSleepJoin 33 GetMeshWorker1 3510 500398604 Normal Background, WaitSleepJoin 34 GetTextureWorker0 1607 500398604 Normal Background, WaitSleepJoin 35 GetTextureWorker1 1404 500398604 Normal Background, WaitSleepJoin 36 InventoryWorkerThread0 3588 500398495 Normal Background, WaitSleepJoin 37 InventoryWorkerThread1 2699 500398495 Normal Background, WaitSleepJoin 6 MapItemRequestThread (Jazzys Island) 1373 500386233 BelowNormal Background, WaitSleepJoin 45 MapBlockSendThread (Jazzys Island) 1950 500386233 BelowNormal Background, WaitSleepJoin 47 Incoming Packets (Jazzys Island) 156 500386218 Highest WaitSleepJoin 48 Outgoing Packets (Jazzys Island) 94 500386218 Highest WaitSleepJoin 51 MapItemRequestThread (Schnargleflatz) 1747 500375235 BelowNormal Background, WaitSleepJoin 52 MapBlockSendThread (Schnargleflatz) 31 500375235 BelowNormal Background, WaitSleepJoin 54 Incoming Packets (Schnargleflatz) 94 500375235 Highest WaitSleepJoin 55 Outgoing Packets (Schnargleflatz) 94 500375235 Highest WaitSleepJoin 56 MapItemRequestThread (Vicky Dreams) 375 500372131 BelowNormal Background, WaitSleepJoin 57 MapBlockSendThread (Vicky Dreams) 3744 500372131 BelowNormal Background, WaitSleepJoin 59 Incoming Packets (Vicky Dreams) 219 500372131 Highest WaitSleepJoin 60 Outgoing Packets (Vicky Dreams) 0 500372115 Highest WaitSleepJoin 63 MapItemRequestThread (ARAUCANA) 2699 500361788 BelowNormal Background, WaitSleepJoin 64 MapBlockSendThread (ARAUCANA) 1997 500361788 BelowNormal Background, WaitSleepJoin 66 Incoming Packets (ARAUCANA) 156 500361788 Highest WaitSleepJoin 67 Outgoing Packets (ARAUCANA) 94 500361788 Highest WaitSleepJoin 78 MapItemRequestThread (Rissys Island) 2730 500350899 BelowNormal Background, WaitSleepJoin 79 MapBlockSendThread (Rissys Island) 4696 500350899 BelowNormal Background, WaitSleepJoin 81 Incoming Packets (Rissys Island) 156 500350899 Highest WaitSleepJoin 82 Outgoing Packets (Rissys Island) 94 500350899 Highest WaitSleepJoin 83 Heartbeat-(Vintage_Lace) 78 500342366 Normal WaitSleepJoin 88 AsyncLSLCmdHandlerThread 0 500326407 Normal Background, WaitSleepJoin 97 Heartbeat-(Jazzys_Island) 31 500304348 Normal WaitSleepJoin 103 Heartbeat-(Schnargleflatz) 16 500301041 Normal WaitSleepJoin 50 Heartbeat-(Vicky_Dreams) 63 500231168 Normal WaitSleepJoin 140 Heartbeat-(ARAUCANA) 47 500205225 Normal WaitSleepJoin 112 Heartbeat-(Rissys_Island) 63 500177597 Normal WaitSleepJoin Framework main threadpool workers: 9 (1000 / 8) Completion: 0 (1000 / 8) Thread pool used: Framework main threadpool Total process threads active: 354 2018-03-26 18:19:57,771 INFO [WEB UTIL]: Slow ServiceOSD request 41319 QUERYACCESS http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] took 21014ms, 0ms writing(0 at Json; 0 at comp), 445 bytes (445 uncomp): {"position":"<255.4192, 153.041, 21.34339>","my_version":"SIMULATION/0.3","simulation_service_supported_min":0.300000011920929,"simulation_service_supported_max":0.600000023841858,"simulation_service_ 2018-03-26 18:19:57,771 DEBUG [LOGHTTP]: JSON-RPC request 41319 QUERYACCESS to http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] FAILED: Unable to connect to the remote server 2018-03-26 18:20:27,817 WARN [LLUDPSERVER]: No packets received from root agent of Henry Prix for 60000ms in Vicky Dreams. Disconnecting. 2018-03-26 18:20:28,332 DEBUG [CLIENT]: Close has been called for Henry Prix attached to scene Vicky Dreams 2018-03-26 18:20:28,862 DEBUG [JobEngine] Stopping AsyncInUDP-8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7 2018-03-26 18:20:28,862 DEBUG [SCENE]: Removing root agent Henry Prix 8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7 from Vicky Dreams 2018-03-26 18:20:28,862 DEBUG [CAPS]: Remove caps for agent 8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7 in region Vicky Dreams 2018-03-26 18:20:28,878 DEBUG [HG MAP]: Resetting 404 blocks 2018-03-26 18:20:28,878 DEBUG [Scene] The avatar has left the building 2018-03-26 18:20:29,985 DEBUG [HG STATUS NOTIFIER]: Notifying 1 friends in http://world.narasnook.com:8900/ [^] 2018-03-26 18:20:31,795 DEBUG [HG STATUS NOTIFIER]: Notifying 2 friends in http://hg.zetaworlds.com:80/ [^] 2018-03-26 18:20:33,870 INFO [WEB UTIL]: Slow ServiceOSD request 41321 QUERYACCESS http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] took 21029ms, 0ms writing(0 at Json; 0 at comp), 445 bytes (445 uncomp): {"position":"<255.506, 153.0092, 21.35902>","my_version":"SIMULATION/0.3","simulation_service_supported_min":0.300000011920929,"simulation_service_supported_max":0.600000023841858,"simulation_service_ 2018-03-26 18:20:33,870 DEBUG [LOGHTTP]: JSON-RPC request 41321 QUERYACCESS to http://Moonglow.uk.to:10500/agent/8aae2fa9-a9bd-4140-9dfe-f7ca88a1e5f7/c0d1ee74-0e21-4c61-bd2d-4f0fe95e5737/ [^] FAILED: Unable to connect to the remote server 2018-03-26 18:20:34,681 ERROR [APPLICATION]: APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs Exception: System.NullReferenceException: Object reference not set to an instance of an object. Server stack trace: at OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule.CrossAsync(ScenePresence agent, Boolean isFlying) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\CoreModules\Framework\EntityTransfer\EntityTransferModule.cs:line 1580 at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase) at OpenSim.Region.Framework.Interfaces.CrossAsyncDelegate.EndInvoke(IAsyncResult result) at OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule.CrossCompleted(IAsyncResult iar) in c:\Users\Administrator\Desktop\Opensim\Opensim Dev Master\OpenSim\Region\CoreModules\Framework\EntityTransfer\EntityTransferModule.cs:line 1550 at System.Runtime.Remoting.Messaging.AsyncResult.SyncProcessMessage(IMessage msg) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() Application is terminating: True 2018-03-26 18:20:35,523 INFO [FORMS]: Slow request 41333 POST http://hg.zetaworlds.com:80/hgfriends [^] took 3307ms, 0ms writing, 3151 2018-03-26 18:20:35,523 DEBUG [HG STATUS NOTIFIER]: Notifying 1 friends in http://lfgrid.com:8002/ [^]
Tags	No tags attached.
Relationships
Attached Files

Issue History
Date Modified	Username	Field	Change
2018-03-27 06:39	kenvc	New Issue
2018-03-27 08:07	UbitUmarov	Note Added: 0032611