Chat log from the meeting on 2018-06-26

[11:00] Kayaker Magic: I reported a problem last week, and miss-stated it, I though there was a problem with the suitcase and region border crossings, but that was wrong.
[11:01] Kayaker Magic: The problem is that when a HG visitor arrives, robust tries to download the names and properties of their suitcase, and
[11:01] Kayaker Magic: that sometimes happens over and over again for a long time, lagging the system.
[11:02] Kayaker Magic: People with large inventories in their suitcase can set this off, apprently.
[11:02] TG.Lucan @hg.viewtwo.net:8600: Is that all HG visitors?
[11:03] Kayaker Magic: Had one person who put her whole inventory into the suitcase "for convenience" and that was one HG visitor that really set this off.
[11:04] TG.Lucan @hg.viewtwo.net:8600: I see HG visitor where many inventory items cannot be resolved, not all items, which suggests their home grid is a bit wonkey
[11:07] Kayaker Magic: So no thoughts on throttling HG suitcase inventories?
[11:07] Andrew.Hellershanks @hg.osgrid.org: Kayaker, Do you know how many items the person had in their inventory?
[11:08] Kayaker Magic: I think Kitely made changes to their version limiting the number of items in the suitcase to the first 1000. But they are not sharing code.
[11:11] Andrew.Hellershanks @hg.osgrid.org: Kayaker, that sounds like a good idea.
[11:14] Andrew.Hellershanks @hg.osgrid.org: In the past week or two I've heard of two grids that are moving to 0.9 (or about to do so).
[11:14] Andrew.Hellershanks @hg.osgrid.org: 3rd Rock is testing 0.9 in a few regions and Tangle Grid has made the move to 0.9
[11:15] Andrew.Hellershanks @hg.osgrid.org: It makes me hopeful that more grids will be soon moving to 0.9
[11:16] Aiden Frimon: 3rd rock grid is still having some issues with grid crossings but they are still working on that.
[11:16] Andrew.Hellershanks @hg.osgrid.org: ok
[11:18] Andrew.Hellershanks @hg.osgrid.org: Grid crossings might not be just an 0.9 issue.
[11:19] Aiden Frimon: Most are using Firestorm
[11:20] Kayaker Magic: Another grid using 0.9 is Discovery Grid which has been using 0.9.1 dev for a while.
[11:20] Aiden Frimon: It seems that there is a studder when crossing some grids; others, you can cross easily, then if you try to cross back, it logs you out
[11:21] Andrew.Hellershanks @hg.osgrid.org: That reminds me of a region crossing issue from a while back that wasn't an HG one.
[11:23] Kayaker Magic: Discovery Grid has generally good border crossings, except with some mesh vehicles. Might be bad mesh that ubODE cannot create convex hulls for so they found marking some mesh parts physics type none fixed those few problems.
[11:24] Ubit Umarov: oops was afk at phone
[11:24] Aiden Frimon: will look into that
[11:24] Ubit Umarov: that is strange
[11:25] Andrew.Hellershanks @hg.osgrid.org: It would be worth knowing if the same physics engine was being used in both places.
[11:25] Ubit Umarov: about crossings on 0.8 grids, large 0.9 regions will not work well :(
[11:25] Ubit Umarov: guess i already lost some of the hacks to suport that
[11:25] Kayaker Magic: I'll try to get a copy of one of the suspect mesh and give it to Ubit to test with.
[11:25] Ubit Umarov: normal size ones are ok
[11:26] Ubit Umarov: ( issue is that part of the code was moved to 0.9 frid side )
[11:26] Aiden Frimon: The ones that we are testig are VARs
[11:26] Ubit Umarov: code about finding large regions...
[11:27] Ubit Umarov: a sintome is that cross works on one direction only
[11:27] Kayaker Magic: DG has only large var regions, 4x4s and 8x8s.
[11:27] Ubit Umarov: ie wet to east, not east to west
[11:27] Ubit Umarov: also who arrives to the region at east will not see the region at west
[11:28] Kayaker Magic: Yeah, I have seen problems like changed(CHANGED_REGION) only getting called in one direction, not the other.
[11:28] Kayaker Magic: Doesn't always happen, so hard to test.
[11:28] Ubit Umarov: well the issue im telling happens 100'% of the time
[11:29] Ubit Umarov: with 2 recent 0.9 regions on a 0.8 obsolete grid :p
[11:29] Ubit Umarov: the release may still work.. no idea
[11:30] Aiden Frimon: We have an 0.82 grid I believe and are trying to work with 0.9 ... difficult sometimes
[11:31] Ubit Umarov: err and does not log out... its like no region there
[11:31] Aiden Frimon: yeah
[11:31] Ubit Umarov: its the region find code.. returns no region on one direction
[11:32] Ubit Umarov: it target size > 256
[11:32] Ubit Umarov: the mesh case, no idea
[11:33] Ubit Umarov: just large meshes can take time to process, and my induce timeouts.. dunno
[11:33] Ubit Umarov: try rez it on both regions before crossing
[11:34] Ubit Umarov: see if it gets better
[11:34] Aiden Frimon: ok
[11:34] Ubit Umarov: (ubode has its own cache for its internal meshs)
[11:35] Ubit Umarov: hmm and on disk
[11:35] Bill.Blight @grid.opensim.life:8002: I fly and drive mesh vehicles across sim borders all the time, but my grid has been a flavor of 9x and ubode since it went live
[11:35] Ubit Umarov: bullet has ?? well no idea ;)
[11:37] Ubit Umarov: last days looked again to viewers mesh uploads and they seem getting worse :(
[11:37] Ubit Umarov: FS fails a lot, so did alchmiy on the test mesh
[11:38] Ubit Umarov: SL viewer at SL was very fast to submit the mesh model to region for fees calculation
[11:38] Ubit Umarov: FS and alch, just styed there bunring cpu doing nothing visible
[11:39] Ubit Umarov: a older kokua 4.2.1 did upload that model jsut a ity slower than the SL one
[11:40] Ubit Umarov: sadly my machine now makes singu 1.8.6 crash :(
[11:40] Ubit Umarov: hmmm you all fallen asleep ?
[11:40] Kayaker Magic: not yet
[11:40] Andrew.Hellershanks @hg.osgrid.org: :)
[11:40] Aiden Frimon: no
[11:40] Ubit Umarov: :)
[11:41] Aiden Frimon: just listening
[11:41] Andrew.Hellershanks @hg.osgrid.org: Ubit, have you tried 1.8.7?
[11:41] Kayaker Magic: I have discovered something about my mantis 8251, the one that reports list feches slow in mono 5.x
[11:42] Andrew.Hellershanks @hg.osgrid.org: What have you discovered?
[11:42] Kayaker Magic: I re-wrote the test script to remove the loop overhead from the time,
[11:43] Kayaker Magic: and the list fetches are not the problem, apparently loop over head was what got slower in 5.x
[11:43] Ubit Umarov: odd
[11:44] Kayaker Magic: I'll post an updated test script, and I am becoming OK with using Mono 5.x on OpenSim now.
[11:44] Ubit Umarov: ok
[11:44] Kayaker Magic: Previously I have been sticking with an old mono 4.something.
[11:45] Ubit Umarov: well we will need 5.x or better saying 5.12 at some point ( httptests )
[11:46] Kayaker Magic: On my OSGrid regions I am using 5.12 now with no problems on OpenSim 0.9.1dev
[11:47] Kayaker Magic: Although new versions of Mono come out faster than I can upgrade....
[11:47] Ubit Umarov: about 3rdrock, they where the ones that told me about that crossing issue on a 0.8 grid ;)
[11:47] Andrew Hellershanks: I've got a grid running on mono 5.12. I was hoping it would fix the memory leak issue with Robust but Robust is still leaking memory.
[11:48] Ubit Umarov: i've been there etc..
[11:48] Kayaker Magic: Yeah, Discovery Grid still has the memory leak in robust problem also
[11:48] Ubit Umarov: guess solution is either patching 0.8 robust if they do need it for some reason, or better just upgrade grid also
[11:49] Ubit Umarov: uff i get no clues about that leak.. and of course i don't see it :(
[11:50] Bill.Blight @grid.opensim.life:8002: Yeah I don't see the leak either
[11:50] Andrew Hellershanks: I have no idea is causing it.
[11:50] TG.Lucan @hg.viewtwo.net:8600: What about map refresh and flotsam cache settings?
[11:50] Ubit Umarov: mb better on httptests ?
[11:51] Ubit Umarov: think robust does not use flotsam cache
[11:52] Ubit Umarov: and its map code is comp different from regions one
[11:52] Andrew Hellershanks: I don't think it does either. The Flotsam cache settings are only set by a file in config-include and that dir isn't used for Robust, AFAIK.
[11:52] Ubit Umarov: and mostly disk storage
[11:52] TG.Lucan @hg.viewtwo.net:8600: yes, just make suggestions for any possible config characteristics
[11:53] Andrew Hellershanks: I keep wishing for a C# version of valgrind.
[11:53] Ubit Umarov: hmm it may do some bitmap operations.. don't remember now :)
[11:53] Ubit Umarov: yeah i think it does
[11:54] Ubit Umarov: hmm a lot actually lol
[11:54] Ubit Umarov: hmm i should review those..
[11:54] Ubit Umarov: may have issues warp3d ones had
[11:54] Andrew Hellershanks: The grid with the problem is using Warp3D.
[11:54] Ubit Umarov: ( cause those missing or white lines on borders etc )
[11:55] Ubit Umarov: ok but robust does not use warp3d :)
[11:56] Ubit Umarov: i meant the bitmap operations ( image resample/ resize, etc )
[11:57] Ubit Umarov: blend.. merge blabla
[11:58] Ubit Umarov: and there you are sleeping again
[11:58] Kayaker Magic: z.z.z.z.zzzz
[11:58] Sheera.Khan @hypergrid.org:8002: nope, just working on the log ^^
[11:59] Ubit Umarov: 3rdrock will of course use a hybrid, 0.9 with their own changes
[11:59] Aiden Frimon: will pass it along
[12:00] Ubit Umarov: pass along ?
[12:00] Andrew Hellershanks: We are at the top of the hour. If there is nothing else for today (we are starting to go to sleep) its time to call it a day. :)
[12:00] Kayaker Magic: Bill: did you have time to look at the changes you made to the DTS money module for httptests?
[12:01] Ubit Umarov: pass along what ? :)
[12:01] Aiden Frimon: will use hybrid
[12:01] Ubit Umarov: and to where ?
[12:01] Sheera.Khan @hypergrid.org:8002: has someone first hand experience with big crowds of realavatars on a 0.9.1 region already?
[12:01] Ubit Umarov: i meant their code is 0.9x with their own changes
[12:01] Bill.Blight @grid.opensim.life:8002: what do you call be crowds, 40?
[12:01] Ubit Umarov: they where not on pure 0.8 either
[12:02] TG.Lucan @hg.viewtwo.net:8600: WE entertained HG Safari using httptests, not much in the way of content, around 20 of them danced around for 1 hour
[12:02] Bill.Blight @grid.opensim.life:8002: been there done that, many times usually around 20 though
[12:02] Sheera.Khan @hypergrid.org:8002: well, yes 40 would be a real crowd
[12:03] Sheera.Khan @hypergrid.org:8002: could you tell me how you optimized the settings for mono etc?
[12:03] TG.Lucan @hg.viewtwo.net:8600: me?
[12:04] Sheera.Khan @hypergrid.org:8002: You or/and Bill :-)
[12:04] Ubit Umarov: 40 avas are still common to find
[12:04] Bill.Blight @grid.opensim.life:8002: More about optimizing your system and your network than so much optimizing mono
[12:04] Kayaker Magic: All of a sudden everyone is awake and asking questions again!
[12:04] TG.Lucan @hg.viewtwo.net:8600: no specific mono tweeks, as comes from compile
[12:04] TG.Lucan @hg.viewtwo.net:8600: we use appdomain true
[12:04] Ubit Umarov: see opensim.sh
[12:05] TG.Lucan @hg.viewtwo.net:8600: on linux
[12:05] Ubit Umarov: do not use appdomain true
[12:05] Bill.Blight @grid.opensim.life:8002: most linux distros these days are UDP anemic, so checking the UDP network mem and buffers is a good place to start
[12:05] Sheera.Khan @hypergrid.org:8002: would it be possible to make those settings public as i.e. in the wiki? I'll write an entry if I knew the settings...
[12:05] TG.Lucan @hg.viewtwo.net:8600: Ubit we are not seeing problems
[12:06] Bill.Blight @grid.opensim.life:8002: I do use additional mono env varibles other than come with what ubit put in the startup file
[12:06] Ubit Umarov: see the mantis kayker mentioned
[12:06] Ubit Umarov: appdomains do slow down scripts a lot
[12:06] Kayaker Magic: Mantis 8127 and 8251
[12:07] TG.Lucan @hg.viewtwo.net:8600: we also use async_call_method = Thread
[12:07] Sheera.Khan @hypergrid.org:8002: I'll take a dive into those Mantis ...
[12:07] TG.Lucan @hg.viewtwo.net:8600: WE have limited memory and require to recover allocations
[12:07] Ubit Umarov: why not the default?
[12:08] Ubit Umarov: default is a threads pool.. saves a lot of thread creation time
[12:08] TG.Lucan @hg.viewtwo.net:8600: Some threads not being cleaned, as I recall
[12:08] Ubit Umarov: grr i really should delete some options :p
[12:08] Ubit Umarov: ;)
[12:08] TG.Lucan @hg.viewtwo.net:8600: notice.. we are experimental and not production
[12:08] Ubit Umarov: no repos on that
[12:10] Ubit Umarov: think i almost deleted the thread option
[12:10] Ubit Umarov: kept it bc our tests still depend on it
[12:10] TG.Lucan @hg.viewtwo.net:8600: we would put it back :)
[12:11] Ubit Umarov: :p
[12:11] TG.Lucan @hg.viewtwo.net:8600: like c# scripts
[12:11] Ubit Umarov: well if there are leaks on the poolthread we need to work them
[12:12] Ubit Umarov: on smartThreadPool i mean
[12:12] TG.Lucan @hg.viewtwo.net:8600: also, what dotnet used to compile threadpool
[12:12] Sheera.Khan @hypergrid.org:8002: @Bill: I hope I can find some documentation on how to monitor and change the variables controlling the UDP-traffic...
[12:12] Ubit Umarov: it is compiled at same time
[12:12] Ubit Umarov: the source is there on opensim
[12:13] Bill.Blight @grid.opensim.life:8002: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html
[12:13] TG.Lucan @hg.viewtwo.net:8600: thinking other libraries
[12:13] Sheera.Khan @hypergrid.org:8002: TY Bill :-)
[12:13] Ubit Umarov: on httptests i did recompile most to target 4.6
[12:14] TG.Lucan @hg.viewtwo.net:8600: again we made small change to use 4.6.2 so in step with our windows servers
[12:14] Ubit Umarov: im considering that also
[12:14] Ubit Umarov: we just lose Vista also, doing that
[12:15] TG.Lucan @hg.viewtwo.net:8600: current win 10 out of the box is 4.6.2, must install 4.6.1
[12:15] Ubit Umarov: don't think you need to install 4.6.1
[12:16] TG.Lucan @hg.viewtwo.net:8600: not if you come from earlier version
[12:16] Ubit Umarov: older than 4.5 possible
[12:16] TG.Lucan @hg.viewtwo.net:8600: earlier win 10
[12:16] Ubit Umarov: but yeap can't tell , my win 10 is a upgrade to win7
[12:16] TG.Lucan @hg.viewtwo.net:8600: my vis studio 2017 does not show 4.6.1
[12:17] Ubit Umarov: have a collection of installed frameworks :)
[12:17] Ubit Umarov: let me check mine
[12:18] Ubit Umarov: 2.0 3.0 3.5 4.0 4.5 4.5.1 4.5.2
[12:18] Ubit Umarov: ..
[12:18] TG.Lucan @hg.viewtwo.net:8600: net 2 3 3.5 4 4.5 etc but no 4.6.1
[12:19] Ubit Umarov: 4.6 4.6.1 4.6.2 4.7 4.7.1
[12:19] Ubit Umarov: plus odd ones :)
[12:19] TG.Lucan @hg.viewtwo.net:8600: yes, then 4.7 and 4.7.1
[12:20] Ubit Umarov: well but ive not plans for 4.6.1
[12:20] Ubit Umarov: not httptests is on 4.6 and i consider moving up to 4.6.2
[12:20] Ubit Umarov: now..
[12:20] TG.Lucan @hg.viewtwo.net:8600: kk
[12:20] Ubit Umarov: no use for ..1
[12:21] Ubit Umarov: just last version suported on vista is 4.6
[12:21] TG.Lucan @hg.viewtwo.net:8600: as now 4.6.x ever think to use dotnet compression, and remove zlib
[12:22] Ubit Umarov: thats why i did pick it, so to only lose XP on windows world