Chat log from the meeting on 2018-05-01

From OpenSimulator

Jump to: navigation, search

[11:06] LaNani Sundara: right of the bat... just something i noticed
[11:06] LaNani Sundara: i noticed 2 using statements in FSAssetService that may lead to a memory leak as a FileStream and a GZipStream are made in one go. I can imagine Mono not dealing with cleaning up the FileStream cleanly. and would make it 2 encapsulating using statements.
[11:07] Ubit Umarov: on some of those things the top level one should close all
[11:07] LaNani Sundara: should
[11:07] Ubit Umarov: in fact code analyses tools complain about double using
[11:08] LaNani Sundara: using (GZipStream gz = new GZipStream(new FileStream(diskFile + ".gz", FileMode.Create), CompressionMode.Compress))
[11:09] Ubit Umarov: yeap that does assume gzip will dispose all
[11:09] LaNani Sundara: hmm just wondering if that is always true, especially in mono
[11:09] Ubit Umarov: should be
[11:10] LaNani Sundara: anyway :) thats all from me :D
[11:10] Ubit Umarov: not looking to code now
[11:10] Ubit Umarov: in many places we do have double using
[11:10] Ubit Umarov: just in case :)
[11:10] LaNani Sundara: yes i seen it
[11:11] Ubit Umarov: but get warnings about illegal close of a already closed stream
[11:11] Ubit Umarov: ( if some code analysis warnings are on )
[11:12] Ubit Umarov: but guess that would be a very visible mem leak
[11:12] Ubit Umarov: and for now reports are about robust, not regions
[11:12] Ubit Umarov: ie large memory use reports
[11:13] LaNani Sundara: i thought some people here complained about having to restart their robusts
[11:13] LaNani Sundara: a few meeting ago
[11:13] Kayaker Magic sits up and pays attention to memory leak talk.
[11:13] Ubit Umarov: yes some do
[11:13] Ubit Umarov: but my tiny grid robust on win10 is happy
[11:14] Ubit Umarov: so hard for me to repo
[11:14] Dragon Moorlord: win10 seems to have no issues either, on robust+regions..
[11:14] Kayaker Magic: Only a few people report this problem, but one of them is Andrew.
[11:14] LaNani Sundara: well if they use FSAssetService with mono these using statements may be leaving unreclaimed FiteStream objects
[11:14] Ubit Umarov: well its is also httptests and win10
[11:15] Ubit Umarov: hmm thats on the robust side ?
[11:15] LaNani Sundara: yes
[11:15] Ubit Umarov: well i also do not run fsassets :)
[11:15] Bill.Blight I'm curious when they talk of the leak are they talking real mem or virtual?
[11:15] LaNani Sundara: FlotsamCache does is cleanly with double using statements
[11:16] Bill.Blight I use FSassets, my robust has been up a week and it is using 2.4 gig of real ram and that is about where it sits
[11:16] LaNani Sundara: *nods
[11:16] Ubit Umarov: well gzip gets a using and a close lol
[11:17] LaNani Sundara: but the FileStream does not
[11:17] Kayaker Magic: Andrew says his leaks so bad he restarts Robust every 4 hours.
[11:17] Ubit Umarov: as i said is suposed to be closed by the gzip
[11:17] Bill.Blight I am wondering if on windows, the mem leak could be a side effect of windows file locking and or Anti-virus file system monitoring ..
[11:17] Dragon Moorlord: wow
[11:17] LaNani Sundara: its not about closing the filestream but about disposing it
[11:18] Ubit Umarov: same thing
[11:18] Ubit Umarov: those play together
[11:18] LaNani Sundara: not according to what i have been reading.. they actually say you should not close a stream but let the using deal with it
[11:19] Ubit Umarov: that is to reduce mistakes
[11:19] Ubit Umarov: using makes sure it does happen
[11:19] Ubit Umarov: close requieres more care
[11:19] LaNani Sundara: anyway :) sorry for hogging you time guys i am sure you have more important topics
[11:19] LaNani Sundara: :)
[11:21] Ubit Umarov: "When the disposing parameter is true, this method releases all resources held by any managed objects that this DeflateStream references. This method invokes the Dispose method of each referenced object."
[11:21] Ubit Umarov: ( usingf ticks it as true, so does close()
[11:23] Ubit Umarov: if you see stream.close help you see they say it calls Dispose(true)
[11:24] LaNani Sundara: thats the gzip yes
[11:24] LaNani Sundara: and i am sure the filestream gets closed yes
[11:24] LaNani Sundara: just not disposed
[11:24] Ubit Umarov: just seems a close was left that inside the using.. but no harm i guess
[11:24] TG.Lucan Interesting points, but does not answer why some see the issue while others do not
[11:24] Ubit Umarov: i just said close calls dispose
[11:24] LaNani Sundara: could be a mono issue
[11:25] Ubit Umarov: and so does using
[11:25] Ubit Umarov: even finalizer :p
[11:25] TG.Lucan I use mono, and have not seen the issue
[11:25] LaNani Sundara: do you use the FSAssetService?
[11:25] TG.Lucan I do not use FSASSet though
[11:25] Bill.Blight the mem leak could be anything really, mono, kernel, glibc , and countless other things are at play there, at least on linux
[11:25] LaNani Sundara: true Bill
[11:26] Bill.Blight It may not be a specific "Opensimulator" issue but an interaction with other libraries
[11:26] TG.Lucan How is the memory usage being monitored, what tools etc?
[11:26] LaNani Sundara: i just was reading code as i do sometimes and noticed that :) which is why i asked
[11:26] Ubit Umarov: guess just top tg
[11:26] TG.Lucan that can be misleading
[11:27] Bill.Blight at least on Linux people freak out when they see a lot of Virtual mem used, freaking ignore that ..
[11:27] Bill.Blight Unused mem is wasted mem
[11:27] Ubit Umarov: seems clearing some mono ? caches does help
[11:27] Dragon Moorlord: anything major or minor about windows10 or winserver 2016 on robust+-regions
[11:27] Kayaker Magic: System memory use grows by about 8GB per day on the effected systems.
[11:27] TG.Lucan I have mentioned before, linux caches files in ram
[11:28] Bill.Blight
[11:28] Ubit Umarov: well but linux file cache is diferent
[11:28] TG.Lucan the linux command #free (not sure if installed by default) will display the cache
[11:28] Ubit Umarov: it does it on "free memory"
[11:28] LaNani Sundara: but do the effected systems actually grind to a halt and run out of mem?
[11:28] Ubit Umarov: at least windows does
[11:29] Ubit Umarov: if processes ask for memory, the caches are deleted
[11:29] Kayaker Magic: Yeah, eventually the effected systems crash.
[11:29] LaNani Sundara: ah
[11:29] Dragon Moorlord: ok
[11:29] Bill.Blight if linux mem caches are not getting cleared that may not be a specific opensimulator issue, but opensimulator might be the trigger
[11:29] LaNani Sundara: then thats no good
[11:29] TG.Lucan ++bill
[11:30] Dragon Moorlord: is robust better to run on win or linux
[11:30] Bill.Blight That is a loaded question
[11:30] Bill.Blight depends on who you ask
[11:30] TG.Lucan Technically no difference in my experience
[11:30] Ubit Umarov: my win10 now has 6GB of things cached
[11:31] Ubit Umarov: 7.3G actually
[11:31] Dragon Moorlord: 6.1
[11:31] Dragon Moorlord: here
[11:31] Ubit Umarov: as part of the 8.1G of free ram
[11:31] Bill.Blight personally the only major difference I see, is linux is a little more liberal on it's networking, and does not have the "overhead" of the windows GUI
[11:31] LaNani Sundara: i do like to run on Win Server 2012 R2 ;)
[11:32] Dragon Moorlord: i have 2016 R2
[11:32] Ubit Umarov: so i would not count that file cache, if bad, it will bad for all things on linux
[11:32] LaNani Sundara: 2016 seems... fat to me
[11:32] TG.Lucan You can run windows in console mode
[11:32] Bill.Blight Personal opinion, windows is a resource hog, you waste a good 25% or more of your resources just for the machine to idle ...
[11:32] Ubit Umarov: but thing mono does have specific caches
[11:32] Bill.Blight that is true TG
[11:32] LaNani Sundara: yes TG )
[11:32] Ubit Umarov: melany did shown me comands do clear them, but i lost them :(
[11:32] Bill.Blight sync; echo 1 > /proc/sys/vm/drop_caches
[11:33] Dragon Moorlord: reason why i do all my work on it, hosting, robust, designing.. '
[11:33] Kayaker Magic: On one of the systems, they tried separating Robust into separate instances, like fsassets in one, everything else in another. But there are a bunch of dependancies that prevent this from working.
[11:33] Dragon Moorlord: lol
[11:33] Kayaker Magic: Does anybody know what services in Robust CAN be split off into separate instances?
[11:33] Ubit Umarov: osgrid does run splited instances.. but yes there are dependencies
[11:33] Bill.Blight Primarily TWO, all assets and all user services
[11:34] Bill.Blight at least how I see it
[11:34] Ubit Umarov: never played with that so far.. you may ask dan etc
[11:34] Bill.Blight but does not really do any good unless you have multiple servers, as one will not work without the other anyway, and running both on the same machine does not save you anything
[11:34] Ubit Umarov: osg had several assets issues exactly bc using a few asset servers.. they did got desync
[11:35] LaNani Sundara: a week ago i think
[11:35] Dragon Moorlord: i noticed that last couple of weeks or so
[11:35] Kayaker Magic: The plan was to watch the different instances and see which one leaked and grew, to narrow down what was leaking.
[11:35] Bill.Blight running a cluster of assets servers is a daunting task
[11:35] Ubit Umarov: dan told me once he suspected presence server
[11:36] LaNani Sundara: moving them, which has not been done yet i think, will not be easy
[11:36] Bill.Blight I could see the presence server being a little "chatty" in the case of an open grid like osgrid
[11:37] Ubit Umarov: well you can ask dan for ideas how to split robust by different machines since osgrid does that
[11:38] Bill.Blight This command sync; echo 1 > /proc/sys/vm/drop_caches will clear your cached mem, but you don't want to run it very often , if at all in a production environment, but in "emergency" memory situations it might help ..
[11:38] Ubit Umarov: HG visits may make caching grow
[11:39] LaNani Sundara: thank you Bill :)
[11:39] Kayaker Magic: Andrews problem system is closed, no HG there and it still leaks.
[11:39] Ubit Umarov: yes think it was what mel told me
[11:39] Ubit Umarov: but echo 3
[11:40] TG.Lucan free -m && sync && echo 3 > /proc/sys/vm/drop_caches && free -m
[11:40] Bill.Blight I wouldn't use the &&
[11:40] Bill.Blight I'd use ;
[11:40] TG.Lucan old timer
[11:40] Bill.Blight that will make it wait for the previous command to finish first
[11:41] Bill.Blight with the && they are going to run mostly all at once
[11:41] TG.Lucan sure
[11:41] LaNani Sundara: now now play nice you echo 2 ;)
[11:41] Dragon Moorlord: lol
[11:41] Ubit Umarov: but well there maybe a true mem leak on code, still not found :(
[11:42] Ubit Umarov: regions did had a few
[11:42] TG.Lucan without aditional info I cannot see a way forward
[11:42] Ubit Umarov: unless we just step on it by chance
[11:43] Ubit Umarov: fixing some other thing or so
[11:43] Ubit Umarov: to know what service does it would help
[11:43] TG.Lucan I am running app domain thingie on linux, and the thread counts returns to normal when the visitors leave
[11:44] Ubit Umarov: script app domain?
[11:44] LaNani Sundara: app domains on linux??
[11:44] TG.Lucan yes
[11:44] Bill.Blight since "everyone" is not seeing it, makes it even harder to find
[11:44] Ubit Umarov: you should not..
[11:44] Kayaker Magic: splitting it into separate instances was an attempt to figure out which service was the problem...
[11:44] Ubit Umarov: that is a assumed memory leak
[11:44] Bill.Blight app domains baddddd
[11:44] LaNani Sundara: i thought that was meant for Win
[11:44] Ubit Umarov: but they are very bad for scripts performance
[11:45] Ubit Umarov: no they where mean to free deleted scripts memory only
[11:45] Bill.Blight bad is an understatement
[11:45] LaNani Sundara: yes Kayaker, as i was saying at that meeting
[11:45] LaNani Sundara: narrowing down which service is causing it
[11:45] LaNani Sundara: good luck
[11:45] Ubit Umarov: me and kayaker have a talk about that on mantis
[11:45] Bill.Blight Maybe we should have the memory leak peeps check for appdomains being on
[11:45] LaNani Sundara: :D
[11:46] Bill.Blight of course only for regions
[11:46] Bill.Blight should not matter to robust
[11:46] Ubit Umarov: Xengine scripts delete is a assumed memory leak
[11:46] LaNani Sundara: if used at all i thought it was meant for Win
[11:46] Sheera.Khan @Bill: the && in that command line isa conditional AND - if the first command succeeds the second is run
[11:46] Bill.Blight Loves me some Yengine, need people to get on board with it
[11:46] Ubit Umarov: Yengine should delete them.. but can't tell i did test it :)
[11:46] Bill.Blight ahh yes I was thinking single &
[11:47] Bill.Blight duh moment
[11:47] TG.Lucan ++Sheera
[11:47] Sheera.Khan that would be a background process
[11:47] LaNani Sundara: what would an average user notice when a sim was running Yengine?
[11:47] Bill.Blight scripts seem faster, more SL compliant,
[11:48] LaNani Sundara: assuming all bugs are fixed :D
[11:48] Kayaker Magic: I'm having trouble getting httptest to run. It won't run with the OpenSim.ini file from the dev version, I'm getting errors starting over trying to write a new OpenSim.ini
[11:48] LaNani Sundara: sounds interesting
[11:48] Ubit Umarov: Yengine father was actually avination script engine :)
[11:48] LaNani Sundara: any chance of porting Yengine into master?
[11:49] Bill.Blight Not at the moment
[11:49] LaNani Sundara: ok
[11:49] Bill.Blight httptests should be master ,
[11:49] Ubit Umarov: httptests is our dev branch now
[11:49] Bill.Blight waits for Ubit
[11:49] LaNani Sundara: hmm k
[11:49] Ubit Umarov: not on master bc issues on our machine
[11:49] Bill.Blight Entirely a dev box issue not a code issue keeping httptests from being master
[11:50] Ubit Umarov: current master is just 0.9 bugs fixes
[11:50] TG.Lucan hg safari will run on httptests tomorrow
[11:50] Ubit Umarov: ( to be released soon i hope )
[11:50] LaNani Sundara: some things i do not like in it ;) but hey :) i can stubbornly covery those things to my coce ;p
[11:50] Bill.Blight my whole grid runs a flavor of httptests, and they have been there a few times, no issues , so you should be good
[11:50] LaNani Sundara: mmm flavor...
[11:51] LaNani Sundara: *drool
[11:51] Bill.Blight we likes flavor
[11:51] TG.Lucan I don't expect any noticeable difference
[11:51] Ubit Umarov: ( issue is that our box is now old, running mono 2.10 etc )
[11:51] Kayaker Magic: Bill, can you give me an example OpenSim.ini that works with httptests?
[11:51] Ubit Umarov: ( httptest needs mono5x )
[11:51] Bill.Blight yes
[11:51] Bill.Blight or it will die
[11:51] LaNani Sundara: .Net 4.6?
[11:51] Bill.Blight I'm not a good one to ask, my INI's are all broken up into pieces for easier editing
[11:51] Ubit Umarov: hmm? httptests has its opensim.ini.example
[11:52] Bill.Blight yes that one seems to work as long as all your includes are also good
[11:52] Bill.Blight httptests is .Net 4.6
[11:52] LaNani Sundara: *nods
[11:52] Kayaker Magic: Yeah, and I can't seem to edit it into shape to work on my system.
[11:52] Ubit Umarov: ini files review is a condition on all opensim version updates :)
[11:53] Bill.Blight what error are you getting on startup kayaker?
[11:53] Ubit Umarov: httptests does have its set of ini file examples
[11:53] Kayaker Magic: Exception when contacting grid server at http:// Error: NameResolutionFailure
[11:53] Bill.Blight umm
[11:53] Ubit Umarov: ( but not that many diferences from master i think )
[11:53] Bill.Blight you have an extra http on there
[11:53] LaNani Sundara: o
[11:54] Bill.Blight your ini is not setup in the [Const] section correctly
[11:54] Bill.Blight adding an extra http://
[11:54] LaNani Sundara: well spotted Bill
[11:54] Kayaker Magic: Dang, I didn't see see the the duplication
[11:54] Ubit Umarov: :)
[11:55] TG.Lucan don't include the HTTP:?? with BaseHostname = ""
[11:55] Ubit Umarov: we do need to rename/change those consts
[11:56] LaNani Sundara: http:// is not part of a name but a protocol indicator
[11:56] TG.Lucan and there should be a local address for when the region and robust are on the same subnet
[11:57] Ubit Umarov: well as you know [const] things are just helpers
[11:57] Dragon Moorlord: yes true .
[11:57] Ubit Umarov: you can set things "inplace"
[11:57] TG.Lucan sure
[11:57] Ubit Umarov: guess actually needed on a few cases
[11:58] LaNani Sundara: what about https, and ... ipv6?
[11:59] Ubit Umarov: httptests does have a bit better https
[11:59] LaNani Sundara: nice
[11:59] Bill.Blight It still complains a lot
[11:59] Bill.Blight but it sort of works ... LOL
[11:59] Ubit Umarov: no ipv6.. no viewers for that
[11:59] Bill.Blight Ubit, what were those settings that needed to be added to the INI for the SSL, something about not verifying certs
[12:00] Ubit Umarov: well money modules and lsl httpsecureurl will mess it up :(
[12:00] TG.Lucan as all our regions are on the same subnet we replace the ${Const|BaseURL}:${Const|PrivatePort} with the actual IP address + :8003 of the robust box
[12:00] Ubit Umarov: that is the issue with ssl
[12:00] LaNani Sundara: the certs
[12:00] Ubit Umarov: we need to allow self signed certs
[12:01] Ubit Umarov: those options are for that
[12:01] Kayaker Magic: Thanks everyone! My httptest system is up and limping! I can test YEngine now!
[12:01] Ubit Umarov: but are global settings.. ( so mess up money modules etc (
[12:01] LaNani Sundara: yay
[12:01] LaNani Sundara: .•*♫ ♬ APPlAUSE! ♬♫ *•.
[12:01] Ubit Umarov: and i made no code for robust
[12:02] Ubit Umarov: only did run a region with full SSL with a self signed cert
[12:02] Bill.Blight with letsencrypt no real need for self signed certs anyway
[12:02] Ubit Umarov: that does work now
[12:02] Ubit Umarov: you need to own a domain to use those certs
[12:03] Bill.Blight makes no comment
[12:03] Bill.Blight LOL
[12:03] LaNani Sundara: :D
[12:03] Dragon Moorlord: noip is good for that
[12:03] Ubit Umarov: opensim is also for home based standalones :p
[12:03] Ubit Umarov: or tiny grids
[12:03] LaNani Sundara:
[12:03] Bill.Blight domains are cheap, and dynamic DNS services are cheap and free in some cases ... Just need to put a bit of effort in
[12:03] LaNani Sundara: for free certs
[12:04] Bill.Blight If you have gone to the trouble of learning to properly setup Opensimulator, domains and dns should be a breeze
[12:04] Ubit Umarov: well ill get back to ssl one day
[12:04] LaNani Sundara: :)
[12:04] Ubit Umarov: .net4.6 also makes it a bit easier
[12:04] Ubit Umarov: but grid side.. a pain :p
[12:05] Ubit Umarov: err and no plans for ipv6
[12:05] LaNani Sundara: i belive Hyacinth has implemented ssl on grid side
[12:05] Bill.Blight I did get robust to work with https for a bit, but it was not happy, worked, but complained a lot
[12:05] LaNani Sundara: recently
[12:05] Ubit Umarov: my internal router does not have it :p
[12:06] Ubit Umarov: some just use a proxy
[12:06] Bill.Blight ssl proxy server may be the best bet for grid side
[12:06] LaNani Sundara: hmmm
[12:07] Ubit Umarov: ( we don't have andrew to close the meeting at the hour :) )
[12:07] Bill.Blight My new setup that I am going to work on soon, uses SSL proxy and GRE tunnels from the region servers to talk to the private ports on the robust server ..
[12:07] LaNani Sundara: :) well the elders can close it ;p
[12:07] Ubit Umarov: ( bahh desertor :p )
[12:08] Ubit Umarov: today i pushed a bit of code
[12:09] Bill.Blight Already snagged the new stuff Ubit
[12:09] Ubit Umarov: region export-map should work better ( httptests )
[12:09] TG.Lucan I see Ubit, but grid in use, later will update
[12:09] Bill.Blight should only be region side TG
[12:09] Bill.Blight so you could roll a restart
[12:09] Ubit Umarov: yes its only region side
[12:10] Ubit Umarov: follows original.. makes a jpg with the region and regions around
[12:10] TG.Lucan true, but I like to keep the robust in step
[12:10] Ubit Umarov: i just reduced the number of regions to the ones on Maxblablablaviewrange
[12:10] Ubit Umarov: lol
[12:11] TG.Lucan really :)
[12:11] Ubit Umarov: MaxRegionViewDistance
[12:11] Ubit Umarov: so the actuall size in meters is now variabel
[12:12] Ubit Umarov: but does work with var regions.. previus code did not
[12:12] Ubit Umarov: i also reduce the jpeg to 2048 pixeis
[12:12] Ubit Umarov: a bit 2 much
[12:12] Ubit Umarov: hmm
[12:13] Ubit Umarov: in place of scaling i could had changed that also
[12:13] Ubit Umarov: but its code evolution :)
[12:13] Ubit Umarov: had you tested it bill ?
[12:13] Bill.Blight Not yet
[12:14] Ubit Umarov: bahh
[12:14] Ubit Umarov: bad bill.Blight
[12:14] Bill.Blight will do that later tonight when I get back from seeing Avengers
[12:15] Ubit Umarov: btw region map textures assets on 0.9 are all 256x256 pixels
[12:15] Dragon Moorlord: yes
[12:16] Ubit Umarov: on 0.8 they were full region size
[12:16] Ubit Umarov: 1 pixel per meter
[12:16] Ubit Umarov: that was making dam large textures
[12:16] Ubit Umarov: and at least old gpus where not happy with some sizes
[12:17] Bill.Blight so my 9 16x16 regions that are all next to each other are not killing it with textures ... GOT IT ... LMAO
[12:17] Dragon Moorlord: jeeze, i just have 1 2x2.. lol
[12:17] Dragon Moorlord: lmao
[12:17] Ubit Umarov: with 0.8 code all my net went down connecting to osg with a 1km side region :p
[12:18] Ubit Umarov: uploading with 700kbps yeack
[12:18] Bill.Blight I have 2 single sims, everything else is at least a 3x3 I think
[12:18] Ubit Umarov: but that is something open to review
[12:18] Ubit Umarov: 256 mb a bit low for some uses
[12:18] Dragon Moorlord: wow bill
[12:19] Bill.Blight I run a lot of boats, planes and buggies
[12:19] Bill.Blight so need the space
[12:19] Ubit Umarov: but 0.8 allowing for 8192x8192 textures was bad karma :p
[12:19] Bill.Blight My boats have been pretty safe for a while, Ubit has not shown up to wreck any ..
[12:20] Dragon Moorlord: hmmmm.. thanks bill you just gave me an expensive idea.. upgrade to more regions for a cruise sim
[12:20] Dragon Moorlord: hahaha
[12:20] Ubit Umarov: yeap.. me not beeing there is the only reason why they still work :P
[12:20] Bill.Blight HAHA
[12:20] Bill.Blight yep
[12:20] Bill.Blight I keep 8x8's around just for my boats
[12:21] Ubit Umarov: well that "low" resolution is just a warning..
[12:21] Dragon Moorlord: wow.. will you give me a hg lm to your land, id like to see it
[12:21] Ubit Umarov: on export-map you may notice it since its zoomed out
[12:22] Ubit Umarov: mb i should also make the export map texture have variable size in pixels
[12:22] Ubit Umarov: do not use an avatar you do like :p
[12:22] Bill.Blight HAHA
[12:23] Bill.Blight As long as Ubit is not driving a boat you should be safe
[12:24] Ubit Umarov: i also added a little text to that map export
[12:24] Dragon Moorlord: i gues my other question would be, how is your regions with the ships crossing.. much lag or anything special to setup server wise
[12:24] Bill.Blight maybe that text should be OFF by default ...

Personal tools
About This Wiki