Chat log from the meeting on 2016-06-28

[11:05] George Equus: So quiet here... [11:07] Lucy Afarensis: Is there a meeting today ? [11:08] George Equus: Should be I suppose [11:09] Robert Adams: there is a meeting and here we are [11:09] Lucy Afarensis: Here we are then [11:10] Gavin.Hird @grid.xmir.org:8002: yup yup [11:10] George Equus: indeed we are [11:10] Gavin.Hird @grid.xmir.org:8002: anyone having fun yet this week? [11:11] Lucy Afarensis: It has been quiet [11:11] Lucy Afarensis: for me [11:11] Billy.Bradshaw @hg.viewtwo.net:8600: I was unable to leave a sim running 0.8.1, either to a 0.8.2 or .9.0 [11:12] Billy.Bradshaw @hg.viewtwo.net:8600: is that correct? [11:12] Billy.Bradshaw @hg.viewtwo.net:8600: I did not know another 8.1 to try that option [11:12] Lucy Afarensis: They crashed ? [11:13] Billy.Bradshaw @hg.viewtwo.net:8600: No, all ok just said something about blocked, should have made a note [11:13] Billy.Bradshaw @hg.viewtwo.net:8600: I had to logout to recover [11:16] Gavin.Hird @grid.xmir.org:8002: Inspired by the European Football Championship i set out ot make a mesh football. Once done I found a script that made it kickable. The problem was a tiny kick sent it across the sim, and trying to find it back I discovered Areas Search was broken, so I had to fix that in the viewer. – One thing led to the other [11:19] Billy.Bradshaw @hg.viewtwo.net:8600: @Robert, can the collision frame around an avatar be any rectangle or has it to be a rectangle with perpendicular angles? [11:21] Robert Adams: the current collision shape of an avatar in BulletSim is a rectangle that's as tall as the avatar [11:22] Robert Adams: I've played with more complex mesh shapes but it's not turned on [11:22] Billy.Bradshaw @hg.viewtwo.net:8600: ok, thanks [11:24] Sheera Khan: I really enjoy the activity in the mantis and git as seen last weeks :-) [11:25] Robert Adams: it is being worked on :) [11:25] Sheera Khan: finally there is some progress again :-) [11:26] Lucy Afarensis: I have been getting some strange errors in my sim consoles lately [11:26] Robert Adams: like what, Lucy? [11:27] Lucy Afarensis: May I past them in Chat ? [11:27] Lucy Afarensis: 2016-06-24 10:25:53,512 WARN (1) - OpenSim.Region.Framework.Scenes.SceneObjectGroup [SCENE OBJECT GROUP]: SavedScriptState element had no UUID in object Ice Fae Forest Foot - Left Lower Leg* 2016-06-24 10:25:54,411 WARN (1) - OpenSim.Region.Framework.Scenes.SceneObjectGroup [SCENE OBJECT GROUP]: SavedScriptState element had no UUID in object Ice Fae Forest Foot - Right Lower Leg* 2016-06-24 10:25:56,993 WARN (1) - OpenSim.Region.Framework.Scenes.SceneObjectGroup [SCENE OBJECT GROUP]: SavedScriptState element had no UUID in object Maleficent's brown wings [11:27] Gavin.Hird @grid.xmir.org:8002: the http server update caused some grief, but there was an update today, and so far it seems more well-behaved [11:27] Lucy Afarensis: None of this is on my sims [11:28] Lucy Afarensis: I dozens of lines of this stuff [11:28] Gavin.Hird @grid.xmir.org:8002: I see some of these too for one particular object that is not even rezzed anywhere [11:28] George Equus: Ice Fae Forest Foot - Left Lower Leg seem very familiar, had that in my console for ages but now it is gone [11:28] George Equus: Forgot what caused it,,, [11:29] Sheera Khan: could it be located on a sim next to yours? [11:29] Lucy Afarensis: No I have looked [11:29] Gavin.Hird @grid.xmir.org:8002: for my part it is not rezzed anywhere at all [11:29] Gavin.Hird @grid.xmir.org:8002: it has been at some time [11:30] Gavin.Hird @grid.xmir.org:8002: but it exist in the asset server as xml [11:31] George Equus: Some form of lingering DB corrupted items maybe? [11:31] Robert Adams: Lucy, you might try cleaning the script cache and letting things recompile... the directory is usually bin/ScriptEngines/ [11:31] Gavin.Hird @grid.xmir.org:8002: probalby some in-world object pointing where it should not be pointing [11:31] Lucy Afarensis: I am seeing the same errors in a sim I have on Franco grid [11:32] Lucy Afarensis: I try that Robert Thanks [11:33] Gavin.Hird @grid.xmir.org:8002: Is the script cache scanned at regular intervals as the message is pretty periodic [11:33] Gavin.Hird @grid.xmir.org:8002: probably once every hour [11:33] Robert Adams: there have also been some complaints about XML serialization problems... but there should be other errors if that was the problem [11:34] Lucy Afarensis: I always see it when rebooting [11:36] Gavin.Hird @grid.xmir.org:8002: Microsoft announced availablility of .NET 1.0 for Linux and macOS yesterday. Does that perhaps mean we can do away with mono? [11:36] Gavin.Hird @grid.xmir.org:8002: not .NET 1.0 but the 1.0 of the open version [11:36] Billy.Bradshaw @hg.viewtwo.net:8600: I am testing core 1.0 on redhat at present, our intention is to replace mono [11:37] Simulator Version v0.5 ruft: OpenSim 0.9.0.0 Dev       bc4e5d8: 2016-06-26 14:40:26 +0100 (Unix/Mono) [11:37] Robert Adams: need to decide if we're supporting both or what [11:37] Gavin.Hird @grid.xmir.org:8002: what is the experience so far? [11:38] Billy.Bradshaw @hg.viewtwo.net:8600: some complications on configurations but seems to run the basic standalone [11:38] Gavin.Hird @grid.xmir.org:8002: ok [11:38] Billy.Bradshaw @hg.viewtwo.net:8600: I am travelling at present so progress is slow [11:38] Snakedance Moonwing: holy batfarts is nebadon :) [11:38] George Equus: Long time no see [11:39] Lucy Afarensis: ☻ [11:39] Nebadon Izumi: guess my wave gesture isnt on lol [11:39] Robert Adams: time to dance!! [11:39] Sheera.Khan @hypergrid.org:8002: Welcome back on board :-) [11:40] Billy.Bradshaw @hg.viewtwo.net:8600: music Maestro [11:40] Nebadon Izumi: thanks, try to be here when I have some time, summer is not so busy now, everyone taking vacations :) [11:40] Ubit Umarov: ( bullet :p ) [11:40] Snakedance Moonwing: i am not a dev btw :) [11:41] Snakedance Moonwing: just a curious resident :) [11:41] Nebadon Izumi: This meeting is for everyone :) [11:41] Snakedance Moonwing: :) [11:41] Snakedance Moonwing: is nice to see you again neb :) [11:42] Nebadon Izumi: you too :) [11:42] Snakedance Moonwing: :P [11:42] Nebadon Izumi: I have been kind of lurking around for a bit [11:42] Nebadon Izumi: im still using opensim on a daily basis for my job [11:42] George Equus: Upsala? [11:42] Nebadon Izumi: finally got my home servers sorted out and my osgrid regions back up [11:42] Nebadon Izumi: Ya, a lot of our projects are in Sweden [11:43] Nebadon Izumi: a few in the US now too [11:43] George Equus: :) [11:43] Nebadon Izumi: i traveled to University of Utah last month to train some students [11:43] Nebadon Izumi: how to build in OpenSim [11:43] Snakedance Moonwing: spiffy :) [11:43] George Equus: Like the model of Upsala Domkyrka! [11:43] Billy.Bradshaw @hg.viewtwo.net:8600: info: there is a DNS issue with later mono developments, had a patch applied recently that reverts some of the problem code but not all. Latest 4.4.0 has the patch [11:43] Nebadon Izumi: nice [11:43] Nebadon Izumi: i just upgraded to 4.4.1 [11:44] Nebadon Izumi: maybe i should upgrade the plaza boxes [11:44] Billy.Bradshaw @hg.viewtwo.net:8600: patch is in 4.4.1 [11:44] Nebadon Izumi: ok that is good to know [11:44] Nebadon Izumi: i will uprade plaza08 and plaza09 later [11:44] Gavin.Hird @grid.xmir.org:8002: I get 3x CPU use on 4.4.1 over 4.2.3.4 [11:44] Nebadon Izumi: dang [11:45] Nebadon Izumi: ive not seen anything like that [11:45] Billy.Bradshaw @hg.viewtwo.net:8600: the issue can appear under specific circumstance, yet to be resolved [11:45] Nebadon Izumi: let me see what version we are running here now [11:45] Gavin.Hird @grid.xmir.org:8002: the release notes says that certain kinds of locking is 3 times more expensive than earlier versions [11:45] Gavin.Hird @grid.xmir.org:8002: perhaps that has something to do with it [11:46] Billy.Bradshaw @hg.viewtwo.net:8600: The issue is with reverse DNS, not all ISP play ball [11:46] Nebadon Izumi: ah that's right we reverted back we have 3.12.1 running here on plaza08 [11:46] Billy.Bradshaw @hg.viewtwo.net:8600: as they say, your mileage will vary [11:46] Nebadon Izumi: 3.2.3 on plaza09 [11:46] Nebadon Izumi: yea [11:46] Nebadon Izumi: 4.4.1 has been running quite well for me locally [11:47] Nebadon Izumi: i havent done a ton of testing though, but its been solid [11:47] Nebadon Izumi: im also running a fairly new kernel too though [11:47] Nebadon Izumi: 4.1.12 kernel [11:47] Nebadon Izumi: opensuse leap 42.1 [11:47] Lucy Afarensis: How can you determine what version of mono you have ? [11:47] Nebadon Izumi: type "mono -V" [11:47] Gavin.Hird @grid.xmir.org:8002: mono --version [11:48] Lucy Afarensis: cool [11:48] Snakedance Moonwing: sits contently listening to the smart people :) [11:49] Billy.Bradshaw @hg.viewtwo.net:8600: benefits of later mono are really apparent with sgen improvements [11:49] Nebadon Izumi: yea, I have seen slightly higher CPU at my 2 physics var regions [11:49] Nebadon Izumi: Sisyphus and Newton [11:49] Nebadon Izumi: sit at about 50% of a single core sitting idle no visitors [11:49] Nebadon Izumi: they are neighbors here on OSgrid [11:50] Nebadon Izumi: someone mentioned that the problem may have something to do with 2 vars being neighbors [11:50] Nebadon Izumi: i havent really tested that fully though [11:50] Billy.Bradshaw @hg.viewtwo.net:8600: How many sripts prims for that figure ned? [11:50] Nebadon Izumi: let me look [11:50] Robert Adams: I'm looking into CPU and BulletSim... it does use more than it should on an idle sim [11:50] Nebadon Izumi: 7945 prims and 753 scripts [11:50] Billy.Bradshaw @hg.viewtwo.net:8600: of course other factors come into play like cpu speed etc [11:50] Nebadon Izumi: Sisyphus runs Bulletsim [11:50] Robert Adams: there do seem to be a lot of ways to use up CPU even on a pretty quiet sim [11:50] Nebadon Izumi: and Newton runs Ubode [11:51] Nebadon Izumi: Ubode sits at like 40% of a single core [11:51] Nebadon Izumi: and Bullet around 50% [11:51] Nebadon Izumi: they flucuate a bit though [11:51] Nebadon Izumi: they both probably have active physical prims [11:51] Nebadon Izumi: like some chains [11:51] Nebadon Izumi: and few spheres rolling around probably [11:51] Nebadon Izumi: but nothing major [11:51] Robert Adams: chains are constant work :)) [11:52] Nebadon Izumi: yea however I dont recall it being so high in the past [11:52] Nebadon Izumi: with the same load [11:52] Nebadon Izumi: I havent changed those regions much in a long while [11:52] Robert Adams: I think there are some other things using up CPU that shouldn't be.... I'm adding some tools to find out where time is being spent [11:52] Nebadon Izumi: cool [11:52] Nebadon Izumi: look forward to testing [11:52] Robert Adams: cpu is higher than it should be [11:53] Gavin.Hird @grid.xmir.org:8002: as stated before I find that idle sim CPU both with ubODe and Bullet is 50% higher on 0.9 over 0.8.2.1 [11:53] Nebadon Izumi: ya that sounds pretty correct [11:53] Nebadon Izumi: i sort of recall these regions hovering around 20-30% at most [11:53] Nebadon Izumi: with 1 avatar it shoots up to around 80-85% [11:53] Nebadon Izumi: even if avatar is just standing not moving [11:53] Gavin.Hird @grid.xmir.org:8002: seeing the same [11:54] Billy.Bradshaw @hg.viewtwo.net:8600: some people report higher memory usage on linux, but they should be aware that linux caches files in memory as a function of the kernel, you can clear the cached files if you wish [11:54] Nebadon Izumi: yea [11:55] Nebadon Izumi: for me memory seems better [11:55] Billy.Bradshaw @hg.viewtwo.net:8600: like so [11:55] Nebadon Izumi: than previous versions [11:55] Billy.Bradshaw @hg.viewtwo.net:8600: free -m && sync && echo 3 > /proc/sys/vm/drop_caches && free -m [11:55] Nebadon Izumi: that could be mono itself though [11:55] Gavin.Hird @grid.xmir.org:8002: if you use something like glances you get a good idea of what's going on with cpu and memory [11:55] Nebadon Izumi: like you said better sgen [11:55] Nebadon Izumi: actually now I think about it too [11:55] Nebadon Izumi: with the C# bullet on my raspberry pi [11:55] Nebadon Izumi: cpu is incredibly higher [11:55] Nebadon Izumi: i forgot about that [11:56] Nebadon Izumi: it's averaging 85-100% of a single core [11:56] Nebadon Izumi: when sitting idle [11:56] Gavin.Hird @grid.xmir.org:8002: with 4.4.1? [11:56] Nebadon Izumi: that is alot higher [11:56] Nebadon Izumi: yea raspberry pi is running 4.4.1 too [11:56] Billy.Bradshaw @hg.viewtwo.net:8600: mono go into some detail as to the sgen improvement with 4.4.0 release notes [11:56] Gavin.Hird @grid.xmir.org:8002: right, because that is what I see for idle sim with 4.4.1 [11:56] Robert Adams: the PI implementation must have been using the software version of Bullet... that will be CPU intensive [11:56] Billy.Bradshaw @hg.viewtwo.net:8600: and a locking issue with the mac [11:57] Nebadon Izumi: the pi is very underpowered and it is also running the C# bullesimXNA [11:57] Nebadon Izumi: but still on older opensim the cpu was much much lower [11:57] Ubit Umarov: hmm my empty var is taking 0% cpu in Win [11:57] Gavin.Hird @grid.xmir.org:8002: macOS version behaves a bit different as it mostly use LLVm and not the mono JIT [11:57] Nebadon Izumi: there is a noticable feeling of lag too when i drive the vehicle around on the pi as well [11:57] Nebadon Izumi: it wasnt horrible but it was noticeable change [11:58] Billy.Bradshaw @hg.viewtwo.net:8600: I have a modest welcome region which rarely goes above 5% cpu [12:00] Gavin.Hird @grid.xmir.org:8002: a simulator with 11 regions used to be 10-12% in 0.8.1. With 0.9 and mono 4.4.1 it is about 90% [12:00] Gavin.Hird @grid.xmir.org:8002: 0.8.2.1 [12:00] Robert Adams: I have also noticed a big difference between Windows and Linux cpu... haven't found out why that would make a difference though [12:02] Gavin.Hird @grid.xmir.org:8002: here is one of the messages we talked about earlier: [12:02] Gavin.Hird @grid.xmir.org:8002: 21:02:08 - [SCENE OBJECT GROUP]: SavedScriptState element had no UUID in object ZHAO Anim Overrider -  SP 0.7 (full perm) [12:03] Gavin.Hird @grid.xmir.org:8002: not rezzed anywhere [12:04] Lucy Afarensis: I have dozens of lines of this in my logs [12:04] Robert Adams: Gavin, that error come out of the script deserializer (complaining about things not found in the XML)... are you on mono? [12:04] Gavin.Hird @grid.xmir.org:8002: yep [12:04] Lucy Afarensis: yes [12:05] Gavin.Hird @grid.xmir.org:8002: but why is it complaining about something that is not rezzed and probably has not been for years [12:05] Lucy Afarensis: none of this was ever on mine [12:06] Nebadon Izumi: I have seen things spew serialization errors on console before and the object in world is actually invisible [12:06] Nebadon Izumi: mostly here on this region heh [12:06] Nebadon Izumi: its been a while since i have seen that happen though [12:07] Lucy Afarensis: Iwas wondering if they might be part of an npc on a dance ball [12:07] Nebadon Izumi: i guess that is a possibility [12:08] Gavin.Hird @grid.xmir.org:8002: even if the NPC is not in-world? [12:08] Robert Adams: don't know why it would be an old object... the code paths seem to be about rezzing an object [12:08] Gavin.Hird @grid.xmir.org:8002: only as part of the description I mean [12:08] Nebadon Izumi: hmm probably not [12:08] Nebadon Izumi: unless it happens when you are making a OAR or something maybe [12:09] Gavin.Hird @grid.xmir.org:8002: as far as I can see it came in via an IAR [12:09] Nebadon Izumi: npcs can spew all kinds of wierd errors on IAR import and exports [12:09] Gavin.Hird @grid.xmir.org:8002: because the name in the asset table is From IAR [12:09] Nebadon Izumi: but generally it wont persist those kind of errors [12:09] Nebadon Izumi: usually you just see it during the import/export [12:09] Nebadon Izumi: if its constantly happening that means it's in the sim or perhaps an attachment maybe [12:10] Gavin.Hird @grid.xmir.org:8002: it is every hour - most likely as part of the flotsamcache update [12:10] Nebadon Izumi: hmm [12:10] Gavin.Hird @grid.xmir.org:8002: I'll see if I can find it stuck somewhere in world [12:11] Nebadon Izumi: ya see if you can track it down by location or to a particular primid [12:11] Nebadon Izumi: may be able to find coordinates in the database [12:11] Nebadon Izumi: I've seen wierd things happen where root prim gets erased [12:11] Nebadon Izumi: but all the child prims remain [12:11] Nebadon Izumi: and cause wierd issues [12:11] Nebadon Izumi: wierd spurious console messages [12:11] Nebadon Izumi: those tend to also be invisible in world [12:11] Gavin.Hird @grid.xmir.org:8002: I can only find it in the asset table in the DB [12:12] Nebadon Izumi: but can still cause problems with the scene [12:12] Nebadon Izumi: in the Robust asset table? [12:12] Gavin.Hird @grid.xmir.org:8002: yes [12:12] Nebadon Izumi: hrmm [12:12] Nebadon Izumi: prim inventory uuids are stored as a blob right? [12:13] Gavin.Hird @grid.xmir.org:8002: yes [12:13] Nebadon Izumi: those may not be so easily searchable on a database query [12:13] Gavin.Hird @grid.xmir.org:8002: but you can regex the blob - at least Postgres can, so it exist in there [12:13] Nebadon Izumi: you could try saving an oar [12:13] Nebadon Izumi: and loading it into a standalone [12:13] Nebadon Izumi: and see if you get similar behavior [12:14] Lucy Afarensis: I have done and it show up [12:14] Lucy Afarensis: in the standalone [12:14] Nebadon Izumi: you have anything running on script timer? [12:14] Nebadon Izumi: try disable script engine in the standalone [12:15] Nebadon Izumi: and see if it goes away [12:15] Lucy Afarensis: will do [12:15] Nebadon Izumi: worth a shot [12:15] Nebadon Izumi: see if you can isolate what is causing it to happen every hour [12:16] Nebadon Izumi: if no change there [12:16] Nebadon Izumi: try disable flotsam cache [12:16] Nebadon Izumi: and see if that has any change [12:17] Gavin.Hird @grid.xmir.org:8002: can do that [12:17] Nebadon Izumi: maybe grep the fcache folder too [12:17] Nebadon Izumi: for the uuid [12:18] Robert Adams: if you have the UUID, you can do a 'show asset UUID' on the console [12:19] Nebadon Izumi: ah ya forgot about that command [12:20] Lucy Afarensis: Error says no UUID [12:20] Lucy Afarensis: SavedScriptState element had no UUID in object Maleficent's brown wings [12:20] Gavin.Hird @grid.xmir.org:8002: the only place I can find the name / description "ZHAO Anim Overrider -  SP 0.7 (full perm)" is if I do a regex search inside the XML in the asset table and thereby get the asset UUID [12:20] Gavin.Hird @grid.xmir.org:8002: it is not rezzed anywhere [12:20] Nebadon Izumi: strange [12:21] Gavin.Hird @grid.xmir.org:8002: very strange [12:21] Nebadon Izumi: maybe those wings are inside a box or a vendor or something? [12:21] Gavin.Hird @grid.xmir.org:8002: they are in a box encoded in the XML [12:21] Nebadon Izumi: a box inside a box [12:22] Lucy Afarensis: I have never seen that term before [12:22] Ubit Umarov: that is an AO, it's an attachment [12:22] Ubit Umarov: or maybe in an attachment.. [12:22] Gavin.Hird @grid.xmir.org:8002: the AO is inside a box of other items made by Arcadia [12:23] Gavin.Hird @grid.xmir.org:8002: it was imported via IAR from OSGrid [12:24] Gavin.Hird @grid.xmir.org:8002: I can't rezz it because there is no corresponding inventory item [12:24] Nebadon Izumi: yea unfortunately Gavin it just may be one of those old unfixable assets [12:24] Gavin.Hird @grid.xmir.org:8002: I can just delete the asset record :-)) [12:25] Nebadon Izumi: ya that might be best Gavin [12:25] Nebadon Izumi: if it's not something you are worried about [12:25] Nebadon Izumi: if it never worked then no major loss [12:25] Nebadon Izumi: unfortunately OSgrid is chock full of old bad assets [12:25] Gavin.Hird @grid.xmir.org:8002: no loss at all [12:26] Nebadon Izumi: things that came in via Second Innventory especially bad [12:26] Lucy Afarensis: I have an inventory full of those [12:26] Nebadon Izumi: no doubt i created 100s of bad assets myself [12:26] Nebadon Izumi: lol [12:26] Gavin.Hird @grid.xmir.org:8002: but the interesting thing is why does the server even chew over it at regular intervals? [12:26] Gavin.Hird @grid.xmir.org:8002: seems like wasted work [12:26] Nebadon Izumi: yea i doubt it has much impact [12:26] Nebadon Izumi: it could just be fcache [12:26] Lucy Afarensis: Thanks for talking about this [12:27] Nebadon Izumi: we have opensim spit out a lot of messages for debugging purposes [12:27] Ubit Umarov: odd is that means region thinks script is running so needs to save its state [12:27] Lucy Afarensis: it has been bugging me [12:27] Ubit Umarov: hmm i guess [12:27] Gavin.Hird @grid.xmir.org:8002: but does the fcache traverse the entire asset table? [12:27] Nebadon Izumi: no [12:27] Nebadon Izumi: i cant think of anything that does that [12:28] Gavin.Hird @grid.xmir.org:8002: I hope not :-) [12:28] Nebadon Izumi: there must be some obscure reference some where in the region [12:29] Nebadon Izumi: bye all, thanks for coming