Chat log from the meeting on 2019-03-19

[11:01] Kayaker Magic: Welcome Andrew!
[11:02] Selby.Evans @grid.kitely.com:8002: hi everyone
[11:02] Gavin.Hird @grid.xmir.org:8002: Hi Andres
[11:02] Andrew Hellershanks: Hello, everyone.
[11:03] Gavin.Hird @grid.xmir.org:8002: Andrew - do I have autoreplace turned on today??
[11:03] Bill Blight waves
[11:03] Andrew Hellershanks: autoreplace?
[11:03] Gavin.Hird @grid.xmir.org:8002: in the viewer
[11:03] Kayaker Magic whispers: waves and beaches!
[11:03] Andrew Hellershanks: I have no idea. I don't know what that is.
[11:03] Andrew Hellershanks: wb, Kayaker.
[11:04] Gavin.Hird @grid.xmir.org:8002: I think I typed Andrew and it came out as Andres
[11:04] Leighton.Marjoram @grid.outandproud.life:8002: hi all
[11:05] Ubit Umarov: hi
[11:05] Gavin.Hird @grid.xmir.org:8002: Hi Ubit
[11:05] Andrew Hellershanks: Gavin, yes it did.
[11:05] Kayaker Magic: Vincent, time to bring up your issue: the Important People are here!
[11:06] Andrew Hellershanks: The time change hasn't thrown people off this week.
[11:07] Andrew Hellershanks: I had some distractions this week so I didn't finish all the work on the OSSL wiki pages but I'm about half way done.
[11:07] Kayaker Magic: California is working on getting off the daylight wastings sytem.
[11:08] Vincent.Sylvester @hg.zetaworlds.com:8002: Okay so I ran into a bit of an issue yesterday updating my databases. I use Mariadb instead of MySQL. Mariadb has tightened restrictions on db queries regarding boolean sets. Meaning inserting a true or false is no longer possible for some fields, instead it has to be 0 or 1. I am not yet sure how this is handled in regards to the queries opensim sends, but it may be a good as time as ever to take a look at the database stuff, especially since postgres is not maintained and revise the system a bit. New changes in mysql, mariadb have also enabled possible performance gains that could be of great benefit.
[11:09] Andrew Hellershanks: Which version of MariaDB introduced that change?
[11:09] Gavin.Hird @grid.xmir.org:8002: why does it has to be 0 or 1? Sounds pretty non.standard to me
[11:09] Vincent.Sylvester @hg.zetaworlds.com:8002: It happened on the move from 10.0 to 10.3
[11:11] Andrew Hellershanks: ok. I have one machine using MariaDB but it is an older version so I haven't seen that issue.
[11:12] Vincent.Sylvester @hg.zetaworlds.com:8002: I just started reading changelogs yesterday, but the changes already seem promising to bring performance improvements if they are taken advantage of. Along the lines there are such changes to the "standard" of SQL and how much it allows. I suppose allowing true or false was a legacy thing that is now removed. Again, not sure if this was just with this particular pdo thing, it may not affect opensim at all depending on how the fields are setup.
[11:13] Gavin.Hird @grid.xmir.org:8002: some database systems can only store 0 and 1 for a boolean filed, but some allow it to be undtermined, meaning not initialized
[11:15] Vincent.Sylvester @hg.zetaworlds.com:8002: The change I made was to the local, temporary and enabled fields in the asset server.
[11:15] Vincent.Sylvester @hg.zetaworlds.com:8002: They are setup as tinyint length 1 so I guess true is a bit longer than 1
[11:18] Andrew Hellershanks: I was just looking for a mention of the change in the MariaDB site but can't find. If the DB was storing the text version of true or false before and now it is not that could lead to some issues.
[11:19] Vincent.Sylvester @hg.zetaworlds.com:8002: I think it just no longer converts true false to 1 0 on the commit, instead it takes all statements literal at which point it will fail.
[11:19] Andrew Hellershanks: If the field is defined as tinyint(1) it shouldn't have allowed the text representations unless a conversion was being done on the fly to make it easier for humans to read.
[11:19] Vincent.Sylvester @hg.zetaworlds.com:8002: It may have been a change in 10.2 since that is a major version change.
[11:19] Andrew Hellershanks: Vincent, that is probably whats going on.
[11:22] Vincent.Sylvester @hg.zetaworlds.com:8002: So the question is how opensim handles this, if it too uses true false for fields that actually have 1 0 in them and relies on the conversion or if all db queries are in accordance to the fieldset. I have not updated any regions to run on newer mariadb since I don't want to brick them. If my power company can stop being weird I will try to setup a vm to test on, in the meantime though. since postgres and the new mysql all over some performance gains, having a look at this stuff may be a good idea regardless.
[11:24] Andrew Hellershanks: Kayaker, were there some particular queries where you noticed the problem?
[11:24] Vincent.Sylvester @hg.zetaworlds.com:8002: Especially from what I read mysql and mariadb now having some form of concurrent queries that is apparently new and faster than before. Haven't gotten to the specifics yet, but if there really is something that could reduce query time then it definitely is something worth looking at. Mesh is ever growing, inventory and assets are under massive loads, anything that can be done to reduce that, even just a few percent, is probably noticeable at this point :)
[11:24] Gavin.Hird @grid.xmir.org:8002: the PGSQL is using code like Convert.ToBoolean
[11:25] Gavin.Hird @grid.xmir.org:8002: so I suppose it is the implementation of that library (NPGSQL) that matters and now exactly how it is stored in the db
[11:28] Gavin.Hird @grid.xmir.org:8002: in Postgres both bool and uuid are native datatypes
[11:28] Vincent.Sylvester @hg.zetaworlds.com:8002: I myself am guilty of using true false when really the field asks for 01, I never gave it much thought, probably because LSL gives no fucks about that normally, but really it is probably good practice to obey fieldsets regardless.
[11:29] Andrew Hellershanks: It is worth investigating this. I was doing a quick check and found a few tables using tinyint(1). I didn't immediately notice where true/false is being used in queries but I suspect it will require digging a little deeper in to the code than just doing a quick search.
[11:29] Bill Blight: Well technically LSL does care the TRUE/FALSE text bools are just constants for 1/0
[11:30] Andrew Hellershanks nods
[11:31] Vincent.Sylvester @hg.zetaworlds.com:8002: Kinda same issue we had with mesh, allowing the broken stuff far too long and then cause trouble when actual standard is imposed.
[11:32] Andrew Hellershanks: First step will be to locate all the boolean fields.
[11:33] Gavin.Hird @grid.xmir.org:8002: first quest is this another non-standard SQL extension in Mariadb?
[11:33] Bill Blight: just a quick glance shows that they are tinyint or int
[11:33] Gavin.Hird @grid.xmir.org:8002: what is the justification to not support somethign that is actually si in the SQL standard?
[11:33] Gavin.Hird @grid.xmir.org:8002: is*
[11:34] Bill Blight: I'd agree with Gavin on that one
[11:34] Bill Blight: and what would this change break in existing infrastructures
[11:34] Andrew Hellershanks: I found 10 tinyint(1) references in the DB migration files.
[11:35] Vincent.Sylvester @hg.zetaworlds.com:8002: I can only guess as to the reasons for the change, I simply observed it and found my asset server suddenly blowing up trying to fetch stuff that wasn't expected
[11:35] Ubit Umarov: i really don't ubderstand what is this talk about
[11:35] Ubit Umarov: boolean is tinyint
[11:36] Gavin.Hird @grid.xmir.org:8002: it is about the definition of 1 and 0
[11:36] Ubit Umarov: on mariadb it is just a alias
[11:36] Ubit Umarov: "These types are synonyms for TINYINT(1). A value of zero is considered false. Non-zero values are considered true:"
[11:36] Ubit Umarov: so what are you talking about ???
[11:36] Ubit Umarov: gezzz
[11:37] Kayaker Magic: Wouldn't the correct thing to do would be to always use some abstration, like TRUE and FALSE in the code, and have a way to define that to whatever the underlying database needed?
[11:38] Ubit Umarov: yes it should kayaker
[11:38] Gavin.Hird @grid.xmir.org:8002: agreed
[11:38] Andrew Hellershanks: Ubit, MariaDB isn't handling true and false as values being passed for storage in boolean fields seems to be the issue.
[11:38] Ubit Umarov: but .. who knows what clever ppl think
[11:39] Ubit Umarov: ( "" on that clever :p )
[11:39] Bill Blight: are there even any actual boolean fields in Opensimulator, or are they all INT/TINYINT ?
[11:39] Ubit Umarov: well let us know if you see a issue on opensim
[11:39] Ubit Umarov: most filelds are tinyint since ever
[11:39] Andrew Hellershanks: Does anyone have some simple SQL queries that can show the problem?
[11:40] Bill Blight: I think you'd have to create the problem first, in this case
[11:40] Andrew Hellershanks: I found only one field that is of type boolean.
[11:40] Andrew Hellershanks: in the OS migrations files.
[11:40] Ubit Umarov: and it is also or should be, a language connector job to provide proper convertion btw types
[11:40] Andrew Hellershanks: agreed.
[11:41] Bill Blight: agreed
[11:41] Ubit Umarov: but DB ppl are so clever.. whatever
[11:41] Gavin.Hird @grid.xmir.org:8002: it should be the db connector's job to convert
[11:41] Bill Blight: so if MariaDB has made a change that is outside, the MySQL spec , we can't really adopt it without possibly causing issues for people using actual mysql
[11:41] Ubit Umarov: utf8 is 3 bytes.. we all should know that..
[11:41] Ubit Umarov: :)
[11:41] Vincent.Sylvester @hg.zetaworlds.com:8002: Don't get hung up on me calling them boolean, that is what I read in the changelog "boolean type fields like (list of them all)" the issue arose with asset server which is based around sras and fsassets, so those potentially have that flaw in them.
[11:42] Vincent.Sylvester @hg.zetaworlds.com:8002: When you give tinyint(1) a true, it fails, give it a 1 and it works
[11:42] Vincent.Sylvester @hg.zetaworlds.com:8002: So searching the code for all queries that push or retrieve fields with tinyints and see if truefalse or 01 is expected/used
[11:43] Ubit Umarov: im not going to upgrade my mariadb now
[11:43] Vincent.Sylvester @hg.zetaworlds.com:8002: The db has 01 in it regardless so that should be what is sent to it, relying on a conversion from truefalse to 01 to happen, well doesn't that also a tiny amount of nanoseconds to each query ;)
[11:43] Andrew Hellershanks: Before digging in to the OS code queries it would be easier to set up a quick test DB with a table containing a boolean field and run a few SQL queries to verify the problem.
[11:44] Ubit Umarov: let us know issues if you upgrade
[11:44] Vincent.Sylvester @hg.zetaworlds.com:8002: I will setup a vm with 10.0 and some regions, then upgrade to 10.3 and see if I get red text
[11:45] Vincent.Sylvester @hg.zetaworlds.com:8002: Though how long until I get to that I can't say
[11:45] Andrew Hellershanks: If you are going to setup a VM just do some simpler tests first.
[11:46] Leighton.Marjoram @grid.outandproud.life:8002: quick question for my own clarity what would your asset server be looking for that has a true/false state I am little confused
[11:46] Andrew Hellershanks: Checking for temporary assets, perhaps.
[11:46] Vincent.Sylvester @hg.zetaworlds.com:8002: $this->put_stmt = $this->pdo->prepare('INSERT INTO '.DB_ASSETS_TABLE_NAME.' (id, asset_type, sha256, name, description, local, temporary, base_dir, created_at, updated_at, enabled) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'); $this->put_stmt->execute([
$asset->id,
$asset->type,
$asset->sha256,
$asset->name,
$asset->description,
$asset->local ? 1 : 0,
$asset->temporary ? 1 : 0,
$asset->base_dir,
$asset->created,
$asset->updated,
$asset->enabled ? 1 : 0
[11:47] Leighton.Marjoram @grid.outandproud.life:8002: ah ok thank you
[11:47] Vincent.Sylvester @hg.zetaworlds.com:8002: noticed 1: 0 was true : false before
[11:47] Ubit Umarov: vincent you did not rebaked
[11:47] Gavin.Hird @grid.xmir.org:8002: he is grey
[11:48] Gavin.Hird @grid.xmir.org:8002: so is Andrew
[11:48] Andrew Hellershanks: Not to me I'm not. :)
[11:48] Kayaker Magic: On to another issue:
[11:48] Kayaker Magic: I'm having a problem with newer versions of OpenSim getting better (more restrictive) at detecting bad mesh prims. This is good, I'm happy to remove those prims or try to fix them. But during region startup I get warning messages that do not help me find the offending mesh. Is there a way I can find out the name or location of the offending prims?
[11:49] Gavin.Hird @grid.xmir.org:8002: isn't there a log message with the region uuid and the asset id?
[11:49] Ubit Umarov: new? those things are as old as 0.9.0.9
[11:49] Andrew Hellershanks: There are several types of error messages that are reported at a low level where the error doesn't propagate up to where you can get information about the offending object.
[11:50] Ubit Umarov: fs did add some restrictions, then flags to disable them
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: By default that part of startup does not "know" which one that is, but you can add some code that does spam the console with that info yes, it may even just be commented in the code somewhere.
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: I think I made a patch some time ago to catch the termination that happens on really bad mesh
[11:50] Bill Blight: the errors you are seeing are most likely from warp3d
[11:50] Bill Blight: about mesh rendering on the map
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: I think Ubit put the same code into master to stop said termination
[11:51] Ubit Umarov: think it was code to try to see the problematic prim
[11:51] Ubit Umarov: not just "you have a bad mesh"
[11:51] Ubit Umarov: :)
[11:52] Vincent.Sylvester @hg.zetaworlds.com:8002: There is a line that is commented in there I think I uncommented that and got like every mesh name uuid it ran through
[11:52] Bill Blight: or you just turn your logging level down
[11:52] Bill Blight: :P
[11:53] Bill Blight: and you don't see the errors
[11:53] Bill Blight: :P
[11:53] Andrew Hellershanks: I have log level at DEBUG and don't see UUID's of bad prims.
[11:53] Ubit Umarov: not all cases are possible to chow
[11:54] Ubit Umarov: or show
[11:54] Andrew Hellershanks: It would be useful to have UUID's reported. Same thing for bad textures.
[11:54] Bill Blight: is that even feasible during map generation
[11:55] Andrew Hellershanks: When bad items are found at a low level some default value gets passed up. I've seen a case of 'null' being passed up the chain. I suspect that the higher level code then skips further processing without reporting on the unexpected(?) value.
[11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: OpenSim\Region\PhysicsModules\ubOdeMeshing\Meshmerizer.cs has a bunch of commented lines that output various things about the meshes it processes, just uncomment them all and watch the console for the warnings it spits out hehe
[11:56] Andrew Hellershanks: I suspect that will spam the console.
[11:56] Vincent.Sylvester @hg.zetaworlds.com:8002: Yes, yes it will :D
[11:57] Andrew Hellershanks: We are almost at the top of the hour. Is there anyone who needs to leave shortly who has a question they wanted to ask?
[11:58] Bill Blight: I try not to get too hung up on console errors if the sim runs fine ... LOL ,, I'd spend all my time chasing things that are not actually causing usage problems ..  :P
[11:58] Gavin.Hird @grid.xmir.org:8002: Not a question: Updated viewers are at https://www.dayturn.com/viewer/index.php?resources/
[11:58] Andrew Hellershanks: Thanks, Gavin.
[11:58] Gavin.Hird @grid.xmir.org:8002: tha macOS version no will not crash a machine with Intel graphics and T2 processor
[11:58] Vincent.Sylvester @hg.zetaworlds.com:8002: Well bad mesh can cause increased resource usage if there are broken physics constantly screwing about, especially if you create physical vehicles and leave them sitting around.
[11:59] Gavin.Hird @grid.xmir.org:8002: the RLV windows version sia major RLV refresh
[11:59] Gavin.Hird @grid.xmir.org:8002: is a*
[11:59] Vincent.Sylvester @hg.zetaworlds.com:8002: It also runs a whole lot better than Firestorm, like almost twice the fps :D
[11:59] Andrew Hellershanks: I thought of some viewer issue just now I was going to comment on but I've forgotten what it was.  :P
[11:59] Cuga Rajal: Dayturn has never run on my Mac system
[12:00] Gavin.Hird @grid.xmir.org:8002: what Mac do you have?
[12:00] Cuga Rajal: Macbook pro 13" 2014, latest High Sierra
[12:00] Bill Blight: If you are leaving non-used physical vehicles sitting around and they are still physical, and causing issues, you need to re-visit your scripts as much as you need to fix the physics mesh
[12:00] Gavin.Hird @grid.xmir.org:8002: OK, try the new build of today
[12:00] Cuga Rajal: I get an error when unpacking, says app is damaged and must be deleted
[12:00] Cuga Rajal: ok will try
[12:01] Cuga Rajal: checking now
[12:01] Vincent.Sylvester @hg.zetaworlds.com:8002: Just beware the windows version may get caught by antivirus because it is built on a vm, rest assured that is a false-positive, I checked :)
[12:01] Vincent.Sylvester @hg.zetaworlds.com:8002: *throws stones at firewall*
[12:02] Gavin.Hird @grid.xmir.org:8002: Cuga, look in Library/Application Support and remove the Dayturn folder and also the KokuaOS folder if you have one
[12:02] Cuga Rajal: I still get message Dayturn Viewer.app” is damaged and can’t be opened. You should move it to the Trash.
[12:02] Cuga Rajal: Wont launch
[12:02] Andrew Hellershanks: How are you downloading the file, Cuga?
[12:02] Gavin.Hird @grid.xmir.org:8002: what is your gatekeeper setting like?
[12:03] Cuga Rajal: The DMG appears to sdownload fine, and I copy the app from the DMG to the desktop
[12:03] Gavin.Hird @grid.xmir.org:8002: it shall not run from the desktop
[12:03] Gavin.Hird @grid.xmir.org:8002: really
[12:03] Gavin.Hird @grid.xmir.org:8002: put it in Applications
[12:03] Cuga Rajal: OK I'll move it to apps
[12:04] Cuga Rajal: I put it in /Applications, right click -> Open, still get damaged message
[12:04] Gavin.Hird @grid.xmir.org:8002: very strange
[12:04] Cuga Rajal: Maybe a high sierra thing?
[12:04] Gavin.Hird @grid.xmir.org:8002: it is even code signed with an Apple dev cert
[12:04] Gavin.Hird @grid.xmir.org:8002: no
[12:05] Gavin.Hird @grid.xmir.org:8002: do you have MacKeeper on your system?
[12:05] Cuga Rajal: Not a code signing issue
[12:05] Gavin.Hird @grid.xmir.org:8002: or some other antivirus shit
[12:05] Cuga Rajal: Ihave no antivisrus
[12:05] Gavin.Hird @grid.xmir.org:8002: MacKeeper?
[12:05] Cuga Rajal: Debug message would be useful
[12:05] Cuga Rajal: Nope
[12:06] Andrew Hellershanks: Is there an sha or md5 sum on the file that can be checked to make sure the file was downloaded correctly?
[12:06] Gavin.Hird @grid.xmir.org:8002: long sisnce you verified your disk?
[12:06] Cuga Rajal: downloaded it from your link, the DMG has its own internal crc
[12:06] Vincent.Sylvester @hg.zetaworlds.com:8002: Maybe start in terminal gives out an error?
[12:06] Cuga Rajal: I can try
[12:06] Gavin.Hird @grid.xmir.org:8002: both the DMG and the app and plugins are signed
[12:07] Cuga Rajal: You mean the Open command, or running the binary inside?
[12:07] Gavin.Hird @grid.xmir.org:8002: use disk utils and verify the disk as it sounds like there might be some corruption
[12:07] Cuga Rajal: I do that regularly
[12:08] Cuga Rajal: Ive downloaded diff versions many times, always same result, never have any other system problems
[12:08] Cuga Rajal: No other app gives me that error, except apps from 10 years ago
[12:08] Gavin.Hird @grid.xmir.org:8002: open Console and look at the system log when it opens
[12:08] Cuga Rajal: ok
[12:09] Cuga Rajal: I am trying to run macos binary at cl
[12:09] Cuga Rajal: will see what happens
[12:09] Cuga Rajal: Oh, it launches when I do ./Dayturn
[12:09] Cuga Rajal: from MacOS folder
[12:09] Cuga Rajal: Lots of console spew
[12:10] Cuga Rajal: But wont run when double-clicking on icon, or using right click->Open
[12:10] Gavin.Hird @grid.xmir.org:8002: very strange
[12:10] Cuga Rajal: Is console spew useful? I can copy it into a NC
[12:10] Gavin.Hird @grid.xmir.org:8002: we had that with Kokua some years ago
[12:11] Cuga Rajal: preparing NC with console output
[12:12] Gavin.Hird @grid.xmir.org:8002: did it generate a log file in ~Libary/Application Support/Dayturn/logs?
[12:12] Cuga Rajal: lemme check
[12:13] Cuga Rajal: There is a log, 39k
[12:13] Cuga Rajal: one launch
[12:14] Cuga Rajal: Want it?
[12:15] Gavin.Hird @grid.xmir.org:8002: actually it would be the same you sent me in the NC
[12:15] Cuga Rajal: ok
[12:15] Andrew Hellershanks: If there isn't anything further for todays meeting I'll call this one to a close and let Gavin and Cuga sort out the problem using the latest Mac viewer.
[12:15] Kayaker Magic: I gota run anyway. Good meeting!
[12:16] Cuga Rajal: Isnt NC limit 32k?
[12:16] Andrew Hellershanks: See you all again next week.