Chat log from the meeting on 2022-02-22

From OpenSimulator

Revision as of 10:45, 18 May 2022 by Tampa (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
[11:03] Gavin.Hird Hi Andrew
[11:03] Andrew Hellershanks: Hello, everyone.
[11:04] Ubit Umarov: Severity    Code    Description    Project    File    Line    Suppression State
Error    CS1069    The type name 'PerformanceCounter' could not be found in the namespace 'System.Diagnostics'. This type has been forwarded to assembly 'System.Diagnostics.PerformanceCounter, Version=, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' Consider adding a reference to that assembly.    OpenSim.Framework.Monitoring    D:\Avination\test\opensim\OpenSim\Framework\Monitoring\ServerStatsCollector.cs    65    Active
[11:04] Ubit Umarov: for example
[11:04] Ubit Umarov: performance counters also gone
[11:04] Andrew Hellershanks: I nodded off after lunch until my PDA beeped at me to remind me of the time.
[11:04] Ubit Umarov: that also means some of our stats code, gone
[11:05] Ubit Umarov: on migration to .net5
[11:05] Kayaker Magic: I'm told Mike Chase got OpenSim to compile and run with .net on Windows servers, but was surprised to find the resulting code does not run on linux.
[11:05] Ubit Umarov: well the most scary part is the API that do look the same, but are not
[11:06] Ubit Umarov: well to do that he will need to remove thing like this
[11:06] Ubit Umarov: world map
[11:06] Ubit Umarov: etc
[11:06] Ubit Umarov: then rewrite with alternative libs..
[11:07] Ubit Umarov: lludp textures also dead
[11:07] Ubit Umarov: well as i said, a ton of things
[11:07] Ubit Umarov: not sure you did read right that he done it
[11:07] Ubit Umarov: i know he is working on it
[11:08] Ubit Umarov: from my side, also as i told, no rush
[11:08] Ubit Umarov: no point on rushing things
[11:08] Andrew Hellershanks: Seems a little odd that MS would make such changes to .net 5 that it completely breaks code of the past.
[11:08] Ubit Umarov: ohh you should had seen .netcore 2
[11:08] Ubit Umarov: a lot worse
[11:08] Ubit Umarov: even 3
[11:09] Ubit Umarov: only then ms seen they could not force all to rewrite all applications
[11:09] Ubit Umarov: so they did add a lot more of .net4.8 on it
[11:10] Andrew Hellershanks: I wonder if they saw the problem internally. That probably made the more like to do something about it.
[11:10] Ubit Umarov: but still a lot out, like image, etc etc
[11:10] Ubit Umarov: parts they did add.. for windows only
[11:10] Ubit Umarov: net5-0-windows does have image code
[11:10] Misterblue Waves: wonder of PrimMesher and libomv can be upgraded to .NET6... there's a weekend project
[11:11] Ubit Umarov: same issue
[11:11] Ubit Umarov: can't
[11:11] Ubit Umarov: depends on bitmaps
[11:11] Ubit Umarov: and other things
[11:11] Vincent.Sylvester Speaking of alternate libs, I sat down the other day and cloned the Mysql interfaces and rewrote migrations to change a bunch of the varchar and char fields to UUID as now supported by MariaDB. The unit tests pass, though it still uses the default Mysql connector and I have not done much testing on whether all parts work. Ubit looked at a MariaDB connector directly, but it seems that, much like newer npgsql just ships with a bunch of stupid libs. I check table lengths and overall across the tables saved over 280 bytes just changing these fields, which comes out to nearly 30% of overall structure size
[11:12] Ubit Umarov: ms refused to suport image on all platform, like linux
[11:12] Ubit Umarov: bc that libgdiplus, cairo etc
[11:12] Ubit Umarov: they refuse to maintain
[11:12] Ubit Umarov: kinda understandable
[11:13] Misterblue Waves: some headaches shouldn't be volunteered for
[11:13] Ubit Umarov: cairo and pango..
[11:13] Misterblue Waves: I've been using MariaDB as my DB. No problems but then I'm not a heavy load tester
[11:14] Ubit Umarov: maria is now what distros love
[11:14] Andrew Hellershanks: It has been for some time.
[11:14] Ubit Umarov: guess since oracle took over mysql
[11:15] Andrew Hellershanks nods
[11:15] Vincent.Sylvester 10.7 was released last week or so, now has UUID field type, which is a bit smaller in size to store UUIDs, thing is, to take advantage of that kinda need to find a way to interface with it on a lower level than pushing strings in order to get performance out, just changing the fields to UUID just reduces database size overall, query performance not so much
[11:16] Vincent.Sylvester Need a direct binary conversion or something to interface with it, but for that you need a connector and the one that is on nuget needs some hacks to work
[11:16] Andrew Hellershanks: If the code uses cairo, pango, etc. they should still be usable in the OS code as external libraries.
[11:16] Ubit Umarov: as i told you, that takes a split on the dbs
[11:16] Ubit Umarov: and use another connector
[11:17] Ubit Umarov: bitmap on mono  depends on pango, cairo and libgdiplus
[11:17] Ubit Umarov: that ms refuses to use
[11:17] Ubit Umarov: so no image code for linux on .net5 or 6
[11:17] Ubit Umarov: as i said .netx-windows does have it.. windows only
[11:18] Andrew Hellershanks: Why can't the code just use those libraries anyway even if MS won't support them in .net?
[11:18] Ubit Umarov: bc they will not add to .net, code that depends on libs they refuse to use
[11:18] Gavin.Hird no money in that
[11:18] Andrew Hellershanks: Other projects use cairo, pango, etc. so those libraries are still around.
[11:19] Gavin.Hird MS has always been about locking in the customers
[11:19] Ubit Umarov: and .netx code is also for android and ios
[11:19] Andrew Hellershanks: That makes no sense. If you use .NET you have no way to use some other 3rd party library??
[11:20] Ubit Umarov: seems they also think no use really uses it
[11:20] Ubit Umarov: seem some telling only seen code to make q-code :)
[11:20] Ubit Umarov: they tell to use other image libraries for .net
[11:21] Ubit Umarov: fine
[11:21] Ubit Umarov: just means all our code will need a rewrite
[11:21] Ubit Umarov: jpeg2000 code,  scultp maps..  maps.. etc
[11:22] Ubit Umarov: dynamic textures
[11:22] Vincent.Sylvester A lot of that stuff is pretty abstract too, figuring out how that currently works is a big task in itself much less rewriting it
[11:22] Ubit Umarov: all gone, until a rewrite with something else
[11:22] Ubit Umarov: think i only did list concret things
[11:22] Misterblue Waves: what our the "supported" target platforms? Window10/11, Ubuntu (mono6), IOS ?
[11:23] Andrew Hellershanks: The simpler solution is  to write our own code to replace the missing/unsupported bits so the main code doesn't need to be completely rewritten.
[11:24] Vincent.Sylvester Andrew you want to write an image library, sure, I'll schedule the release party, see you in 2035
[11:24] Ubit Umarov: well and still no need to rush
[11:24] Ubit Umarov: sadly mono lost a lot of fixes but still around
[11:24] Ubit Umarov: so no need to rush
[11:25] Ubit Umarov: any change to .net5(6 also means to lose a lot of good hardware outthere
[11:25] Andrew Hellershanks: If current/older .NET is using pango, cairo, etc. Why not just make a library that is an interface between the current code and the backend libraries that were already being used?
[11:25] Ubit Umarov: just because that hardware can't run last windows
[11:25] Ubit Umarov: or even last linux
[11:25] Andrew Hellershanks: What does MS say to developers about moving to .NET 5/6? Rewrite your programs?
[11:26] Vincent.Sylvester I'm more worried that latest mono no longer supports postgres without massive changes, not sure if mysql is not to follow soon
[11:26] Ubit Umarov: yes that is what they say
[11:26] Ubit Umarov: as i said where a lot worse in the times of .netcore 2.0
[11:26] Misterblue Waves: @Andrew it's true that we don't use all the features of Image. Most of the "will not support" stuff is around the functions that reach into Images and twiddle the internal formats (for preformance and stuff)
[11:26] Ubit Umarov: bc most code jsut did not run on it
[11:26] Andrew Hellershanks: If they keep doing that it is a great advertisement for why you should not use .NET in your programs.
[11:27] Ubit Umarov: go tell original opensim devs that the choice of c# was a very stupid one
[11:27] Vincent.Sylvester At that point might as well rewrite in C directly and throw the framework dependency out the window, but man does that sound like fun... not
[11:27] Ubit Umarov: it was very stupid,  but too late to change
[11:28] Ubit Umarov: ofc to start such thing in c++ with multi.os support was a lot harder to do
[11:28] Gavin.Hird someone needs to download the SL server code :-)
[11:29] Ubit Umarov: so possible no one would had done it
[11:29] Ubit Umarov: without the c# original promise to run (badly) on windows and linux
[11:29] Andrew Hellershanks: I haven't done enough C# programming to realize that you have to use .NET.
[11:29] Ubit Umarov: and so far it did..
[11:30] Ubit Umarov: there is no c# without .net
[11:30] Ubit Umarov: basicly
[11:30] Ubit Umarov: well at least some core parts of it
[11:30] Ubit Umarov: we can not use linq for example
[11:31] Vincent.Sylvester Only option is mkbundle, stuff .net into the app, which blows its size up and isn't guaranteed to work either, especially not at OpenSim complexity levels
[11:31] Ubit Umarov: but well academic talks
[11:31] Ubit Umarov: its c# now
[11:31] Ubit Umarov: just a hard crossroad ahead
[11:31] Ubit Umarov: but again, no need to rush it
[11:31] Vincent.Sylvester Well hard if MS decides to continue to ruin the fun
[11:31] Ubit Umarov: the more we wait the easier is should get
[11:31] Ubit Umarov: hopefuly
[11:32] Ubit Umarov: bc others will do it :p
[11:32] Vincent.Sylvester I'm crazy enough to have hope they eventually grow a brain
[11:32] Andrew Hellershanks: :)
[11:32] Ubit Umarov: all software is getting more insane
[11:32] Vincent.Sylvester I blame javascript
[11:33] Ubit Umarov: and ppl will be forced to rewrite things
[11:33] Andrew Hellershanks: hehe
[11:33] Ubit Umarov: well don't know
[11:33] Ubit Umarov: just guess we have like 2 years to see the .net6 issue
[11:34] Ubit Umarov: possible more
[11:34] Ubit Umarov: about  opensim
[11:34] Ubit Umarov: not many changes last week, i think
[11:34] Ubit Umarov: in fac i almost did release today
[11:35] Ubit Umarov: the date 22-2-22 was a fun release date
[11:35] Vincent.Sylvester hehe
[11:35] Vincent.Sylvester I didn't even notice that
[11:35] Andrew Hellershanks: Just one change this past week to the profiles module and a fix to a log message.
[11:36] Ubit Umarov: well i did update libomv with a bit faster OSD to json code
[11:36] Andrew Hellershanks: For me it would be a 2-22-22 release date. Not quite as fun but almost.
[11:36] Ubit Umarov: and that was basicly it
[11:36] Kayaker Magic: I have a question about de-duplication. Does OpenSim do this on images? I hear Kitely does.
[11:37] Ubit Umarov: sounds you need to wash those ears to get full indo
[11:37] Ubit Umarov: and info
[11:37] Ubit Umarov: :p
[11:37] Vincent.Sylvester Dedupe on assets is done hash based, if the hash is the same, the item is just linked via the DB, dynamic textures and such are regionside so they are cleared anyways
[11:37] Andrew Hellershanks: Kayaker, There is an asset system that eliminates duplications.
[11:37] Ubit Umarov: fsassets
[11:38] Ubit Umarov: and xassets
[11:38] Ubit Umarov: or how its called
[11:38] Vincent.Sylvester sras and phpsas do it too :)
[11:38] Andrew Hellershanks: Yup, fsassets.
[11:38] Andrew Hellershanks: SRAS came before fsassets.
[11:38] Ubit Umarov: yes fsassets is a c# version of sras
[11:38] Ubit Umarov: basible
[11:38] Ubit Umarov: and basicly
[11:39] Ubit Umarov: xassets does it, still storing all on mysql
[11:39] Ubit Umarov: well still marked experimental
[11:39] Kayaker Magic: When is the hash on images calculated? After decoding images into raw bitmaps?
[11:40] Ubit Umarov: sha256
[11:40] Ubit Umarov: the SL asset data
[11:40] Ubit Umarov: and exactly as SL asset
[11:40] Ubit Umarov: a fun thing ppl talking about nrew viewers forget
[11:41] Ubit Umarov: we only have assets in SL format
[11:41] Kayaker Magic: So after compressing them for the asset database? Is that zip or are images just jpg2000 compressed?
[11:41] Ubit Umarov: any convertion to other formats is just impossible to do in real time
[11:41] Ubit Umarov: unless on small prof of concept things
[11:41] Ubit Umarov: actually not need, bc no concept to prove
[11:42] Ubit Umarov: to use other formats, the asset storage will need to change to add them
[11:43] Ubit Umarov: well the avination unreal viewer was using SL assets formats
[11:43] Misterblue Waves: ya... new asset server. New formats, new access permissions, scalable... another whole project
[11:44] Ubit Umarov: yeah
[11:44] Misterblue Waves: in existing viewers the conversion from SL asset format to a usable format is all done in the client
[11:44] Ubit Umarov: a asset UUID will need to point to several formats
[11:44] Ubit Umarov: like a texture in jp2000 and dds
[11:45] Misterblue Waves: maybe make it an nginx plugin
[11:45] Ubit Umarov: well and do that FAST
[11:45] Ubit Umarov: the FAST part is something all forget
[11:45] Ubit Umarov: and without it, it is useless
[11:46] Ubit Umarov: we all know any viewer can be made...
[11:46] Ubit Umarov: the unreal one was should most of a region
[11:46] Ubit Umarov: at 6fps !!!
[11:46] Ubit Umarov: so a nice work... useless.. but nice
[11:47] Ubit Umarov: fs was showing same region with 40 fps and doing all things
[11:47] Ubit Umarov: see what i mean?
[11:48] Ubit Umarov: err would help if i did type right :p
[11:48] Ubit Umarov: the unreal one was SHOWING a region at 6fps :)
[11:48] Ubit Umarov: still no terrain, water or avatars
[11:48] Vincent.Sylvester Meanwhile new FS release is looking like it will improve performance by a good 10% from what I can tell, I don't recall ever getting more than 60fps here
[11:49] Ubit Umarov: ll is improving performance
[11:49] Ubit Umarov: they have another batch of changes for a future viewer
[11:49] Ubit Umarov: fs is adding those and a few own
[11:50] Ubit Umarov: ofc it does depend on region creators
[11:50] Ubit Umarov: you will need to pay atention to region build, if you want fps, also
[11:50] Ubit Umarov: and have customs police :p
[11:51] Ubit Umarov: not letting avatars with complexity off 700000 to get in
[11:51] Ubit Umarov: :)
[11:51] Andrew Hellershanks: :)
[11:52] Ubit Umarov: so what news do you bring ?
[11:52] Ubit Umarov: about opensim..
[11:52] Andrew Hellershanks: No news is good news?  ;)
[11:53] Andrew Hellershanks: Will there be a release today?
[11:53] Ubit Umarov: if about ukraine yes
[11:53] Ubit Umarov: nah guess i will not do it today
[11:54] Ubit Umarov: osgrid also did none on basicly
[11:55] Ubit Umarov: but diva is waiting for it  to release her code
[11:56] Ubit Umarov: i had promissed her a reease in Jan
[11:56] Ubit Umarov: and we are almost in march :)
[11:56] Gavin.Hird but what year?
[11:57] Ubit Umarov: yeah i didn't told her the year..  but she assumed 2022 :)
[11:57] Gavin.Hird :-)
[11:57] Ubit Umarov: so i can't use that trick on her :p
[11:58] Ubit Umarov: well and the users that are asking for the code
[11:58] Gavin.Hird they must be begging, asking will not do
[11:59] Ubit Umarov: :)
[11:59] Andrew Hellershanks: Wow. This hour has just flown by. We are at the top of the hour already.
[12:00] Andrew Hellershanks: Any final questions or comments for today before we wrap up this gathering?
[12:02] Andrew Hellershanks: Kayaker, fyi, in a large database a hash based asset storage system can reduce the asset table substantially. I've seen anywhere from 25% to 50% reductions in table size.
[12:02] Ubit Umarov: sadly if table size means nothing :p
[12:03] Ubit Umarov: ofc tables size is smaller, just because the asset data is not stored on it, but on  a disk file
[12:03] Kayaker Magic: I was curious that hashes of images are so likely to be different for similar images.
[12:03] Andrew Hellershanks: yes, but it is a substantial savings in used disk space.
[12:04] Kayaker Magic: And I don't trust jpg to produce the same imagage from the same data 2ice in a row...
[12:04] Ubit Umarov: guess a lot of ppl uploads the same things
[12:04] Ubit Umarov: well it is not jpg, it is jpeg2000, a bit diferent
[12:05] Ubit Umarov: and with paramenters fixed on viewers
[12:05] Ubit Umarov: so end data may be more identick
[12:05] Ubit Umarov: other wise, the hash is just waste of time
[12:05] Ubit Umarov: same for mesh..
[12:06] Ubit Umarov: hash only does anything if mesh is upload with same paramenters and identical viewer
[12:06] Ubit Umarov: so finla bits are exactly the same
[12:06] Ubit Umarov: or then again hashing is uselsss
[12:07] Ubit Umarov: so in fact o do have doubts on how effective it really is
[12:07] Ubit Umarov: but ppl tell it is.. so..  one must believe,,
[12:08] Ubit Umarov: and good viewers do not add Cretion date and uploader uuid to the binary data
[12:08] Ubit Umarov: assuming region code will do it ( and does not )
[12:08] Ubit Umarov: if mesh where as spec, the hash would fail always
[12:09] Kayaker Magic: Some sort of hash that only looks at the image data, or vertex data of mesh would be nice.
[12:09] Misterblue Waves: gonna run in RL.. will be here virtually
[12:09] Ubit Umarov: no
[12:09] Ubit Umarov: hash only looks to raw asset data
[12:09] Andrew Hellershanks: ok, Misterblue.
[12:09] Ubit Umarov: cna't look to asset details
[12:09] Vincent.Sylvester Somewhere in this mess someone set xml utf16 for linksets even if all data is utf8 -.-
[12:09] Ubit Umarov: no time for that
[12:10] Ubit Umarov: nahh think most are actually utf8 telling they are utf16
[12:10] Gavin.Hird are you sure it cannot look at asset data?
[12:10] Ubit Umarov: not was it is done
[12:10] Vincent.Sylvester Yeah that's what I mean, the xml header claims utf16 for no reason, threw me off at first
[12:10] Ubit Umarov: and no..  would be very slow to look per asset type
[12:10] Gavin.Hird on postgres I have a tool that when looking at single records will decode the images in the asset table on the fly
[12:11] Gavin.Hird they are just stored in a blob filed
[12:11] Ubit Umarov: even doing sha256 is not that fast
[12:12] Ubit Umarov: some cpus do have hardware help for that.. not sure .net uses
[12:12] Gavin.Hird you only have to do it on first write in that table
[12:12] Ubit Umarov: no
[12:12] Ubit Umarov: on any upload
[12:12] Gavin.Hird sure, but uploads does not happen that often
[12:13] Ubit Umarov: and does not matter
[12:13] Ubit Umarov: any bit changed means other hash
[12:13] Andrew Hellershanks: yup
[12:13] Ubit Umarov: for now we are not filling the creation date, and creator id  of mesh spec
[12:13] Gavin.Hird image data cannot be edited in-world, so only uploads needs to be rehased
[12:13] Ubit Umarov: so at least that is not a issue
[12:14] Ubit Umarov: even so same mesh even with same parameters will upload diferently
[12:14] Ubit Umarov: acording to particular viewer
[12:14] Gavin.Hird yes
[12:14] Gavin.Hird but image data should be fairly constant
[12:15] Ubit Umarov: but well as i said seems that hash does savings.. so.. better than nothing
[12:15] Ubit Umarov: jpeg2000 constant why?
[12:15] Gavin.Hird there are not many updates to that spec...
[12:15] Ubit Umarov: done with diferent versions of openjpeng on kdu
[12:15] Gavin.Hird we don't use kdu
[12:16] Gavin.Hird no viewer does for opensim
[12:16] Ubit Umarov: thin lower precision bits
[12:16] Ubit Umarov: just think...
[12:16] Ubit Umarov: thin fs payed kdu
[12:16] Ubit Umarov: think
[12:16] Gavin.Hird nobody are interested in paying thousands of dollars for a KDU lisence for opensim
[12:17] Gavin.Hird fs got KDU from LL
[12:17] Ubit Umarov: it is not for opensim
[12:17] Ubit Umarov: is for all
[12:17] Gavin.Hird Kokua paid for
[12:17] Andrew Hellershanks: kdu?
[12:17] Ubit Umarov: viewers do not pay kdu for sl or opensim
[12:17] Ubit Umarov: that is not like havok
[12:17] Gavin.Hird they pay for KDU
[12:17] Ubit Umarov: they pay per viewer
[12:18] Gavin.Hird with no chance of getting the cost covered for opensim, nobody will want to pay for it
[12:18] Ubit Umarov: so if they pay "for sl" they payed "for opensim"
[12:18] Andrew Hellershanks: I looked up kdu and the top result was General Sir John Kotelawala Defence University.  :)
[12:18] Ubit Umarov: unlike havok, where they just use LL license
[12:18] Gavin.Hird yes, but the only one who have paid for KDU is kokua
[12:18] Gavin.Hird whispers: FS got it from LL
[12:19] Ubit Umarov: ok
[12:19] Gavin.Hird apart from that nobody use it because of cost
[12:19] Ubit Umarov: yeha fs ships with openjpg for opensim
[12:19] Gavin.Hird yes
[12:20] Gavin.Hird not allowed to use the license outside of SL
[12:21] Gavin.Hird so I have had to strip the KDU support out of the 3.0 builds
[12:22] Ubit Umarov: well all those diferences make the deduplication fail
[12:22] Ubit Umarov: not going to the detail of all cpu FPUs are diferent etc
[12:23] Ubit Umarov: intel things have bugs remember ? :P
[12:24] Ubit Umarov: so i have no real numbers on how good is fs deduplication good
[12:24] Ubit Umarov: hope it does dedupe the "new script" :p
[12:25] Vincent.Sylvester Good question, I know it certainly doesn't do notecards, cause they have creation date and such saved in their data so the hash is always different
[12:25] Ubit Umarov: yeah
[12:25] Vincent.Sylvester I wrote an article on the garbage OpenSim collects on these things
[12:25] Ubit Umarov: some assets have date etc
[12:25] Ubit Umarov: mesh where suposed to have
[12:26] Ubit Umarov: but it is server job to add those, and we never did
[12:26] Ubit Umarov: not sure what viewers add to the jpeg2000 metadada
[12:27] Ubit Umarov: possible also date and creator there, making ghe hash useless on mesh
[12:27] Ubit Umarov: on texture i mean
[12:27] Ubit Umarov: wel easy for you to check if using fsassets :)
[12:28] Ubit Umarov: upload same textyre twcie and compare the hashs
[12:29] Ubit Umarov: not sure if who made sras and fsassets knew the metadata viewers add
[12:30] Gavin.Hird I don't think the viewer adds any metadata to the raw imagedata
[12:30] Ubit Umarov: well it is j2000 data not raw image
[12:31] Gavin.Hird you understand what I mean
[12:31] Ubit Umarov: ok :)
[12:31] Ubit Umarov: like crreation data on the info areas of the j2k
[12:33] Ubit Umarov: animation seems to be just anim
[12:34] Selby.Evans bye all
[12:34] Ubit Umarov: cya
[12:34] Andrew Hellershanks: Another half hour has passed. I need to get going so I will close the meeting for today. Thank you all for coming. See you again next week.
Personal tools
About This Wiki