[12:00 PM PT] Andrew Hellershanks: Hello, everyone.
[12:01 PM PT] Andrew Hellershanks: Good thing you are all seated. You might have fainted by my having shown up a couple minutes early. ;)
[12:02 PM PT] Andrew Hellershanks: Hello, Motoko.
[12:02 PM PT] Motoko.Karu @login.digiworldz.com:8002: Hihi Andrew....
[12:02 PM PT] Motoko.Karu @login.digiworldz.com:8002: hello everyone....
[12:03 PM PT] Ubit Umarov: hi
[12:04 PM PT] Andrew Hellershanks: We are missing some of the regulars.
[12:04 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Devs taking a vacation, no commits to report
[12:05 PM PT] Ubit Umarov: lasy devs!
[12:05 PM PT] Ubit Umarov: oops
[12:07 PM PT] Andrew Hellershanks: I noticed that. I usually check the changelog before I start the meeting.
[12:07 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Cleaning out all those pesky notecards over the weekend, 365000 asset entries, about 43000 files with a total of 210MB so it's definitely worth cleaning assets once in a while
[12:07 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Adjusting scripts to stop further spam to a rate of about 60 per day now
[12:08 PM PT] Andrew Hellershanks: How were you able to tell that they were referenced anywhere?
[12:08 PM PT] Vincent.Sylvester @hg.zetaworlds.com: I wrote a routine to fetch the hash of the asset and check if it exists in the database, if not moving it out of assets
[12:08 PM PT] Andrew Hellershanks: um... were not referenced
[12:09 PM PT] Andrew Hellershanks: Reasonably safe to do as long as the NC UUID wasn't referenced in a script.
[12:09 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Takes a while to read through given the amount of files somewhere around 7 million or so
[12:09 PM PT] Andrew Hellershanks nods
[12:10 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Well I nuked the notecards knowing that they are only used as a sort of backup anyways
[12:10 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Plus the bad scripts creating them every five minutes they don't deserve the respect to continue to exist
[12:11 PM PT] Andrew Hellershanks: It makes me wonder how some of the breeder systems work. If they are updating the status of the objects to NCs that would also be filling a DB with a lot of unneeded assets.
[12:11 PM PT] Jagga Meredith: hang on, I'm confused. You looked up the hash of the asset where? woudnt it already be in the database even if it was an orphan ?
[12:12 PM PT] Vincent.Sylvester @hg.zetaworlds.com: The system I use is somewhat based on SRAS so I have a hash.gz on disk that is mapped to an asset uuid via the database. So yeah I just look at the filename and see if it exists in the database at all, if not the asset is either removed or was temporary
[12:13 PM PT] Jagga Meredith: ah ok
[12:13 PM PT] Vincent.Sylvester @hg.zetaworlds.com: SRAS and similar systems store assets as files via their hash, so that multiple asset uuids can reference the same data if the data happens to be the same thing. Deduplication to reduce storage requirements and increase lookup speed
[12:14 PM PT] Ubit Umarov: ( ie like fsassets )
[12:14 PM PT] Andrew Hellershanks: It is amazing to see just how much space can be saved via the deduplication it provides.
[12:15 PM PT] Jagga Meredith: Deduplicatin - there's a word I haven't heard in awhile
[12:15 PM PT] Andrew Hellershanks: :)
[12:15 PM PT] Vincent.Sylvester @hg.zetaworlds.com: It seems to be about 30% from what I can tell, just from around 7 million files on disk and about 13 million assets listed in db
[12:16 PM PT] Vincent.Sylvester @hg.zetaworlds.com: The larger the asset the more likely that an alteration creates a new one so it's not as effective as it might seem at first, but it is definitely better than storing them all directly in the database
[12:16 PM PT] Jagga Meredith: I'm working with a developer. He's not grasping the concept NowI have a word to throw at him
[12:17 PM PT] Ubit Umarov: well talking just about hash, the concept is nonosense
[12:17 PM PT] Andrew Hellershanks: The amount of savings does vary greatly depending on the nature of the stored assets. I have seen savings of over 40% on another system.
[12:17 PM PT] Ubit Umarov: hash are not unique...
[12:18 PM PT] Ubit Umarov: but one as large as shs256 is almost unique...
[12:18 PM PT] Ubit Umarov: sha256
[12:19 PM PT] Vincent.Sylvester @hg.zetaworlds.com: I have also seen uuid collisions happening. Whenever you represent data with anything less you run the risk of that as you need good ciphers for lossless compression
[12:19 PM PT] Vincent.Sylvester @hg.zetaworlds.com: In the grand scheme the likelihood is low enough for issues to not be catastrophic, but they are possible
[12:20 PM PT] Ubit Umarov: well timebased uuid collisions should be mininal
[12:20 PM PT] Ubit Umarov: guess the ones we now use by default
[12:20 PM PT] Ubit Umarov: keep forgetting :)
[12:20 PM PT] Ubit Umarov: well we use the ms default for guid
[12:21 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Last time I saw one was with a prim id on a region. No idea how it happened. Just removed an object, restored from a backup, all back to normal. It's rare, but knowing me I seem to attract such bugs
[12:21 PM PT] Ubit Umarov: "this is a convenient static method that you can call to get a new Guid. The method creates a Version 4 Universally Unique Identifier (UUID) as described in RFC 4122, Sec. 4.4"
[12:21 PM PT] Ubit Umarov: this...
[12:22 PM PT] Ubit Umarov: ofc some claim dbs hate this.. but well dbs are jsut silly things...
[12:22 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Data is based heh
[12:23 PM PT] Ubit Umarov: think type 7 or 8 has "db" friendly changes.. pfff..
[12:24 PM PT] Ubit Umarov: note dotnet 8 uses type 4... don't remember what others(mono) did
[12:25 PM PT] Ubit Umarov: ofc if you do not fix the amchine time, and it starts always in 1970.. well.. things may be bad :)
[12:26 PM PT] Vincent.Sylvester @hg.zetaworlds.com: When it comes to databases in that regard sql might as well stand for stop questioning logic
[12:26 PM PT] Jagga Meredith: but but but I like SQL
[12:27 PM PT] Ubit Umarov: issue is on search algos..
[12:27 PM PT] Ubit Umarov: or so they claim
[12:28 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Compared to what lookup was like in the past it's like comparing bogo to bubble sort. Though apparently that's been something mysql is looking to advance now, while mariadb seems to focus on memory consumption lately
[12:29 PM PT] Ubit Umarov: well ther use of hashs on fsassets did scare me also.. ( still not that confortable with it )
[12:30 PM PT] Vincent.Sylvester @hg.zetaworlds.com: As assets get bigger with mesh and things it does worry me as well
[12:30 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Could increase the size of the hash, but then would have to re-hash everything
[12:31 PM PT] Ubit Umarov: and burn cpu on the3 generators :)
[12:31 PM PT] Ubit Umarov: hashs are not light things..
[12:31 PM PT] Vincent.Sylvester @hg.zetaworlds.com: The reading and writing also isn't great for the disks
[12:32 PM PT] Ubit Umarov: well guess no one else has such use of hashs for deserielize...
[12:32 PM PT] Vincent.Sylvester @hg.zetaworlds.com: I see a constant load of 2mbit on the database, which seems to mostly be from asset requests so it's already heavy on the reading
[12:32 PM PT] Ubit Umarov: err deduplicate i mean
[12:33 PM PT] Ubit Umarov: bc iti if formaly wrong..
[12:33 PM PT] Ubit Umarov: ...it is...
[12:33 PM PT] Ubit Umarov: well the hash does not prevent reads from disk...
[12:34 PM PT] Ubit Umarov: may make disk cache a bit more effective...
[12:34 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Hash-based caching and such things are fairly common, so are uuids. Just the scale is massive compared to most other systems to the point certain filesystems will struggle to even keep a journal
[12:34 PM PT] Ubit Umarov: no
[12:34 PM PT] Ubit Umarov: hashs are speedup searchs things.. not full ID
[12:35 PM PT] Ubit Umarov: hashsets and dictionaries of use them..
[12:35 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Caching is a great idea on assets, but implementing it is even more scary than hashes given how fluid data can be
[12:35 PM PT] Ubit Umarov: but at end there is always a final compare with real keys ( unless on trivial cases )
[12:36 PM PT] Jagga Meredith: if you're doing that much activity, is the disk cache ever going to get a chance to write?
[12:36 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Don't really need cache much on nvme
[12:36 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Would have to read past 100mbit to see performance dips
[12:36 PM PT] Ubit Umarov: well proper nvme has internal cache :)
[12:38 PM PT] Ubit Umarov: dont remembe if windows and linux disk caches are writeh though..
[12:38 PM PT] Ubit Umarov: win used to ask
[12:39 PM PT] Ubit Umarov: bc power loss could mean data loss
[12:39 PM PT] Vincent.Sylvester @hg.zetaworlds.com: I did experiment with a CDN, but the invalidation was just a pain. You get very little caching out of it due to fluidity of data and what is cached is often cached once and never requested again for as long as you set validity so unless the validity is infinite it's basically the same as direct read. The only upside then is transfer latency being reduced due to closer proximity, but with decent connections that's not something you ultimately notice
[12:40 PM PT] Vincent.Sylvester @hg.zetaworlds.com: The time it takes for the viewer to process and deal with all the data locally ends up being the cap on things more than the server itself
[12:40 PM PT] Vincent.Sylvester @hg.zetaworlds.com: That's very evident in localhost setups when you have a lot of content and basically instant delivery and yet it still takes the same time to load and render it all
[12:41 PM PT] Ubit Umarov: and not using region asset service kills dynamic textures etc
[12:41 PM PT] Ubit Umarov: hmm and npcs textures?
[12:42 PM PT] Ubit Umarov: don't remember abt those :)
[12:42 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Well my tests were mostly just CDN used by regions to bring them closer, not direct viewer assets
[12:42 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Direct viewer assets could work if a fallback to regions was implemented in the viewer
[12:42 PM PT] Ubit Umarov: hmm aren't yr regions all on same local net ??
[12:43 PM PT] Vincent.Sylvester @hg.zetaworlds.com: No I have servers all over the place close to wherever the users are
[12:43 PM PT] Ubit Umarov: ofc at least 10Gb net by now no? :)
[12:43 PM PT] Ubit Umarov: ohh users to tp all over the place
[12:44 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Yeah but you generally want to place a region close to the owner so at least they have a decent experience with it
[12:44 PM PT] Ubit Umarov: but yeah to serve regions using CDN may be fine
[12:45 PM PT] Vincent.Sylvester @hg.zetaworlds.com: It does work, but only to reduce the distance, it doesn't reduce the load on the primary copy due to just how unique assets are
[12:45 PM PT] Lyr.Lobo @cc.opensimulator.org:8002: Hello *grins*
[12:45 PM PT] Ubit Umarov: hi
[12:45 PM PT] Vincent.Sylvester @hg.zetaworlds.com: I set caches to hours or even days and only ever saw I single request, just not enough duplicates across to have multiple
[12:46 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Invalidating on update is also a bit iffy and happens often enough that some things won't be cached for long
[12:46 PM PT] Vincent.Sylvester @hg.zetaworlds.com: It likely would have bigger hit rates if directly going to viewers, but breaking dynamic textures isn't worth that
[12:47 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Not that they work great as of late anyways, but that's a viewer issue
[12:48 PM PT] Andrew Hellershanks: We are getting close to the top of the hour. Does anyone have another topic of discussion for today?
[12:49 PM PT] Jagga Meredith: Kinda off topic, but I'm attempting to develop a Zoom app One of the first things they ask for is the URL of your OAuth authenticator.
[12:49 PM PT] Jagga Meredith: Nowhere, anywhere can I find how to set up one.
[12:49 PM PT] Jagga Meredith: Zoom isn't on the list of providers
[12:50 PM PT] Jagga Meredith: am I even making sense?
[12:52 PM PT] Vincent.Sylvester @hg.zetaworlds.com: Google provides one. They are a special url schema with an email, issuer and secret in there
[12:52 PM PT] Vincent.Sylvester @hg.zetaworlds.com: The application then requests validation from that endpoint
[12:52 PM PT] Jagga Meredith: ah, good. I'll look into that
[12:53 PM PT] Ubit Umarov: i only remember that that is a pain :)
[12:53 PM PT] Ubit Umarov: wel lost gmail bc of that
[12:54 PM PT] Ubit Umarov: on llemail etc
[12:56 PM PT] Andrew Hellershanks: A family member used to used gmail. I set up Thunderbird to pull from their normal mail provider and also gmail. Gmail changed their security feature and that killed the ability for Thunderbird to retrieve mail from their site.
[12:56 PM PT] Lyr.Lobo @cc.opensimulator.org:8002: Thunderbird had an update around the 10th of January and ever since, crashes frequently for me
[12:57 PM PT] Ubit Umarov: but ti does get email from gmail
[12:57 PM PT] Lyr.Lobo @cc.opensimulator.org:8002: they may have fixed it...not crashing for the first time in 11 days *grins*
[12:57 PM PT] Ubit Umarov: guess you have a config issue somewhere
[12:58 PM PT] Ubit Umarov: my thunderbird is getting email from gmail even using pop3
[12:58 PM PT] Lyr.Lobo @cc.opensimulator.org:8002: yes
[12:58 PM PT] Ubit Umarov: wel with the security options etc
[12:58 PM PT] Andrew Hellershanks: Hello, Lyr.
[12:58 PM PT] Lyr.Lobo @cc.opensimulator.org:8002: Hello Andrew, sorry, class ran over *smiles*
[12:58 PM PT] Andrew Hellershanks: It became a non-issue as the person must started using a web browser to access their gmail account.
[12:59 PM PT] Andrew Hellershanks: s/must/just/
[12:59 PM PT] Lyr.Lobo @cc.opensimulator.org:8002 grins, thinking back to Kernighan and Ritchie's ed tutorial for was/is line editor changes
[1:00 PM PT] Ubit Umarov: im not in the mood to use open a webmail per each of my email accounts
[1:00 PM PT] Andrew Hellershanks: Ubit, yes, it was retrieved via pop3 at one time but gmail changed so you couldn't do that any more.
[1:00 PM PT] Ubit Umarov: review the settings.. it worlks
[1:01 PM PT] Ubit Umarov: you need to tell it to use oauth etc etc
[1:01 PM PT] Ubit Umarov: read gmail docs :)
[1:01 PM PT] Jagga Meredith: (wait, weren't we just talking about that?)
[1:01 PM PT] Andrew Hellershanks: I tried many times after gmail changed their system. I could just never get it to work again. Anyway, no longer an issue.
[1:01 PM PT] Ubit Umarov: tehy don't like pop3 buit still there
[1:02 PM PT] Ubit Umarov: not to make regions to use it.. is a pain bc Oauth.. so not supported now
[1:03 PM PT] Ubit Umarov: you need a mailler that still supports simple logins
[1:04 PM PT] Andrew Hellershanks: Any other last minute items for today?
[1:06 PM PT] Andrew Hellershanks: Seeing no response I will close todays gathering to a close. Thank you all for coming. See you again next week.
[1:06 PM PT] Ubit Umarov: :)
