[11:09 AM] Vincent.Sylvester @hg.zetaworlds.com: Hello everyone
[11:10 AM] Vincent.Sylvester @hg.zetaworlds.com: We had some commits this week and apparently just now
[11:10 AM] Ubit Umarov: who is everyone?
[11:10 AM] Ubit Umarov: well about last week code changes...
[11:10 AM] Ubit Umarov: there was one at least
[11:12 AM] Ubit Umarov: a fix to IllInsertString adding missing upper limit check on index argument
[11:12 AM] Lyr.Lobo @cc.opensimulator.org:8002: Hello *grins*
[11:13 AM] Ubit Umarov: also a small change on http accept, moving a line of code inside the try /catch section
[11:14 AM] Ubit Umarov: set of tcp no delay could fail bc reasons, killing the service for all
[11:14 AM] Cuga.Rajal @rajal.org:9000: The socket error happened again last night at 3 am. On the Dec 8 build with the original null socket check.
[11:14 AM] Ubit Umarov: gavin says hi.. he is also traveling
[11:15 AM] Vincent.Sylvester @hg.zetaworlds.com: Did screen catch anything Cuga?
[11:15 AM] Ubit Umarov: well with this change it sill be silent ignored.. we hope
[11:15 AM] Cuga.Rajal @rajal.org:9000: I was not running screen or tcpdump because last week I convinced myself it wasn;t happening anymore
[11:15 AM] Cuga.Rajal @rajal.org:9000: It hasn;t happened for a month
[11:16 AM] Ubit Umarov: and just before the meeting i did push some code in part from vincent, fixing a problem on grid stats regions count
[11:16 AM] Vincent.Sylvester @hg.zetaworlds.com: I still have the honeypot running so maybe it caught it, yes yes still haven't replayed it, been busy with other things :S
[11:16 AM] Cuga.Rajal @rajal.org:9000: So I'm going to update to the latest today, run screen and tcp dump for the next 2 weeks
[11:16 AM] Ubit Umarov: it did not work for standalones
[11:16 AM] Ubit Umarov: ofc now it may not work for all.. details :)
[11:16 AM] Cuga.Rajal @rajal.org:9000: What did not work for standalones?
[11:17 AM] Vincent.Sylvester @hg.zetaworlds.com: The thing I gave you a patch for
[11:17 AM] Ubit Umarov: grid stats
[11:17 AM] Cuga.Rajal @rajal.org:9000: Oh yes, latest patch good
[11:17 AM] Ubit Umarov: well i changed it, ofc
[11:17 AM] Cuga.Rajal @rajal.org:9000: In master now?
[11:18 AM] Ubit Umarov: <osgithub> dotnet compile: success
[11:18 AM] Ubit Umarov: yeap
[11:21 AM] Ubit Umarov: ohh already made a change...
[11:21 AM] Ubit Umarov: removed a silly line with one spaces :)
[11:22 AM] Ubit Umarov: err only spaces
[11:23 AM] Ubit Umarov: so waht news do you hace about opensimulator ? :)
[11:23 AM] Ubit Umarov: err ...have...
[11:23 AM] Vincent.Sylvester @hg.zetaworlds.com: Updated: https://operations.zetamex.com/grid_stats to include time, a search and cleaned it up a bit
[11:24 AM] Vincent.Sylvester @hg.zetaworlds.com: I got some reports of objects not loading in. Apparently LL made a "server side fix" for that which is... great, but it seems either renderer or object cache related
[11:25 AM] Vincent.Sylvester @hg.zetaworlds.com: Need to keep an eye if that keeps happening, could be something on OpenSim end not sending things, though that seems more a bandaid than a fix
[11:25 AM] Vincent.Sylvester @hg.zetaworlds.com: I had some trouble with the link and unlink routines eating a script. The async nature of that has been causing glitches between scene and db data being out of sync for a while now
[11:26 AM] Vincent.Sylvester @hg.zetaworlds.com: Not sure how it ate the script, but I suspect changes to the primitems storing routine unearthed how broken the link routine actually is
[11:26 AM] Vincent.Sylvester @hg.zetaworlds.com: Else why would it have eaten the script
[11:27 AM] Ubit Umarov: the changes i didnt merge?
[11:27 AM] Vincent.Sylvester @hg.zetaworlds.com: Yes the db routine part
[11:28 AM] Ubit Umarov: well link code does not change prim contents...
[11:28 AM] Ubit Umarov: so no idea
[11:28 AM] Vincent.Sylvester @hg.zetaworlds.com: I think that ultimately ate the script as it didn't find the right place to put the script in or something, though not sure how the id would have changed either
[11:29 AM] Vincent.Sylvester @hg.zetaworlds.com: It has been a problem either way. Usually if I place a linkset from inventory, unlink a part, link the rest and take it into inventory. Next time I clear the region cache out I either find the linkset or the unlinked part still on the region
[11:29 AM] Vincent.Sylvester @hg.zetaworlds.com: The way it links and writes to db are async so taking into inventory breaks that path and it deletes it from scene, but keeps the db record
[11:30 AM] Vincent.Sylvester @hg.zetaworlds.com: You can watch it make the entry in the db and then forget to delete it when the inventory take comes in before it has a chance to query the scene for the change
[11:30 AM] Vincent.Sylvester @hg.zetaworlds.com: Never had that eat a script though, so that's worrying
[11:30 AM] Cuga.Rajal @rajal.org:9000: This is on the build with the reduced DB activity?
[11:30 AM] Cuga.Rajal @rajal.org:9000: Or master?
[11:31 AM] Vincent.Sylvester @hg.zetaworlds.com: master + primitems routine that I changed a few weeks ago
[11:31 AM] Vincent.Sylvester @hg.zetaworlds.com: The read first write later type thing
[11:31 AM] Cuga.Rajal @rajal.org:9000: Think the issue is related to primitems change?
[11:32 AM] Vincent.Sylvester @hg.zetaworlds.com: I think it eating the script is, but the underlying cause is more the link routine being async from writing to db
[11:33 AM] Cuga.Rajal @rajal.org:9000: bummer
[11:33 AM] Vincent.Sylvester @hg.zetaworlds.com: Like if you take something into inventory before the region notices the objects have been linked it writes the objects to db, but the scene removes them
[11:33 AM] Vincent.Sylvester @hg.zetaworlds.com: So scene is empty, but db still has the objects. When you then remove the region cache via binary change it loads them from db and they appear again
[11:33 AM] Ubit Umarov: but then sets the need to anothe write
[11:33 AM] Ubit Umarov: bc that flag is set at end
[11:34 AM] Ubit Umarov: anyways mulitiasking is a fun thing
[11:34 AM] Ubit Umarov: but we never had missing scrips
[11:35 AM] Ubit Umarov: as i said the link process does not change prim contents
[11:35 AM] Vincent.Sylvester @hg.zetaworlds.com: If you run the backup command after every action it doesn't break, but if you leave it to the region to write things, because it doesn't do that the moment you make a change to the scene it might go out of sync
[11:35 AM] Ubit Umarov: so if the prim and its contents are saved they should be there
[11:36 AM] Vincent.Sylvester @hg.zetaworlds.com: Yeah I'm not sure how it managed to eat the script. I suspect it is the routine, but I can't figure out how since the dependency is the object uuid, which should remain the same
[11:37 AM] Vincent.Sylvester @hg.zetaworlds.com: It's just a more severe symptom of the underlying issue though
[11:37 AM] Ubit Umarov: and on link/unlik prims are not deleted.. only the linkset
[11:37 AM] Ubit Umarov: they just get a new parent
[11:38 AM] Ubit Umarov: well of db does silly deletes.. bc reasons...
[11:38 AM] Ubit Umarov: well guess more debug is needed on that
[11:38 AM] Vincent.Sylvester @hg.zetaworlds.com: It's relatively easy to reproduce. On an empty region you can easily see it by watching the prims table. Rezz linkset from inventory. Unlink. Delete one part. Link everything. Take linkset back into inventory and delete the unlinked part. Scene is now empty, but db still shows things
[11:38 AM] Ubit Umarov: assuming it is easy to repo
[11:39 AM] Ubit Umarov: delete is also async
[11:39 AM] Ubit Umarov: and can take ages because delete is a take
[11:40 AM] Vincent.Sylvester @hg.zetaworlds.com: Yeah
[11:40 AM] Vincent.Sylvester @hg.zetaworlds.com: I don't know how making that synchronous would end up in terms of performance, probably not great
[11:41 AM] Ubit Umarov: yeah nsty
[11:41 AM] Vincent.Sylvester @hg.zetaworlds.com: Leaving stuff is one thing, but what that might cause and it somehow managing to eat a script, that's scary
[11:41 AM] Vincent.Sylvester @hg.zetaworlds.com: I have to try and repro that script eating and see if the primitems routine is to blame
[11:41 AM] Ubit Umarov: delete does send to viewers that the linkset was deleted to they delete it
[11:42 AM] Ubit Umarov: but it even stays in region colliding
[11:42 AM] Ubit Umarov: until the take to inventory ends
[11:42 AM] Ubit Umarov: so delete adds its own worms to unlink/link tests
[11:42 AM] Vincent.Sylvester @hg.zetaworlds.com: In this case the scene data does have it removed, but it sticks in the db, which is then masked by caches so even a region restart doesn't make it show up again
[11:43 AM] Vincent.Sylvester @hg.zetaworlds.com: The region cache itself has to run out to fetch db data which then makes it pop up again
[11:43 AM] Ubit Umarov: well i tested delete and never seem prims show up after restart
[11:44 AM] Ubit Umarov: * with proper shutdown ofc )
[11:44 AM] Cuga.Rajal @rajal.org:9000: Oh I saw that recently on my region
[11:44 AM] Cuga.Rajal @rajal.org:9000: stuff I deleted was sitting there
[11:44 AM] Ubit Umarov: hmm but it may happen if the take was still in proguess
[11:44 AM] Ubit Umarov: of if it failed
[11:45 AM] Cuga.Rajal @rajal.org:9000: Better for something not to delete than something to disappear that shouldn't
[11:45 AM] Ubit Umarov: well when i do really want to delete a object i jsut set it to temp
[11:45 AM] Vincent.Sylvester @hg.zetaworlds.com: I have started to wait a couple more seconds on unlink and link operations or straight up run backup command in between for larger linksets, but when it ate the script that was just 30 parts
[11:46 AM] Vincent.Sylvester @hg.zetaworlds.com: We also know users have no patience so it is odd this hasn't come up more or they just never report it
[11:46 AM] Ubit Umarov: well no changes on that code in years
[11:46 AM] Ubit Umarov: ( except the inv )
[11:46 AM] Cuga.Rajal @rajal.org:9000: I've seen the issue sporatically for years. But its not a big deal for me
[11:47 AM] Lyr.Lobo @cc.opensimulator.org:8002: /me nods and grins
[11:47 AM] Vincent.Sylvester @hg.zetaworlds.com: I think the more other stuff is changed and runs faster the more we unearth these long standing timing and async issues
[11:47 AM] Vincent.Sylvester @hg.zetaworlds.com: Big engine in car, no brakes to stop kinda thing
[11:47 AM] Ubit Umarov: well and more ram with face ECC etc
[11:47 AM] Ubit Umarov: fake
[11:47 AM] Cuga.Rajal @rajal.org:9000: Yeah the preformance improvement is huge, so if not extra bugs, not a show stopper
[11:47 AM] Ubit Umarov: ( or none )
[11:48 AM] Ubit Umarov: ( ddr5 is local ECC only, but you cant buy it now anyways )
[11:48 AM] Ubit Umarov: is -> has
[11:49 AM] Vincent.Sylvester @hg.zetaworlds.com: I will investigate more and write up a proper mantis for it, perhaps with all the performance increase some async stuff no longer needs to be async to be fast. I rather have linking take two seconds to actually show as linked knowing I won't find random stuff next time I come back home :)
[11:49 AM] Ubit Umarov: but don-t think that is the issue
[11:49 AM] Ubit Umarov: juse some fun bugs dancing around
[11:50 AM] Ubit Umarov: all on that needs to be async
[11:50 AM] Ubit Umarov: ( db updates.. local or remote.. etc )
[11:51 AM] Vincent.Sylvester @hg.zetaworlds.com: Not really. The routines all schedule db writes marking things as changed. They could be the ones strictly executing them as well. Granted that's not ideal for performance on large linksets, but it becomes a question of what part of the routine ultimately is the most prone to cause the issue
[11:51 AM] Vincent.Sylvester @hg.zetaworlds.com: I suspect it's the linking routines, but it might be delete
[11:52 AM] Vincent.Sylvester @hg.zetaworlds.com: Need to test that more to figure out how that interacts with each other
[11:52 AM] Ubit Umarov: yeah stop the simulation writing to mysql
[11:52 AM] Ubit Umarov: fun...
[11:53 AM] Vincent.Sylvester @hg.zetaworlds.com: Yeah
[11:54 AM] Ubit Umarov: :)
[11:54 AM] Vincent.Sylvester @hg.zetaworlds.com: It's a question of finding the worst offender and figuring out if it needs to be fully synchronous or if there is a way to mitigate the problem some other way, not fun not fun at all
[11:54 AM] Ubit Umarov: well we are getting close to end time.. any other issue for this meeting?
[11:54 AM] Vincent.Sylvester @hg.zetaworlds.com: But neither is finding random objects in places that I don't remember putting there, the stupid viewer object cache did that too me one to many times for me to let it just do that you know xD
[11:54 AM] Ubit Umarov: forget the sync..
[11:55 AM] Ubit Umarov: any code that goes out to disk on net must be async
[11:55 AM] Ubit Umarov: disk or..
[11:55 AM] Vincent.Sylvester @hg.zetaworlds.com: Groups isn't
[11:55 AM] Ubit Umarov: at most pseudo sync
[11:56 AM] Vincent.Sylvester @hg.zetaworlds.com: Group info reload halts all other tasks basically
[11:56 AM] Vincent.Sylvester @hg.zetaworlds.com: There is a mantis on that I think
[11:57 AM] Vincent.Sylvester @hg.zetaworlds.com: I totally agree that it cannot be made to do full sync like that, the db stuff just takes too long, but something needs to be made quicker on cleanup or a faster change somewhere so this doesn't get lost
[11:58 AM] Ubit Umarov: well better debug is needed :(
[11:58 AM] Vincent.Sylvester @hg.zetaworlds.com: It's a complex issue certainly
[11:58 AM] Vincent.Sylvester @hg.zetaworlds.com: async handling, timing and db, all the big fun things all in one, yay
[11:58 AM] Ubit Umarov: hp lost entire backups of a client bc similar issues
[11:59 AM] Ubit Umarov: hmm think kings college backups.. not sure
[11:59 AM] Vincent.Sylvester @hg.zetaworlds.com: I remember that heh
[11:59 AM] Ubit Umarov: multitaks side effects on a sh script
[11:59 AM] Cuga.Rajal @rajal.org:9000: I have an unrelated issue/question when there's time
[12:00 PM] Ubit Umarov: opps rl is about to call me
[12:00 PM] Ubit Umarov: you can tell it.. we may think about it until next week :)
[12:00 PM] Cuga.Rajal @rajal.org:9000: Well I can reach Tampa in discord
[12:00 PM] Cuga.Rajal @rajal.org:9000: OK
[12:01 PM] Cuga.Rajal @rajal.org:9000: I'm looking over the tcpdump logs during normal use of my standalone
[12:01 PM] Cuga.Rajal @rajal.org:9000: and I see something not right
[12:01 PM] Cuga.Rajal @rajal.org:9000: When the server receives a request that includes <methodName>get_server_urls</methodName>
[12:02 PM] Cuga.Rajal @rajal.org:9000: It returns an xml payload of the server URLs
[12:02 PM] Cuga.Rajal @rajal.org:9000: But one of them on my site is wrong, it's a Comcast address
[12:02 PM] Ubit Umarov: several of those are set on robust,ini
[12:03 PM] Cuga.Rajal @rajal.org:9000: I am on a standalone
[12:03 PM] Ubit Umarov: well of somewhere on a standalone
[12:03 PM] Cuga.Rajal @rajal.org:9000: The gatekeeper URI that it reports is not correct, but the IP address shown in the xm does not appear in any of my config files
[12:03 PM] Ubit Umarov: [LoginService] WelcomeMessage = "Welcome, Avatar!" SRV_HomeURI = "${Hypergrid|HomeURI}" SRV_InventoryServerURI = "${Const|BaseURL}:${Const|PublicPort}" SRV_AssetServerURI = "${Const|BaseURL}:${Const|PublicPort}"
[12:03 PM] Cuga.Rajal @rajal.org:9000: grep -R "weird IP addres" *
[12:03 PM] Ubit Umarov: ....
[12:04 PM] Ubit Umarov: well and what is the entry name?
[12:04 PM] Cuga.Rajal @rajal.org:9000: Like I said I grepped for the IP address and it does not exist in any of the flat files in my whoile opensim tree
[12:04 PM] Cuga.Rajal @rajal.org:9000: Could ti be in the database? if so, where?
[12:05 PM] Ubit Umarov: (paste above was from standaloneCommon.ini)
[12:05 PM] Cuga.Rajal @rajal.org:9000: The server response reports rajal.org:9000 for all the URIs except one
[12:05 PM] Ubit Umarov: what is the service name on that ?
[12:06 PM] Cuga.Rajal @rajal.org:9000: the server xml payload contains <member><name>SRV_GatekeeperURI</name><value><string>http://73.222.93.33:9000</string></value></member>
[12:06 PM] Cuga.Rajal @rajal.org:9000: tha is, the data returned from the request, as shown in my tcpdump log
[12:06 PM] Cuga.Rajal @rajal.org:9000: Thats a Comcast address, I haven't had Comcast for 8 years!
[12:07 PM] Cuga.Rajal @rajal.org:9000: Its not in any of my config files
[12:07 PM] Ubit Umarov: i dont see that in code
[12:07 PM] Cuga.Rajal @rajal.org:9000: It must be either in a config file or the db
[12:08 PM] Ubit Umarov: check standalonecommon
[12:08 PM] Vincent.Sylvester @hg.zetaworlds.com: It could be a request to another grid returning that back
[12:09 PM] Cuga.Rajal @rajal.org:9000: It is a response to a query
[12:09 PM] Cuga.Rajal @rajal.org:9000: The POST request had:
[12:10 PM] Ubit Umarov: yeah hg users will display odd urls
[12:10 PM] Cuga.Rajal @rajal.org:9000: <?xml version="1.0" encoding="utf-8"?><methodCall><methodName>get_server_urls</methodName><params><param><value><struct><member><name>userID</name><value><string>81b3fb03-3444-4a73-bb8b-f57558741520</string></value></member></struct></value></param></params></methodCall>
[12:10 PM] Cuga.Rajal @rajal.org:9000: So it is just a request for grid info correct?
[12:10 PM] Cuga.Rajal @rajal.org:9000: server URL info
[12:10 PM] Ubit Umarov: think it is about a user servers
[12:10 PM] Vincent.Sylvester @hg.zetaworlds.com: Send me a full log in discord, I'll have a look in a bit
[12:10 PM] Cuga.Rajal @rajal.org:9000: Ok thanks Vincent
[12:11 PM] Ubit Umarov: it is per user
[12:11 PM] Cuga.Rajal @rajal.org:9000: So if one of the URLs is wrong, what table holds that data?
[12:11 PM] Cuga.Rajal @rajal.org:9000: So I can fix
[12:12 PM] Cuga.Rajal @rajal.org:9000: The gatekeeper URI info that is
[12:12 PM] Cuga.Rajal @rajal.org:9000: If its not simple answer I'll reach out to Vincent in Discord
[12:13 PM] Ubit Umarov: that is on the ini files not dbs
[12:13 PM] Cuga.Rajal @rajal.org:9000: Thats what I thought, which concerns me bc that IP address isn't in there
[12:14 PM] Cuga.Rajal @rajal.org:9000: StandaloneCommon.ini: GatekeeperURI = "${Const|BaseURL}:${Const|PublicPort}"
[12:14 PM] Ubit Umarov: hmm some may be stored on user account
[12:15 PM] Ubit Umarov: ok rl calls me
[12:15 PM] Ubit Umarov: hope to see you all next week :)
[12:15 PM] Cuga.Rajal @rajal.org:9000: Right, so thats why i'm asking what db table it is in
[12:15 PM] Cuga.Rajal @rajal.org:9000: tc Ubit
[12:16 PM] Cuga.Rajal @rajal.org:9000: I couldn't find it
[12:16 PM] Ubit Umarov: ini files should override for local users
[12:17 PM] Ubit Umarov: but hg is.. hg :)
[12:17 PM] Ubit Umarov: cya next week :)
[12:17 PM] Cuga.Rajal @rajal.org:9000: I was on Comcast when I set up my grid 10 years ago
[12:18 PM] Ubit Umarov: mb it is stuck somewhere ?
[12:18 PM] Cuga.Rajal @rajal.org:9000: I'm just wondering if this could be a cause for some folks to have trouble with their inbound hg
[12:18 PM] Cuga.Rajal @rajal.org:9000: mb?
[12:18 PM] Ubit Umarov: maybe
[12:18 PM] Ubit Umarov: ok cya
