MantisBT - opensim
View Issue Details
0004875opensim[GRID] User Servicepublic2010-07-15 10:182019-02-09 20:32
samiam 
 
normalminoralways
closedno change required 
 
 
.7 rc2
Grid (Multiple Regions per Sim)
ODE
.NET / Windows32
None
0004875: After migration Some users that could log in .69 cannot in .7 rc2
After looking at the tables data and after migration I see that user and useracounts are the same (can trace uuid ok) But the table auth is missing some.
Correct row count in both users and useraccounts = 69 (correct)
auth = 56 (Wrong) and the user uuids ect are missing totaly and of corse thoughs users cannot log in.
I tryed to set the auth migration level to 0 and it re-did it but still only 56 users show up out of the normal 69 in auth table.

No tags attached.
Issue History
2010-07-15 10:18samiamNew Issue
2010-07-15 10:18samiamGit Revision => .7 rc2
2010-07-15 10:18samiamSVN Revision => 0
2010-07-15 10:18samiamRun Mode => Grid (Multiple Regions per Sim)
2010-07-15 10:18samiamPhysics Engine => ODE
2010-07-15 10:18samiamEnvironment => .NET / Windows32
2010-07-15 10:18samiamMono Version => None
2010-07-15 10:27samiamNote Added: 0016076
2010-07-15 10:44DivaNote Added: 0016078
2010-07-15 10:54samiamNote Added: 0016079
2010-07-15 10:59samiamNote Added: 0016080
2010-07-15 11:31aiaustinNote Added: 0016081
2010-07-15 11:32aiaustinNote Edited: 0016081
2010-07-15 11:33aiaustinNote Edited: 0016081
2010-07-15 11:34aiaustinNote Edited: 0016081
2010-07-15 12:06samiamNote Added: 0016083
2010-07-15 13:53DivaNote Added: 0016084
2010-07-15 18:45samiamNote Added: 0016087
2010-07-16 07:36samiamNote Added: 0016094
2010-07-16 08:03samiamNote Added: 0016095
2010-07-16 08:55samiamNote Added: 0016096
2010-07-16 11:03samiamNote Added: 0016097
2010-07-16 11:22samiamNote Edited: 0016094
2010-07-16 11:22samiamNote Edited: 0016097
2010-07-16 11:24samiamNote Added: 0016098
2010-07-16 11:49samiamNote Added: 0016099
2010-07-16 13:30samiamNote Added: 0016101
2010-07-16 13:50samiamNote Added: 0016102
2010-07-16 13:57samiamNote Added: 0016103
2010-07-16 14:08DivaNote Added: 0016104
2010-07-16 16:41samiamNote Added: 0016112
2010-07-16 19:45samiamNote Added: 0016116
2010-07-16 19:56samiamNote Added: 0016117
2010-07-16 20:13samiamNote Added: 0016118
2010-07-16 20:14samiamNote Edited: 0016116
2010-07-17 12:39samiamNote Added: 0016133
2010-07-18 10:25samiamNote Added: 0016148
2010-07-18 11:41samiamNote Added: 0016150
2010-09-01 02:21aiaustinNote Edited: 0016081
2019-02-09 20:28tampaStatusnew => resolved
2019-02-09 20:28tampaResolutionopen => no change required
2019-02-09 20:28tampaAssigned To => tampa
2019-02-09 20:32BillBlightStatusresolved => closed
2019-02-09 20:32BillBlightAssigned Totampa =>

Notes
(0016076)
samiam   
2010-07-15 10:27   
note: the 56 users that are in the auth table can log in and things are normal for them. prob dosent help. :)
(0016078)
Diva   
2010-07-15 10:44   
Please figure out what's common among the ones that fail and the ones that don't fail. Or was there an error during migrations?
(0016079)
samiam   
2010-07-15 10:54   
I know i should NOT do this but was on a JUNK mysql test data base. I dropped the migration tables totaly (i know i know lol) but intrestingly enough all 3 tables show the correct users at 69 in auth.
Mabey something during the migration is being done out of order? (guessing)
(0016080)
samiam   
2010-07-15 10:59   
Looking at the data in the tables useracounts and cross checking uuids and user password hashes i see nothing that is dis simular to other acounts that can log in. Errors i couldent find as far as miragtion but ill take a closer look at the logs, easy to miss something.
(0016081)
aiaustin   
2010-07-15 11:31   
(edited on: 2010-09-01 02:21)
Some of the tables ar going to be unused in 0.7 remember. But worth keeping them until you resolve the issues. I had one avatar that could not log in after 0.6.9 to 0.7 upgrade from my 20 or so. Only difference I could see in the one that did not work was that it had a password "salt" field complete as well as it's password hash. All others had the password salt field empty. But clearing that field entry in a MySQL query/db browser tool did not fix things. I deleted the user and reestablished it as a new avatar to fix this.

(0016083)
samiam   
2010-07-15 12:06   
Humm im not finding anything that is ovbious. like bad data to start with or somehing. After i droped the migrations table totaly, forcing it to re-migrate everything. I can log in with one of the afflicted accounts now.
I got lucky and my main account is fine and i had a test avie of mine that was effected. Now both can log in and all the accounts show up in auth table like the rest. One thing is that most of the ones that seem to have this problem are newer accounts, going by the dates it seems the accounts made in .69 are the ones it dosent seem to like. But data wise i dont see any diff from one to another. All i know is the auth table is not being filled with all the users there are on the inital migration.

I wouldent have a problem to force a 're-migration' (dropping the migration table) but was told thats a really bad idea and wouldent help find the why of it.
(0016084)
Diva   
2010-07-15 13:53   
I think I've heard that before -- that accounts created with 0.6.9 don't migrate well. Was that Master Mirage on -dev? Nothing reliable, as far as I remember. But worth trying to investigate further, if you can.
(0016087)
samiam   
2010-07-15 18:45   
Ok, ill reset the tests back and run it all again. It is alot of data it needs to convert in one shot. Be easyer now that i know what to watch for.
(0016094)
samiam   
2010-07-16 07:36   
(edited on: 2010-07-16 11:22)
After reseting all test (dumped the opensim table) reloaded a working .69 backup again (and the large assest's still migrate fine btw) I then see all the users accounts in the users table. Useraccounts table however only has some and the missing ones are accounts made .69 not older ones.
Auth table at this point is 56 and useraccounts is also 56.
Yesterday i applyed a man. step (had this happen before) was to drop the usersaccounts table, set migration level for this to 0 then rerun robust.
Then for whatever resion lets the usersaccounts table = the users table to (69 users) correctly.
However auth table remains at 56 users so seems the problem starts sooner than it seemed in the mirgration process as useraccounts is made 1st form the users table.
Right now i have not done the drop usersaccounts table hack so i can see the diff between user and useraccounts. If there are test proceedures i can apply its at the state to do that. Durring inital migration of users there were no logged warrning's that tell me anything new.

At least reseting the tests lets me know it wasent some large migration gremlin
as the result is the same as the 1st time.

(0016095)
samiam   
2010-07-16 08:03   
Is there some migration level i can change that would make it just re-do the users accounts process to see if its skipping over something it shouldent?
All the data is there still in user's table and i know all the accounts in it are working fine (old or new) in .69 that the backup is made from.
(0016096)
samiam   
2010-07-16 08:55   
@ AI, i looked at password salt you had mentioned and None of the .69 accounts have anything in that field. (users table), Useraccounts dosent have this feild at all. Auth table has the field again with 2 of the 56 accounts there with data in it. I assume thats the result of me logging in testing at rev .7rc2 though. The remaining auth accounts remain blank at this point.
(0016097)
samiam   
2010-07-16 11:03   
(edited on: 2010-07-16 11:22)
Looking at other tables i notice also that agents only has 56 but griduser has 69. I dont know if that helps. Not shure of the process that sorts what data to where. The uuid formats seem ok as in there not missing a dash or lack the correct format. Its hard to do an apples to oranges diff with the actual data as they would differ from account to account anyway.

(0016098)
samiam   
2010-07-16 11:24   
Corrected minor number error on my part there are 69 users not 67.. oops
(0016099)
samiam   
2010-07-16 11:49   
Ahh mabey this is something. I use xoopensim and make a new avie account, I can then log in to .7 rc2 fine with it. So at least that process is ok. The account made shows up fine in auth table and its data looks like the other 56 that are allready there. So at least making newusers accounts and whatever process it uses is working.
(0016101)
samiam   
2010-07-16 13:30   
Ok i did this working backwards from my hack this time. Dropping useraccounts table then working backward's with the useracount migration setting that end up as 4.
level 3 did nothing Level 2 did nothing dame with 1. Setting it to 0 then triggered robust to make the usersacounts table and fully populated it with the correct number of users from the users table. But the auth table remains the same (dident look like it this hack triggers it as well). This just gets it to a point that user table and useraccounts table are the same at 69 users.
The inital migration dident do that on its own.

At this point im not shure what other migration to trigger so it can also update the auth table from there. AuthStore mabey? or is that something else totaly?
I can work backwards if i know what next to try.
Anway it took a migration level of 0 to force it to re trigger and fill in useraccounts totaly.
(0016102)
samiam   
2010-07-16 13:50   
2010-07-16 16:16:47,859 INFO - OpenSim.Server.OpenSimServer [SERVER]: Loading LLLoginServiceInConnector on port 8002
2010-07-16 16:16:47,875 INFO - OpenSim.Data.Migration [MIGRATIONS]: Upgrading UserAccount to latest revision 4.
2010-07-16 16:16:47,875 INFO - OpenSim.Data.Migration [MIGRATIONS]: NOTE - this may take a while, don't interrupt this process!
2010-07-16 16:16:47,921 INFO - OpenSim.Data.Migration [MIGRATIONS]: Creating UserAccount at version 1
2010-07-16 16:16:47,953 INFO - OpenSim.Data.Migration [MIGRATIONS]: Updating UserAccount to version 2
2010-07-16 16:16:48,687 INFO - OpenSim.Data.Migration [MIGRATIONS]: Updating UserAccount to version 3
2010-07-16 16:16:49,156 INFO - OpenSim.Data.Migration [MIGRATIONS]: Updating UserAccount to version 4
2010-07-16 16:16:49,171 INFO - OpenSim.Data.Migration [MIGRATIONS]: GridStore up to date, no migrations to apply
2010-07-16 16:16:49,187 DEBUG - OpenSim.Services.GridService.GridService [GRID SERVICE]: Starting...
2010-07-16 16:16:49,187 INFO - OpenSim.Data.Migration [MIGRATIONS]: AuthStore up to date, no migrations to apply
2010-07-16 16:16:49,187 INFO - OpenSim.Data.Migration [MIGRATIONS]: InventoryStore up to date, no migrations to apply
2010-07-16 16:16:49,187 INFO - OpenSim.Data.Migration [MIGRATIONS]: GridUserStore up to date, no migrations to apply
2010-07-16 16:16:49,187 DEBUG - OpenSim.Services.UserAccountService.GridUserService [USER GRID SERVICE]: Starting user grid service
2010-07-16 16:16:49,187 INFO - OpenSim.Data.Migration [MIGRATIONS]: AuthStore up to date, no migrations to apply
2010-07-16 16:16:49,203 INFO - OpenSim.Data.Migration [MIGRATIONS]: InventoryStore up to date, no migrations to apply
2010-07-16 16:16:49,203 INFO - OpenSim.Data.Migration [MIGRATIONS]: GridStore up to date, no migrations to apply
2010-07-16 16:16:49,203 DEBUG - OpenSim.Services.GridService.GridService [GRID SERVICE]: Starting...
2010-07-16 16:16:49,203 INFO - OpenSim.Data.Migration [MIGRATIONS]: Presence up to date, no migrations to apply
2010-07-16 16:16:49,203 DEBUG - OpenSim.Services.PresenceService.PresenceService [PRESENCE SERVICE]: Starting presence service
2010-07-16 16:16:49,218 INFO - OpenSim.Data.Migration [MIGRATIONS]: Avatar up to date, no migrations to apply
2010-07-16 16:16:49,218 DEBUG - OpenSim.Services.AvatarService.AvatarService [AVATAR SERVICE]: Starting avatar service
2010-07-16 16:16:49,218 INFO - OpenSim.Data.Migration [MIGRATIONS]: FriendsStore up to date, no migrations to apply
2010-07-16 16:16:49,218 DEBUG - OpenSim.Services.LLLoginService.LLLoginService [LLOGIN SERVICE]: Using instantiated LibraryService
2010-07-16 16:16:49,218 DEBUG - OpenSim.Services.InventoryService.LibraryService [LIBRARY]: Starting library service...

Is what the hack triggers in robust log but nothing about auth..
This also seems to have no ill effects to anything else i see.
(0016103)
samiam   
2010-07-16 13:57   
Humm in that log i see it checked authstore x2. If thats the migration that causes auth tables to update, Why check it 2 times if its already up to date?
(0016104)
Diva   
2010-07-16 14:08   
Yes, you want to play with AuthStore. Delete it from the Migrations table and run again. As long as you keep the old users tables around, your data is safe, and you should be able to try the migration as many times as you want.

Migrations are checked several times upon startup, that's normal behavior.
(0016112)
samiam   
2010-07-16 16:41   
Ok backing authstore down one level at a time from 3, robust now isant happy.
(i have not dropped the auth table is prob why but might be something in the log).

2010-07-16 19:29:08,718 INFO - OpenSim.Server.OpenSimServer [SERVER]: Loading AuthenticationServiceConnector
2010-07-16 19:29:08,718 INFO - OpenSim.Data.Migration [MIGRATIONS]: Upgrading AuthStore to latest revision 3.
2010-07-16 19:29:08,718 INFO - OpenSim.Data.Migration [MIGRATIONS]: NOTE - this may take a while, don't interrupt this process!
2010-07-16 19:29:08,734 DEBUG - OpenSim.Data.Migration [MIGRATIONS]: Cmd was Table 'auth' already exists in SQL: begin;
 CREATE TABLE `auth` (
   `UUID` char(36) NOT NULL,
   `passwordHash` char(32) NOT NULL default '',
   `passwordSalt` char(32) NOT NULL default '',
   `webLoginKey` varchar(255) NOT NULL default '',
   PRIMARY KEY (`UUID`)
 ) ENGINE=InnoDB;
 CREATE TABLE `tokens` (
   `UUID` char(36) NOT NULL,
   `token` varchar(255) NOT NULL,
   `validity` datetime NOT NULL,
   UNIQUE KEY `uuid_token` (`UUID`,`token`),
   KEY `UUID` (`UUID`),
   KEY `token` (`token`),
   KEY `validity` (`validity`)
 ) ENGINE=InnoDB;
 commit;
 
2010-07-16 19:29:08,734 DEBUG - OpenSim.Data.Migration [MIGRATIONS]: An error has occurred in the migration. This may mean you could see errors trying to run OpenSim. If you see database related errors, you will need to fix the issue manually. Continuing.
2010-07-16 19:29:08,734 INFO - OpenSim.Data.Migration [MIGRATIONS]: Creating AuthStore at version 1
2010-07-16 19:29:08,750 DEBUG - OpenSim.Data.Migration [MIGRATIONS]: Cmd was Duplicate entry '0922026d-a6fe-4c4c-9c2c-36fdb3187b39' for key 'PRIMARY' in SQL: BEGIN;
 INSERT INTO auth (UUID, passwordHash, passwordSalt, webLoginKey) SELECT `UUID` AS UUID, `passwordHash` AS passwordHash, `passwordSalt` AS passwordSalt, `webLoginKey` AS webLoginKey FROM users;
 COMMIT;
 
2010-07-16 19:29:08,750 DEBUG - OpenSim.Data.Migration [MIGRATIONS]: An error has occurred in the migration. This may mean you could see errors trying to run OpenSim. If you see database related errors, you will need to fix the issue manually. Continuing.
2010-07-16 19:29:08,750 INFO - OpenSim.Data.Migration [MIGRATIONS]: Updating AuthStore to version 2
2010-07-16 19:29:08,781 DEBUG - OpenSim.Data.Migration [MIGRATIONS]: Cmd was Duplicate column name 'accountType' in SQL: BEGIN;
 ALTER TABLE `auth` ADD COLUMN `accountType` VARCHAR(32) NOT NULL DEFAULT 'UserAccount';
 COMMIT;
 
2010-07-16 19:29:08,781 DEBUG - OpenSim.Data.Migration [MIGRATIONS]: An error has occurred in the migration. This may mean you could see errors trying to run OpenSim. If you see database related errors, you will need to fix the issue manually. Continuing.
2010-07-16 19:29:08,781 INFO - OpenSim.Data.Migration [MIGRATIONS]: Updating AuthStore to version 3
2010-07-16 19:29:08,796 INFO - OpenSim.Server.OpenSimServer [SERVER]: AuthenticationServiceConnector loaded successfully
2010-07-16 19:29:08,796 INFO - OpenSim.Server.Base.HttpServerBase [SERVER]: Requested port 8002

At that test at least robust is protesting about it.
(still dosent change anything in auth table and remains at 56 users)

Next step will be to do that again after i drop the table auth and looks like also start it at migration level 0 for authstore.
Its a bit hard to tell that though because it hates it when the auth table is already there but to not skip over something ill do it the slow way lol.
(0016116)
samiam   
2010-07-16 19:45   
(edited on: 2010-07-16 20:14)
Yaaahhh!! :) Ok now all tables have 69 users and all user accounts (new or old) can log in with out issues.

So it looks like the problem starts from the inital migration by not filling useraccount table totaly. In this case outta he box only got 56 from the users table. (no ovb data reasion or log saying it aborted it)
To test passed that and to get useracounts table filled correctly i had to manualy drop the useraccounts table and also set user acount migration level back to 0.
Then ran robust. (This forced both users and users acounts tables to have the correct amout of users at that point but not auth table.

Next I dropped the auth table and also set Auth store migration to level 0.
Ran robust again
This then forced it remake that table and filled it to the correct number of users as there should be.

Now i know thats just a hack way to do this but at least it seems to eliminate alot of mabey from the WHY of it.
It cant be some bit of data making it barf or it would do that all the time and this hack wouldent have worked no matter what.

I altered no table data at all.

Anyway i cant tell if the 2and part was the result of the 1st part not doing it right to start with on the inital migration.
Something is wonky there at least.

What might help is that migration checks the total user number in the user table then looks at the auth table to see its the right number or not.
Mabey it could then trigger this type of hack to correct for it should it happen. Or at least Log some usefull debug info at that point.

(0016117)
samiam   
2010-07-16 19:56   
All other migrations worked fine (no hammer needed) this may be the last of these kinda problems and if solved i think a normal migration can happen with out any supprises finnaly :) I know The way im testing it is rather harsh given the ammout of data it has to chew on but eventualy db's of this size will be common and hopfully doing tests like this now will save others there sanity a bit :)
(0016118)
samiam   
2010-07-16 20:13   
geer ment authStore migration level to 0..
(0016133)
samiam   
2010-07-17 12:39   
Well the HACK works, reset and re-tested what i found 3x now :)
Though the normal migration still wount do it on its own (same restult each time). So it remains a bug but ppl can work around it at least.

Tnx all
(0016148)
samiam   
2010-07-18 10:25   
Another note to this and a warning to others. DONT get the idea to drop amd change both useraccounts and auth and change both migration levels then run robust.
You have to do it order , including running robust between each change.
If you dont do it in the right way it can do verry bad things.
(0016150)
samiam   
2010-07-18 11:41   
My theory at this point is that sometime in the .69 series a migration was done or skiped or not un-done that went wrong but was ignored. Then in .7rc2 its not ignored but it cant correct it normaly. It could make it look like one problem when its really not what it seems.
Of corse im not shure thats the case and i dont fully understand the process.
But it might explain why some have this and others dont ?