We are using 4 node Oracle 10GR2 RAC in Windows x64. 3 instances of our database crashed today and later started normally. I started them by connecting (Remote Desktop) to individual instances and executing startup command in sqlplus.
I can connect to database using SYS account but when I try to connect using application user account, I get following error:
An error was encountered performing the requested operation:
ORA-01033: ORACLE initialization or shutdown in progress 01033. 00000 - "ORACLE initialization or shutdown in progress" *Cause: An attempt was made to log on while Oracle is being started up or shutdown. *Action: Wait a few minutes. Then retry the operation. Vendor code 1033
Interesting thing is my instances started and database are open according to following queries.
SELECT * FROM GV$INSTANCE; SELECT * FROM GV$DATABASE;
I tried to use following command thinking that there is a mistake.
SQL> alter database open; alter database open * ERROR at line 1: ORA-01531: a database already open by the instance
I also tried to control RAC node statuses using following command.
srvctl status database -d DBNAME
database status command is following:
Instance DBNAME1 is running on node NODE01 Instance DBNAME2 is running on node NODE02 Instance DBNAME3 is running on node NODE03 Instance DBNAME4 is running on node NODE04
So I can connect to database using SYS account, but WHY not application user account?
Situation resolved itself in 15 minutes. I can connect to database now. But there is nothing in the alert logs about this situation. Should I check additional places to learn about real problem? Is there anything I can do in similar situation?
After a while, I understood why this is happening! Oracle executes following command which we can see in alert logs.
ALTER DATABASE OPEN
Some time later , We can see following line in alert.log
Completed ALTER DATABASE OPEN
Only after this line, we can connecto to database. Interesting thing is in this 4 node RAC , we need to wait for this event in all instances before database start serving.