OS 환경 : Oracle Linux 8.4 (64bit)
DB 환경 : Oracle Database 19.27.0.0
방법 : ORA-01613: instance ORA19DB3 (thread 3) only has 0 logs - at least 2 logs required to enable.
2node rac에서 3번노드를 delnode 한 뒤 다시 add node 함
참고 : 오라클 19c RAC 노드 제거 방법(silent) ( https://positivemh.tistory.com/1244 )
참고 : 오라클 19c RAC 노드 추가 방법(silent) ( https://positivemh.tistory.com/1243 )
이때 발생한 에러들을 게시글로 정리함
내용이 많아 글을 나눠서 업로드함
메인 게시글 : 오라클 19c RAC 제거한 노드 다시 추가 시 발생한 에러들 ( https://positivemh.tistory.com/1312 )
에러1. ORA-01613: instance ORA19DB3 (thread 3) only has 0 logs - at least 2 logs required to enable. ( https://positivemh.tistory.com/1308 )
에러2. ORA-30012: undo tablespace 'UNDOTBS3' does not exist or of wrong type ( https://positivemh.tistory.com/1309 )
에러3. [INS-43042] The cluster nodes [ora19rac3] specified for addnode is already part of a cluster. ( https://positivemh.tistory.com/1310 )
에러4. PRCR-1079 : Failed to start resource ora.cvu ( https://positivemh.tistory.com/1311 )
grid addnode, db addnode는 정상적으로 수행됨
하지만 dbca단계에서 에러 발생함
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
$ ./dbca -silent -addInstance -gdbName ORA19DB -nodeName ora19rac3 -instanceName ORA19DB3
Prepare for db operation
40% complete
Adding instance
48% complete
[WARNING] ORA-01119: error in creating database file '+DATA'
ORA-17502: ksfdcre:4 Failed to create file +DATA
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15018: diskgroup cannot be created
ORA-15001: diskgroup "DATA" does not exist or is not mounted
52% complete
53% complete
[WARNING] ORA-00349: failure obtaining block size for '+DATA'
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15064: communication failure with ASM instance
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
[WARNING] ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15064: communication failure with ASM instance
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
54% complete
[WARNING] ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15064: communication failure with ASM instance
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
58% complete
[WARNING] ORA-01613: instance ORA19DB3 (thread 3) only has 0 logs - at least 2 logs required to enable.
62% complete
66% complete
70% complete
74% complete
80% complete
Completing instance management.
86% complete
100% complete
[FATAL] PRCR-1013 : Failed to start resource ora.ora19db.db
PRCR-1064 : Failed to start resource ora.ora19db.db on node ora19rac3
CRS-5017: The resource action "ora.ora19db.db start" encountered the following error:
ORA-01618: redo thread 3 is not enabled - cannot mount
. For details refer to "(:CLSN00107:)" in "/oracle/app/oracle/diag/crs/ora19rac3/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.ora19db.db' on 'ora19rac3' failed
Look at the log file "/oracle/app/oracle/cfgtoollogs/dbca/ORA19DB/ORA19DB2.log" for further details.
|
thread 3이 redo가 없고 disable 상태라고 나옴
*참고로 DATA diskgroup 없다는 에러는 내가 이전에 diskgroup 이름을 rename해서 DATA가 아닌 DATANEW로 되어있어서 발생하는듯함. 이부분은 무시해도됨 추후 수동으로 redo 및 undo 추가가능함
참고 : 오라클 19c 디스크 그룹 이름 변경 ( https://positivemh.tistory.com/1316 )
현재 grid 상태 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
|
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE ora19rac1 STABLE
ONLINE ONLINE ora19rac2 STABLE
ONLINE ONLINE ora19rac3 STABLE
ora.chad
ONLINE ONLINE ora19rac1 STABLE
OFFLINE OFFLINE ora19rac2 STABLE
ONLINE ONLINE ora19rac3 STABLE
ora.net1.network
ONLINE ONLINE ora19rac1 STABLE
ONLINE ONLINE ora19rac2 STABLE
ONLINE ONLINE ora19rac3 STABLE
ora.ons
ONLINE ONLINE ora19rac1 STABLE
ONLINE ONLINE ora19rac2 STABLE
ONLINE ONLINE ora19rac3 STABLE
ora.proxy_advm
OFFLINE OFFLINE ora19rac1 STABLE
OFFLINE OFFLINE ora19rac2 STABLE
OFFLINE OFFLINE ora19rac3 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE ora19rac1 STABLE
2 ONLINE ONLINE ora19rac2 STABLE
3 ONLINE ONLINE ora19rac3 STABLE
ora.DATANEW.dg(ora.asmgroup)
1 ONLINE ONLINE ora19rac1 STABLE
2 ONLINE ONLINE ora19rac2 STABLE
3 ONLINE ONLINE ora19rac3 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ora19rac2 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE ora19rac1 STABLE
2 ONLINE ONLINE ora19rac2 STABLE
3 ONLINE ONLINE ora19rac3 STABLE
ora.RECONEW.dg(ora.asmgroup)
1 ONLINE ONLINE ora19rac1 STABLE
2 ONLINE ONLINE ora19rac2 STABLE
3 ONLINE ONLINE ora19rac3 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE ora19rac1 Started,STABLE
2 ONLINE ONLINE ora19rac2 Started,STABLE
3 ONLINE ONLINE ora19rac3 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE ora19rac1 STABLE
2 ONLINE ONLINE ora19rac2 STABLE
3 ONLINE ONLINE ora19rac3 STABLE
ora.cvu
1 ONLINE ONLINE ora19rac2 STABLE
ora.ora19db.db
1 ONLINE ONLINE ora19rac1 Open,HOME=/oracle/ap
p/oracle/product/19c
,STABLE
2 ONLINE ONLINE ora19rac2 Open,HOME=/oracle/ap
p/oracle/product/19c
,STABLE
3 ONLINE OFFLINE Dismounted,Mount Ini
tiated,HOME=/oracle/
app/oracle/product/1
9c,STABLE
ora.ora19rac1.vip
1 ONLINE ONLINE ora19rac1 STABLE
ora.ora19rac2.vip
1 ONLINE ONLINE ora19rac2 STABLE
ora.ora19rac3.vip
1 ONLINE ONLINE ora19rac3 STABLE
ora.qosmserver
1 ONLINE ONLINE ora19rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE ora19rac2 STABLE
--------------------------------------------------------------------------------
|
3번 리소스들은 정상적으로 온라인이지만 db만 offline 상태임
thread 확인
|
1
2
3
4
5
6
|
SQL> select thread#, status, enabled from v$thread;
THREAD# STATUS ENABLED
---------- ------ --------
1 OPEN PUBLIC
2 OPEN PUBLIC
|
3번은 아직 없음
파라미터 파일에서 확인
|
1
2
3
4
5
6
7
8
9
10
|
SQL> show spparameter thread
SID NAME TYPE VALUE
-------- ----------------------------- ----------- ----------------------------
* ofs_threads integer
* parallel_threads_per_cpu integer
ORA19DB1 thread integer 1
ORA19DB2 thread integer 2
ORA19DB3 thread integer 3
* threaded_execution boolean
|
파라미터파일에는 3번이 일단 지정은 되어 있음
일단 3번 thread enable 시도
|
1
2
3
4
5
6
|
SQL> alter database enable public thread 3;
alter database enable thread 3
*
ERROR at line 1:
ORA-01613: instance ORA19DB3 (thread 3) only has 0 logs - at least 2 logs
required to enable.
|
redo가 없어서 안된다고 나옴
현재 redo 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
SQL>
set lines 200
set pages 1000
col member for a60
select thread#, l.group#, member, archived, l.status, (bytes/1024/1024) MB, sequence#
from v$log l, v$logfile f
where f.group# = l.group#
order by 1, 2;
THREAD# GROUP# MEMBER ARC STATUS MB SEQUENCE#
---------- ---------- ------------------------------------------------------------ --- ---------------- ---------- ----------
1 1 +DATANEW/ORA19DB/ONLINELOG/group_1.272.1201188457 YES INACTIVE 200 149
1 2 +DATANEW/ORA19DB/ONLINELOG/group_2.273.1201188459 NO CURRENT 200 150
2 3 +DATANEW/ORA19DB/ONLINELOG/group_3.280.1201189697 NO CURRENT 200 107
2 4 +DATANEW/ORA19DB/ONLINELOG/group_4.281.1201189697 YES INACTIVE 200 106
|
1,2번 thread용 리두만 존재함
thread 3용 리두 추가
|
1
2
3
4
5
6
7
|
SQL> alter database add logfile thread 3 group 5 '+DATANEW' size 200m;
Database altered.
SQL> alter database add logfile thread 3 group 6 '+DATANEW' size 200m;
Database altered.
|
현재 redo 재확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
SQL>
set lines 200
set pages 1000
col member for a60
select thread#, l.group#, member, archived, l.status, (bytes/1024/1024) MB, sequence#
from v$log l, v$logfile f
where f.group# = l.group#
order by 1, 2;
THREAD# GROUP# MEMBER ARC STATUS MB SEQUENCE#
---------- ---------- ------------------------------------------------------------ --- ---------------- ---------- ----------
1 1 +DATANEW/ORA19DB/ONLINELOG/group_1.272.1201188457 YES INACTIVE 200 149
1 2 +DATANEW/ORA19DB/ONLINELOG/group_2.273.1201188459 NO CURRENT 200 150
2 3 +DATANEW/ORA19DB/ONLINELOG/group_3.280.1201189697 NO CURRENT 200 107
2 4 +DATANEW/ORA19DB/ONLINELOG/group_4.281.1201189697 YES INACTIVE 200 106
3 5 +DATANEW/ORA19DB/ONLINELOG/group_5.345.1218470839 YES UNUSED 200 0
3 6 +DATANEW/ORA19DB/ONLINELOG/group_6.344.1218470847 YES UNUSED 200 0
6 rows selected.
|
추가됨
다시 thread 확인
|
1
2
3
4
5
6
7
|
SQL> select thread#, status, enabled from v$thread;
THREAD# STATUS ENABLED
---------- ------ --------
1 OPEN PUBLIC
2 OPEN PUBLIC
3 CLOSED DISABLED
|
3번이 생김, 하지만 closed 및 disable 상태임
database config 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
$ srvctl config database -d ora19db
Database unique name: ORA19DB
Database name:
Oracle home: /oracle/app/oracle/product/19c
Oracle user: oracle
Spfile: +datanew/ora19db/parameterfile/spfile.294.1203586513
Password file: +DATANEW/ora19db/PASSWORD/pwdora19db.270.1201188441
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATANEW,RECONEW
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: dba
Database instances: ORA19DB1,ORA19DB2,ORA19DB3
Configured nodes: ora19rac1,ora19rac2,ora19rac3
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
|
3번 인스턴스명은 ORA19DB3임(대문자)
3번 인스턴스 기동 시도
|
1
2
3
4
5
6
7
8
|
$ srvctl start instance -d ora19db -i ORA19DB3
PRCR-1013 : Failed to start resource ora.ora19db.db
PRCR-1064 : Failed to start resource ora.ora19db.db on node ora19rac3
CRS-5017: The resource action "ora.ora19db.db start" encountered the following error:
ORA-01618: redo thread 3 is not enabled - cannot mount
. For details refer to "(:CLSN00107:)" in "/oracle/app/oracle/diag/crs/ora19rac3/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.ora19db.db' on 'ora19rac3' failed
|
thread 3을 enable하라고 나옴
다시 3번 thread enable 시도
|
1
2
3
|
SQL> alter database enable public thread 3;
Database altered.
|
다시 thread 확인
|
1
2
3
4
5
6
7
|
SQL> select thread#, status, enabled from v$thread;
THREAD# STATUS ENABLED
---------- ------ --------
1 OPEN PUBLIC
2 OPEN PUBLIC
3 CLOSED PUBLIC
|
closed 상태이지만 PUBLIC 상태가 됨
다시 3번 인스턴스 기동 시도
|
1
2
3
4
5
6
7
8
9
10
11
12
|
$ srvctl start instance -d ora19db -i ORA19DB3
PRCR-1013 : Failed to start resource ora.ora19db.db
PRCR-1064 : Failed to start resource ora.ora19db.db on node ora19rac3
CRS-5017: The resource action "ora.ora19db.db start" encountered the following error:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-30012: undo tablespace 'UNDOTBS3' does not exist or of wrong type
Process ID: 73679
Session ID: 237 Serial number: 14511
. For details refer to "(:CLSN00107:)" in "/oracle/app/oracle/diag/crs/ora19rac3/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.ora19db.db' on 'ora19rac3' failed
|
thread 문제는 해결이 되었지만 UNDOTBS3를 못찾는듯함
다음 게시글로 이어짐
참고 : ORA-30012: undo tablespace 'UNDOTBS3' does not exist or of wrong type ( https://positivemh.tistory.com/1309 )
참조 :
https://positivemh.tistory.com/284
https://positivemh.tistory.com/1244
Database fails to start on <Node 2> due to ORA-01618: redo thread 2 is not enabled - cannot mount (Doc ID 1677362.1)
Top Issues When Adding Node for Grid Infrastructure via GridSetup.sh (Doc ID 2955583.1)
https://pat98.tistory.com/731
https://docs.oracle.com/en/database/oracle/oracle-database/19/cwadd/cluster-verification-utility-reference.html#GUID-B445A858-9F00-4423-990E-109545AC11C3
https://dbmentors.blogspot.com/2013/11/clusterware-resource-oracvu.html
Clusterware resource ora.cvu FAQ (Doc ID 1524235.1)
Cluster Verification Utility (CLUVFY) FAQ (Doc ID 316817.1)
"Roottfa.sh: Not Found" after executing root.sh script. (Doc ID 2960836.1)
root.sh Hanging During Relink on AIX (Doc ID 3091869.1)
Root.sh Failed During Rac Installation Due To Antivirus (Doc ID 3058530.1)
https://dbacentrals.blogspot.com/2017/08/srvm1337crs-10051-cvu-found-following.html