OS환경 : Oracle Linux 7.6 (64bit)
DB 환경 : Oracle Database 19.10.0.0 RAC
방법 : 오라클 19c RAC sqlnet.ora SQLNET.AUTHENTICATION_SERVICES = (NONE) 설정 시 발생하는 문제
오라클 19c RAC 환경에서
$ORACLE_HOME/network/admin/sqlnet.ora와
$GRID_HOME/network/admin/sqlnet.ora에
SQLNET.AUTHENTICATION_SERVICES = (NONE)를 추가했을때 발생하는 문제를 확인하기위해 테스트함
기존 rac 상태 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
|
# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.chad
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.net1.network
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.ons
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.proxy_advm
OFFLINE OFFLINE oel19db1 STABLE
OFFLINE OFFLINE oel19db2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel19db2 STABLE
ora.OCR_VOTE.dg(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oel19db1 Started,STABLE
2 ONLINE ONLINE oel19db2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.cvu
1 ONLINE ONLINE oel19db2 STABLE
ora.oel19db1.vip
1 ONLINE ONLINE oel19db1 STABLE
ora.oel19db2.vip
1 ONLINE ONLINE oel19db2 STABLE
ora.oradb.db
1 ONLINE ONLINE oel19db1 Open,HOME=/oracle/ap
p/oracle/product/19c
,STABLE
2 ONLINE ONLINE oel19db2 Open,HOME=/oracle/ap
p/oracle/product/19c
,STABLE
ora.qosmserver
1 ONLINE ONLINE oel19db2 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel19db2 STABLE
--------------------------------------------------------------------------------
|
양쪽노드 모두 정상
$ORACLE_HOME/network/admin/sqlnet.ora 파일 수정(양쪽 노드)
|
1
2
|
$ cat $ORACLE_HOME/network/admin/sqlnet.ora
SQLNET.AUTHENTICATION_SERVICES = (NONE)
|
grid 종료(양쪽 노드)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
|
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel19db1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel19db1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel19db1'
CRS-2673: Attempting to stop 'ora.oradb.db' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.chad' on 'oel19db1'
CRS-2677: Stop of 'ora.oradb.db' on 'oel19db1' succeeded
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'oel19db1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel19db1'
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel19db1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.oel19db1.vip' on 'oel19db1'
CRS-2677: Stop of 'ora.oel19db1.vip' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel19db1'
CRS-2677: Stop of 'ora.chad' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'oel19db1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'oel19db1' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'oel19db1' succeeded.
CRS-2672: Attempting to start 'ora.oel19db1.vip' on 'oel19db2'
CRS-2676: Start of 'ora.oel19db1.vip' on 'oel19db2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel19db1'
CRS-2677: Stop of 'ora.ons' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel19db1'
CRS-2677: Stop of 'ora.net1.network' on 'oel19db1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel19db1' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.crf' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel19db1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.crf' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.storage' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel19db1'
CRS-2677: Stop of 'ora.mdnsd' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel19db1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel19db1'
CRS-2677: Stop of 'ora.evmd' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel19db1'
CRS-2677: Stop of 'ora.cssd' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel19db1'
CRS-2677: Stop of 'ora.gipcd' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel19db1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel19db1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
|
grid 기동(양쪽 노드)
|
1
2
|
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
|
잠시 뒤 grid 상태 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.chad
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.net1.network
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.ons
ONLINE ONLINE oel19db1 STABLE
ONLINE ONLINE oel19db2 STABLE
ora.proxy_advm
OFFLINE OFFLINE oel19db1 STABLE
OFFLINE OFFLINE oel19db2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel19db1 STABLE
ora.OCR_VOTE.dg(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE oel19db1 Started,STABLE
2 ONLINE ONLINE oel19db2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE oel19db1 STABLE
2 ONLINE ONLINE oel19db2 STABLE
ora.cvu
1 ONLINE ONLINE oel19db1 STABLE
ora.oel19db1.vip
1 ONLINE ONLINE oel19db1 STABLE
ora.oel19db2.vip
1 ONLINE ONLINE oel19db2 STABLE
ora.oradb.db
1 ONLINE OFFLINE Instance Shutdown,ST
ABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.qosmserver
1 ONLINE ONLINE oel19db1 STABLE
ora.scan1.vip
1 ONLINE ONLINE oel19db1 STABLE
--------------------------------------------------------------------------------
|
grid 프로세스는 정상적으로 올라왔지만 db는 자동으로 기동되지 못함
$ORACLE_HOME/network/admin/sqlnet.ora 파일 설정으로 인해
sqlplus 를 sysdba로 바로 접속하지 못해서 db가 자동으로 기동되지 못한것
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# su - oracle
$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Dec 26 22:14:20 2021
Version 19.12.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
ERROR:
ORA-01017: invalid username/password; logon denied
Enter user-name:
|
$GRID_HOME/network/admin/sqlnet.ora 파일 수정(양쪽 노드)
|
1
2
|
$ cat $GRID_HOME/network/admin/sqlnet.ora
SQLNET.AUTHENTICATION_SERVICES = (NONE)
|
grid 종료(1번 노드)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
|
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel19db1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel19db1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel19db1'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'oel19db1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.chad' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel19db1'
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.cvu' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel19db1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.oel19db1.vip' on 'oel19db1'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.oel19db1.vip' on 'oel19db1' succeeded
CRS-5017: The resource action "ora.DATA.dg stop" encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00108:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/crsd_oraagent_oracle.trc".
CRS-2677: Stop of 'ora.DATA.dg' on 'oel19db1' succeeded
CRS-2679: Attempting to clean 'ora.DATA.dg' on 'oel19db1'
CRS-2681: Clean of 'ora.DATA.dg' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel19db1'
CRS-2677: Stop of 'ora.qosmserver' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel19db1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'oel19db1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'oel19db1' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'oel19db1' succeeded.
CRS-2677: Stop of 'ora.chad' on 'oel19db1' succeeded
CRS-2672: Attempting to start 'ora.qosmserver' on 'oel19db2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel19db2'
CRS-2672: Attempting to start 'ora.cvu' on 'oel19db2'
CRS-2672: Attempting to start 'ora.oel19db1.vip' on 'oel19db2'
CRS-2676: Start of 'ora.cvu' on 'oel19db2' succeeded
CRS-2676: Start of 'ora.oel19db1.vip' on 'oel19db2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel19db2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel19db2'
CRS-2676: Start of 'ora.qosmserver' on 'oel19db2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel19db2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel19db1'
CRS-2677: Stop of 'ora.ons' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel19db1'
CRS-2677: Stop of 'ora.net1.network' on 'oel19db1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel19db1' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.crf' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel19db1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.crf' on 'oel19db1' succeeded
ORA-01017: invalid username/password; logon denied
CRS-2677: Stop of 'ora.mdnsd' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel19db1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel19db1'
CRS-2677: Stop of 'ora.ctssd' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel19db1'
CRS-2677: Stop of 'ora.cssd' on 'oel19db1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel19db1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel19db1'
CRS-2677: Stop of 'ora.gipcd' on 'oel19db1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel19db1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel19db1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
|
grid 종료시 에러가 발생함
grid 종료(2번 노드)
|
1
2
3
4
5
6
7
8
9
|
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel19db2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel19db2'
CRS-4549: Unexpected disconnect while executing shutdown request.
Cluster registry server error. Server error stack -
PROC-00026: Error while accessing the physical storage
- end of server error stackCluster Ready Service aborted due to Oracle Cluster Registry error
[PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices]
[0]]. Details at (:CRSD00111:) in /oracle/app/oracle/diag/crs/oel19db2/crs/trace/crsd.trc.
|
위 메세지에서 더이상 진행되지 않고 hang 걸린것처럼 반응이 없음
2번 노드 grid 종료시 마지막줄에 나온 로그파일 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# tail -f /oracle/app/oracle/diag/crs/oel19db2/crs/trace/crsd.trc
2021-12-26 22:20:51.731*:kgfo.c@1014: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied
2021-12-26 22:20:51.731 : OCRRAW:1995566848: -- trace dump on error exit --
2021-12-26 22:20:51.731 : OCRRAW:1995566848: Error [kgfoAl06] in [kgfokge] at kgfo.c:3180
2021-12-26 22:20:51.731 : OCRRAW:1995566848: ORA-01017: invalid username/password; logon denied
2021-12-26 22:20:51.731 : OCRRAW:1995566848: Category: 7
2021-12-26 22:20:51.731 : OCRRAW:1995566848: DepInfo: 1017
2021-12-26 22:20:51.731 : OCRRAW:1995566848: -- trace dump end --
OCRASM:1995566848: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge
2021-12-26 22:20:51.731 : OCRASM:1995566848: ASM Error Stack : ORA-01017: invalid username/password; logon denied
2021-12-26 22:20:51.731 : OCRASM:1995566848: proprasmo: kgfoCheckMount returned [7]
2021-12-26 22:20:51.731 : OCRASM:1995566848: proprasmo: The ASM instance is down
2021-12-26 22:20:51.731 : OCRRAW:1995566848: proprioo: Failed to open [+OCR_VOTE/oel19db/OCRFILE/registry.255.1086015727]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2021-12-26 22:20:51.731 : OCRRAW:1995566848: proprioo: No OCR/OLR devices are usable
OCRUTL:1995566848: u_fill_errorbuf: Error Info : [Insufficient quorum to open OCR devices]
default:1995566848: u_set_gbl_comp_error: comptype '107' : error '0'
2021-12-26 22:20:51.732 : default:1995566848: clsvactversion:4: Retrieving Active Version from local storage.
2021-12-26 22:20:51.734 : CSSCLNT:1995566848: clssgsgrppubdata: group (ocr_oel19db) not found
2021-12-26 22:20:51.734 : OCRRAW:1995566848: proprio_repairconf: Failed to retrieve the group public data. CSS ret code [20]
2021-12-26 22:20:51.734 : OCRRAW:1995566848: proprioo: Failed to auto repair the OCR configuration.
2021-12-26 22:20:51.734 : OCRRAW:1995566848: proprinit: Could not open raw device
2021-12-26 22:20:51.739 : OCRAPI:1995566848: a_init:16!: Backend init unsuccessful : [26]
2021-12-26 22:20:51.739 : OCRAPI:1995566848: estack 'PROC-00026: Error while accessing the physical storage'
2021-12-26 22:20:51.740 : CRSOCR:1995566848: [ ERROR] OCR context init failure. Error: PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
2021-12-26 22:20:51.741 : CRSD:1995566848: [ NONE] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
2021-12-26 22:20:51.741 : CRSD:1995566848: [ ERROR] [PANIC] CRSD exiting: Could not init OCR, code: 26
2021-12-26 22:20:51.741 : CRSD:1995566848: [ INFO] Done.
|
ORA-01017 등 로그인에 실패했다는 메세지와 OCR 파일 관련 문제가 발생함
grid 강제 종료(2번 노드)
|
1
|
# crsctl stop crs -f
|
grid 기동(양쪽 노드)
|
1
2
|
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
|
grid 상태확인
|
1
2
3
|
# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
|
grid 프로세스 기동문제로 상태가 확인되지 않음
crsctl stst res -t -init 명령으로 grid 상태확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
# crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE oel19db1 STABLE
ora.crf
1 ONLINE ONLINE oel19db1 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE oel19db1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE oel19db1 STABLE
ora.ctssd
1 ONLINE ONLINE oel19db1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE oel19db1 STABLE
ora.evmd
1 ONLINE ONLINE oel19db1 STABLE
ora.gipcd
1 ONLINE ONLINE oel19db1 STABLE
ora.gpnpd
1 ONLINE ONLINE oel19db1 STABLE
ora.mdnsd
1 ONLINE ONLINE oel19db1 STABLE
ora.storage
1 ONLINE OFFLINE oel19db1 STARTING
--------------------------------------------------------------------------------
|
ora.storage가 기동중이고 crsd는 offline 상태임
로그파일 경로 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# cd /oracle/app/oracle/diag/crs/oel19db1/crs/trace
# ls -ltr
.
.
-rw-rw---- 1 oracle dba 4934130 Dec 26 22:42 gipcd.trc
-rw-rw---- 1 root dba 484816 Dec 26 22:42 ohasd_cssdmonitor_root.trm
-rw-rw---- 1 root dba 3244962 Dec 26 22:42 ohasd_cssdmonitor_root.trc
-rw-rw---- 1 root dba 2136792 Dec 26 22:42 ohasd.trm
-rw-rw---- 1 root dba 13679817 Dec 26 22:42 ohasd.trc
-rw-rw---- 1 oracle dba 1156169 Dec 26 22:42 ocssd.trm
-rw-rw---- 1 oracle dba 9207387 Dec 26 22:42 ocssd.trc
-rw-rw---- 1 root dba 2500855 Dec 26 22:42 ohasd_orarootagent_root.trm
-rw-rw---- 1 root dba 16308436 Dec 26 22:42 ohasd_orarootagent_root.trc
|
ohasd_orarootagent_root.trc 파일 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
# vi ohasd_orarootagent_root.trc
2021-12-26 22:43:28.270 : CLSNS:22181632: clsns_SetTraceLevel:trace level set to 1.
2021-12-26 22:43:28.301 : USRTHRD:22181632: [ INFO] {0:5:3} [ora.storage] 9665 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2021-12-26 22:43:28.306 : USRTHRD:22181632: [ INFO] {0:5:3} [ora.storage] 9665 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2021-12-26 22:43:28.341 : USRTHRD:22181632: [ INFO] {0:5:3} [ora.storage] 9383 Error 4 opening dom root in 0x7fbbac257a20
2021-12-26 22:43:28.909 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
.
.
2021-12-26 22:43:31.722 : USRTHRD:22181632: [ INFO] {0:5:3} Thread:kgfoCheckMountExt isRunning is reset to false here
2021-12-26 22:43:31.722 : USRTHRD:22181632: [ INFO] {0:5:3} Thread:kgfoCheckMountExt isFinished set to true
2021-12-26 22:43:31.722 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
2021-12-26 22:43:31.722 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask(kgfoCheckMountExt) executed in 7 seconds
2021-12-26 22:43:31.722 :CLSDYNAM:5371648: [ora.storage]{0:5:3} [start] StorageAgent::parsekgforetcodes retcode = 7, kgfoCheckMount(OCR_VOTE), flag 2
2021-12-26 22:43:31.722 :CLSDYNAM:5371648: [ora.storage]{0:5:3} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 15077, other: ORA-15077: could not locate ASM instance serving a required diskgroup
2021-12-26 22:43:31.722 :CLSDYNAM:5371648: [ora.storage]{0:5:3} [start] StorageAgent::check 260 kgfo returncode 1
2021-12-26 22:43:31.722 :CLSDYNAM:5371648: [ora.storage]{0:5:3} [start] (:CLSN00140:)StorageAgent::check 300 parsekgforretcodes OCR dgName OCR_VOTE state 1
2021-12-26 22:43:31.722 :CLSDYNAM:5371648: [ora.storage]{0:5:3} [start] Storage::start waiting for check to not return PARTIAL or UNPLANNED_OFFLINE 1
2021-12-26 22:43:32.723 :CLSDYNAM:5371648: [ora.storage]{0:5:3} [start] StorageAgent::check 160 NODEROLE_HUB getOCRdetails
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt thread constructor exit this:bc03c2f0 m_pThnd:0 m_thndMX:bc03c310, m_tintMX:bc03c360 &m_postMX:0x7fbbbc03c340
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} TaskletObjManager::manageTasklets START
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt thread destructor entry { this:0x7fbbbc03cd40 m_pThnd:0x7fbbbc0b7e90 m_thndMX:0x7fbbbc03cd60, m_tintMX:0x7fbbbc03cdb0 &m_postMX:0x7fbbbc03cd90
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt thread destructor stop
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt stop { m_pThnd:0x7fbbbc0b7e90 tid:22181632 running:0 alive:0
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread stop:thread is not running:0 or alive:0
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread stop:thread acquire m_pThnd:0x7fbbbc0b7e90 m_thndMX:0x7fbbbc03cd60
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread stop:thread sltstjn
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread::stop release m_thndMX:0x7fbbbc03cd60 m_tid:22181632 thd count:9 }
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt thread destructor m_pThnd:(nil) exit }
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} TaskletObjManager::manageTasklets END
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread::start { acquire thndMX:bc03c310
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread::start spawn pThnd:0x7fbbbc041dc0 thndType:1
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread::start thread spawned tid:22181632 thd count:10
2021-12-26 22:43:32.724 : USRTHRD:5371648: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Thread::start spawned release thndMX:bc03c310 }
2021-12-26 22:43:32.732 : CLSNS:22181632: clsns_SetTraceLevel:trace level set to 1.
2021-12-26 22:43:32.764 : USRTHRD:22181632: [ INFO] {0:5:3} [ora.storage] 9665 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2021-12-26 22:43:32.768 : USRTHRD:22181632: [ INFO] {0:5:3} [ora.storage] 9665 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2021-12-26 22:43:32.802 : USRTHRD:22181632: [ INFO] {0:5:3} [ora.storage] 9383 Error 4 opening dom root in 0x7fbbac0344f0
|
각종 에러 메세지가 발생함
21번 째 줄에 Storage::start waiting 메세지가 발생하고
39~43번 째 줄에 ASM_DISCOVERY_ADDRESS 관련 메세지가 발생함
grid alert.log 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
# vi alert.log
2021-12-26 22:38:06.656 [CSSDMONITOR(62859)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 62859
2021-12-26 22:38:07.036 [CSSDAGENT(62893)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 62893
2021-12-26 22:38:07.959 [OCSSD(62910)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 62910
2021-12-26 22:38:09.018 [OCSSD(62910)]CRS-1713: CSSD daemon is started in hub mode
2021-12-26 22:38:10.255 [OCSSD(62910)]CRS-1707: Lease acquisition for node oel19db1 number 1 completed
2021-12-26 22:38:11.351 [OCSSD(62910)]CRS-1621: The IPMI configuration data for this node stored in the Oracle registry is incomplete; details at (:CSSNK00002:) in /oracle/app/oracle/diag/crs/oel19db1/crs/trace/ocssd.trc
2021-12-26 22:38:11.351 [OCSSD(62910)]CRS-1617: The information required to do node kill for node oel19db1 is incomplete; details at (:CSSNM00004:) in /oracle/app/oracle/diag/crs/oel19db1/crs/trace/ocssd.trc
2021-12-26 22:38:11.355 [OCSSD(62910)]CRS-1605: CSSD voting file is online: /dev/oracleasm/disks/OCR_VOTE3; details in /oracle/app/oracle/diag/crs/oel19db1/crs/trace/ocssd.trc.
2021-12-26 22:38:11.360 [OCSSD(62910)]CRS-1605: CSSD voting file is online: /dev/oracleasm/disks/OCR_VOTE2; details in /oracle/app/oracle/diag/crs/oel19db1/crs/trace/ocssd.trc.
2021-12-26 22:38:11.364 [OCSSD(62910)]CRS-1605: CSSD voting file is online: /dev/oracleasm/disks/OCR_VOTE1; details in /oracle/app/oracle/diag/crs/oel19db1/crs/trace/ocssd.trc.
2021-12-26 22:38:20.620 [OCSSD(62910)]CRS-1601: CSSD Reconfiguration complete. Active nodes are oel19db1 oel19db2 .
2021-12-26 22:38:21.913 [OCSSD(62910)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2021-12-26 22:38:22.423 [OCTSSD(63106)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 63106
2021-12-26 22:38:23.248 [OCTSSD(63106)]CRS-2403: The Cluster Time Synchronization Service on host oel19db1 is in observer mode.
2021-12-26 22:38:24.508 [OCTSSD(63106)]CRS-2407: The new Cluster Time Synchronization Service reference node is host oel19db1.
2021-12-26 22:38:24.510 [OCTSSD(63106)]CRS-2401: The Cluster Time Synchronization Service started on host oel19db1.
2021-12-26 22:38:31.941 [OLOGGERD(63139)]CRS-8500: Oracle Clusterware OLOGGERD process is starting with operating system process ID 63139
2021-12-26 22:38:36.296 [ORAROOTAGENT(62422)]CRS-5019: All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_orarootagent_root.trc".
2021-12-26 22:38:38.542 [ORAAGENT(62534)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_oraagent_oracle.trc"
2021-12-26 22:38:38.571 [ORAAGENT(62534)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_oraagent_oracle.trc"
2021-12-26 22:38:40.530 [ORAAGENT(62534)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_oraagent_oracle.trc"
2021-12-26 22:38:40.530 [ORAAGENT(62534)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_oraagent_oracle.trc"
2021-12-26 22:38:46.619 [ORAAGENT(62534)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_oraagent_oracle.trc"
2021-12-26 22:38:46.620 [ORAAGENT(62534)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/app/oracle/diag/crs/oel19db1/crs/trace/ohasd_oraagent_oracle.trc"
2021-12-26 22:38:51.601 [OHASD(62331)]CRS-2878: Failed to restart resource 'ora.asm'
|
각종 에러 메세지가 발생함
20~26번 째 줄을 보면 asm 리소스의 check 가 실패했다고 나옴
결론
SQLNET.AUTHENTICATION_SERVICES = (NONE) 파라미터를
$ORACLE_HOME/network/admin/sqlnet.ora 에 설정하면 grid 프로세스는 올라오지만 db 가 자동기동되지 못하고
$GRID_HOME/network/admin/sqlnet.ora 에 설정하면 grid 프로세스가 정상적으로 기동되지 못함(asm 등)
$GRID_HOME/network/admin/sqlnet.ora 설정시에는 grid 기동 뿐만아니라 grid 중지시에도 문제가 발생함
감사, 보안 점검 후 조치사항으로 rac 환경에서 해당 파라미터를 추가하라고 하는 경우 적용하지 말고 제외 요청을 해야함
참조 : 2392762.1, 1681849.1
'ORACLE > Rac' 카테고리의 다른 글
| 오라클 19c RAC /etc /opt 등 파일이 제거된 경우 조치방안 (2) | 2022.01.01 |
|---|---|
| 오라클 19c RAC ora.chad 리소스 offline 인 경우 조치방안 (0) | 2021.12.31 |
| 오라클 19c RAC OS 커널 패치시 작업 가이드 (0) | 2021.12.06 |
| 오라클 11g R2 RAC grid 재기동시 db 자동기동 중지 (5) | 2021.10.27 |
| 오라클 19c RAC network interface 변경 후 grid 기동불가 시 해결방법2(gpnptool 이용) (0) | 2021.09.27 |