내맘대로긍정이 알려주는
Oracle 23ai 신기능
무료 세미나 발표자료
다운로드
trending_flat
OS환경 : Oracle Linux 6.8 (64bit)
DB 환경 : Oracle Database 11.2.0.4
에러 : CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted
* 가상 환경임
rac 환경 운영 중 1번노드 기동 후 2번노드 기동 시 2번노드의 grid 가 정상 기동되지 않아 확인 하던 중 발견한 에러
1번노드 crsctl stat 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | $ crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE rac1 ora.OCR_VOTE.dg ONLINE ONLINE rac1 ora.ORADATA.dg ONLINE ONLINE rac1 ora.ORAFRA.dg ONLINE ONLINE rac1 ora.asm ONLINE ONLINE rac1 Started ora.gsd OFFLINE OFFLINE rac1 ora.net1.network ONLINE ONLINE rac1 ora.ons ONLINE ONLINE rac1 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac1 ora.cvu 1 ONLINE ONLINE rac1 ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE INTERMEDIATE rac1 FAILED OVER ora.racdb.db 1 OFFLINE OFFLINE Instance Shutdown 2 OFFLINE OFFLINE ora.scan1.vip 1 ONLINE ONLINE rac1 |
2번노드 확인
1 2 3 | $ crsctl stat res -t CRS-4535: Cannot communicate with Cluster Ready Services CRS-4000: Command Status failed, or completed with errors. |
1번노드 -init 옵션 붙여서 확인
*crsctl start crs를 진행 후, start 되고 있는 도중에 확인이 필요한 경우 init 옵션 사용
* 기본적으로 -init 옵션을 붙이지 않으면 crs 데몬들이 완전 구동되지 않은 경우 확인이 불가능함
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | $ crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE ONLINE rac1 Started ora.cluster_interconnect.haip 1 ONLINE ONLINE rac1 ora.crf 1 ONLINE ONLINE rac1 ora.crsd 1 ONLINE ONLINE rac1 ora.cssd 1 ONLINE ONLINE rac1 ora.cssdmonitor 1 ONLINE ONLINE rac1 ora.ctssd 1 ONLINE ONLINE rac1 ACTIVE:0 ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE ONLINE rac1 ora.gipcd 1 ONLINE ONLINE rac1 ora.gpnpd 1 ONLINE ONLINE rac1 ora.mdnsd 1 ONLINE ONLINE rac1 |
2번노드 -init 옵션 붙여서 확인
*crsctl start crs를 진행 후, start 되고 있는 도중에 확인이 필요한 경우 init 옵션 사용
* 기본적으로 -init 옵션을 붙이지 않으면 crs 데몬들이 완전 구동되지 않은 경우 확인이 불가능함
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | $ crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE INTERMEDIATE rac2 OCR not started ora.cluster_interconnect.haip 1 ONLINE ONLINE rac2 ora.crf 1 ONLINE ONLINE rac2 ora.crsd 1 ONLINE OFFLINE ora.cssd 1 ONLINE ONLINE rac2 ora.cssdmonitor 1 ONLINE ONLINE rac2 ora.ctssd 1 ONLINE ONLINE rac2 ACTIVE:0 ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE INTERMEDIATE rac2 ora.gipcd 1 ONLINE ONLINE rac2 ora.gpnpd 1 ONLINE ONLINE rac2 ora.mdnsd 1 ONLINE ONLINE rac2 |
ora.asm이 OCR not started 상태임
ora.crsd와 ora.emvd도 offline 상태임
(diskmon은 엑사 스토리지 관련 리소스여서 1번,2번노드 모두 offline 상태)
2번노드 grid alert 로그 확인
1 2 3 4 5 6 7 | $ tail -f $GRID_HOME/log/rac2/alertrac2.log 2020-06-15 01:04:03.561: [/app/grid/product/11.2.0/grid/bin/oraagent.bin(2044)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/app/grid/product/11.2.0/grid/log/rac2/agent/ohasd/oraagent_oracle/oraagent_oracle.log". 2020-06-15 01:04:33.585: [/app/grid/product/11.2.0/grid/bin/oraagent.bin(2044)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/app/grid/product/11.2.0/grid/log/rac2/agent/ohasd/oraagent_oracle/oraagent_oracle.log". 2020-06-15 01:05:03.600: [/app/grid/product/11.2.0/grid/bin/oraagent.bin(2044)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/app/grid/product/11.2.0/grid/log/rac2/agent/ohasd/oraagent_oracle/oraagent_oracle.log". |
CRS-5019 발생중
모든 OCR 위치가 ASM 디스크 그룹 [OCR_VOTE]에 있으며이 디스크 그룹 중 어느 것도 마운트되지 않았다고 나옴
alert log에 있는 로그 다시 확인
1 2 3 4 5 6 7 8 9 10 11 12 | $ tail -f /app/grid/product/11.2.0/grid/log/rac2/agent/ohasd/oraagent_oracle/oraagent_oracle.log 2020-06-15 01:06:24.643: [ora.asm][2788726528]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184 2020-06-15 01:06:24.643: [ora.asm][2788726528]{0:0:2} [check] clsnUtils::error Exception type=2 string= 2020-06-15 01:06:24.643: [ora.asm][2788726528]{0:0:2} [check] AsmAgent::checkCbk: Exception UserErrorException 2020-06-15 01:06:24.643: [ora.asm][2788726528]{0:0:2} [check] 2020-06-15 01:06:24.643: [ora.asm][2788726528]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 4 current clsagfw_res_status 4 2020-06-15 01:06:25.504: [ora.mdnsd][2788726528]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0 2020-06-15 01:06:25.644: [ora.asm][2788726528]{0:0:2} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 1582 2020-06-15 01:06:25.644: [ora.asm][2788726528]{0:0:2} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1 2020-06-15 01:06:25.645: [ COMMCRS][2786625280]clsc_connect: (0x1897dd0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET)) ... 같은내용 반복중 |
해결 방법 : ASM 디스크 그룹 정상화
ocr 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | 1번 # ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2900 Available space (kbytes) : 259220 ID : 1910076027 Device/File Name : +OCR_VOTE Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded 2번 노드 # ocrcheck PROT-602: Failed to retrieve data from the cluster registry PROC-26: Error while accessing the physical storage |
2번 노드에서 오류가 발생함
physical storage에 접근하지 못했다고 나옴
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | 1번 노드 # crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6b3b255e16d54ffdbfb97e35e1a54cf7 (ORCL:OCR_VOTE1) [OCR_VOTE] 2. ONLINE 09a5df48125c4fc8bf3be326452218bc (ORCL:OCR_VOTE2) [OCR_VOTE] 3. ONLINE 5f29017e8a524f79bf90d4f0580d4a52 (ORCL:OCR_VOTE3) [OCR_VOTE] Located 3 voting disk(s). 2번 노드 # crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. OFFLINE 6b3b255e16d54ffdbfb97e35e1a54cf7 () [] 2. ONLINE 09a5df48125c4fc8bf3be326452218bc (ORCL:OCR_VOTE2) [OCR_VOTE] 3. ONLINE 5f29017e8a524f79bf90d4f0580d4a52 (ORCL:OCR_VOTE3) [OCR_VOTE] Located 3 voting disk(s). |
2번노드의 OCR_VOTE 디스크 1번를 제대로 인식하지 못함
oracleasm listdisks 명령확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | 1번 노드 # oracleasm listdisks ASM01 ASM02 FRA01 OCR_VOTE1 OCR_VOTE2 OCR_VOTE3 2번 노드 # oracleasm listdisks ASM01 ASM02 FRA01 OCR_VOTE2 OCR_VOTE3 |
동일하게 OCR_VOTE 디스크 1번를 제대로 인식하지 못함
2번 노드 서버 종료
vmx 파일 확인
1번 노드의 경우 ocr_vote디스크의 물리적 위치가 ~ocr_vote01.vmdk 로 정상적으로 바라보지만
2번 노드의 경우 ocr_vote디스크의 물리적 위치가 node1.vmdk로 잘못 작성되어 있음
해당부분 정상적으로 수정
서버 기동 후 확인
1번 노드 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | # crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE rac1 OFFLINE OFFLINE rac2 ora.OCR_VOTE.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ORADATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ORAFRA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac1 ora.cvu 1 ONLINE ONLINE rac1 ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.racdb.db 1 OFFLINE OFFLINE Instance Shutdown 2 OFFLINE OFFLINE Instance Shutdown ora.scan1.vip 1 ONLINE ONLINE rac1 |
2번 노드 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | # crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE rac1 OFFLINE OFFLINE rac2 ora.OCR_VOTE.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ORADATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ORAFRA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac1 ora.cvu 1 ONLINE ONLINE rac1 ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.racdb.db 1 OFFLINE OFFLINE Instance Shutdown 2 OFFLINE OFFLINE Instance Shutdown ora.scan1.vip 1 ONLINE ONLINE rac1 |
모두 정상적으로 표시됨
1번노드 -init 옵션 붙여서 확인
*crsctl start crs를 진행 후, start 되고 있는 도중에 확인이 필요한 경우 init 옵션 사용
* 기본적으로 -init 옵션을 붙이지 않으면 crs 데몬들이 완전 구동되지 않은 경우 확인이 불가능함
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE ONLINE rac1 Started ora.cluster_interconnect.haip 1 ONLINE ONLINE rac1 ora.crf 1 ONLINE ONLINE rac1 ora.crsd 1 ONLINE ONLINE rac1 ora.cssd 1 ONLINE ONLINE rac1 ora.cssdmonitor 1 ONLINE ONLINE rac1 ora.ctssd 1 ONLINE ONLINE rac1 ACTIVE:0 ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE ONLINE rac1 ora.gipcd 1 ONLINE ONLINE rac1 ora.gpnpd 1 ONLINE ONLINE rac1 ora.mdnsd 1 ONLINE ONLINE rac1 |
2번노드 -init 옵션 붙여서 확인
*crsctl start crs를 진행 후, start 되고 있는 도중에 확인이 필요한 경우 init 옵션 사용
* 기본적으로 -init 옵션을 붙이지 않으면 crs 데몬들이 완전 구동되지 않은 경우 확인이 불가능함
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE ONLINE rac2 Started ora.cluster_interconnect.haip 1 ONLINE ONLINE rac2 ora.crf 1 ONLINE ONLINE rac2 ora.crsd 1 ONLINE ONLINE rac2 ora.cssd 1 ONLINE ONLINE rac2 ora.cssdmonitor 1 ONLINE ONLINE rac2 ora.ctssd 1 ONLINE ONLINE rac2 ACTIVE:0 ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE ONLINE rac2 ora.gipcd 1 ONLINE ONLINE rac2 ora.gpnpd 1 ONLINE ONLINE rac2 ora.mdnsd 1 ONLINE ONLINE rac2 |
ora.asm, ora.crsd와 ora.emvd 모두 online 상태임
(diskmon은 엑사 스토리지 관련 리소스여서 1번,2번노드 모두 offline 상태)
ocr 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | 1번 노드 # ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2940 Available space (kbytes) : 259180 ID : 1910076027 Device/File Name : +OCR_VOTE Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded 2번 # ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2940 Available space (kbytes) : 259180 ID : 1910076027 Device/File Name : +OCR_VOTE Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded |
votedisk 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | 1번 노드 # crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6b3b255e16d54ffdbfb97e35e1a54cf7 (ORCL:OCR_VOTE1) [OCR_VOTE] 2. ONLINE 09a5df48125c4fc8bf3be326452218bc (ORCL:OCR_VOTE2) [OCR_VOTE] 3. ONLINE 5f29017e8a524f79bf90d4f0580d4a52 (ORCL:OCR_VOTE3) [OCR_VOTE] Located 3 voting disk(s). 2번 노드 # crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6b3b255e16d54ffdbfb97e35e1a54cf7 (ORCL:OCR_VOTE1) [OCR_VOTE] 2. ONLINE 09a5df48125c4fc8bf3be326452218bc (ORCL:OCR_VOTE2) [OCR_VOTE] 3. ONLINE 5f29017e8a524f79bf90d4f0580d4a52 (ORCL:OCR_VOTE3) [OCR_VOTE] Located 3 voting disk(s). |
모두 정상적으로 표시됨
2번 노드 crs alert log 확인
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | 2020-06-15 01:21:05.924: [cssd(2120)]CRS-1713:CSSD daemon is started in clustered mode 2020-06-15 01:21:07.701: [ohasd(1540)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2020-06-15 01:21:07.701: [ohasd(1540)]CRS-2769:Unable to failover resource 'ora.diskmon'. 2020-06-15 01:21:25.539: [cssd(2120)]CRS-1707:Lease acquisition for node rac2 number 2 completed 2020-06-15 01:21:26.768: [cssd(2120)]CRS-1605:CSSD voting file is online: ORCL:OCR_VOTE3; details in /app/grid/product/11.2.0/grid/log/rac2/cssd/ocssd.log. 2020-06-15 01:21:26.773: [cssd(2120)]CRS-1605:CSSD voting file is online: ORCL:OCR_VOTE2; details in /app/grid/product/11.2.0/grid/log/rac2/cssd/ocssd.log. 2020-06-15 01:21:26.792: [cssd(2120)]CRS-1605:CSSD voting file is online: ORCL:OCR_VOTE1; details in /app/grid/product/11.2.0/grid/log/rac2/cssd/ocssd.log. 2020-06-15 01:21:32.135: [cssd(2120)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 . 2020-06-15 01:21:34.229: [ctssd(2245)]CRS-2401:The Cluster Time Synchronization Service started on host rac2. 2020-06-15 01:21:34.229: [ctssd(2245)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1. 2020-06-15 01:21:35.938: [ohasd(1540)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2020-06-15 01:21:35.939: [ohasd(1540)]CRS-2769:Unable to failover resource 'ora.diskmon'. 2020-06-15 01:21:38.238: [ctssd(2245)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time. 2020-06-15 01:21:51.805: [crsd(2353)]CRS-1012:The OCR service started on node rac2. 2020-06-15 01:21:51.821: [evmd(2270)]CRS-1401:EVMD started on node rac2. 2020-06-15 01:21:53.346: [crsd(2353)]CRS-1201:CRSD started on node rac2. |
ocr도 정상적으로 올라오고
evmd, crsd 도 정상적으로 기동됨을 확인할 수 있음
원인 : 물리적 디스크가 제대로 인식되지 못한 문제
물리적으로 디스크가 제대로 인식되지 못한 문제
참조 :
http://junghodata.blogspot.com/2018/01/20180106.html