프린트 하기

OS환경 : Oracle Linux 7.4 (64bit)

 

DB 환경 : Oracle Database 11.2.0.4 RAC

 

방법 : 오라클 11g R2 RAC PSU 패치 가이드

11.2.0.4 기본 설치만 되어있는 2node RAC db에 181016 PSU를 적용하는 내용

롤링 방식으로 1번 노드 패치완료 후 2번 노드도 패치를 진행함

Patch 28429134 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.181016

패치파일 support.oracle.com/ 에서 다운로드 후 진행

#는 root 유저를 $는 oracle 유저를 나타냄

 

 

패치전 백업

Oracle 11g R2 RAC PSU 패치전 백업 가이드(https://positivemh.tistory.com/769) 참조

 

 

기존 패치 내역 확인(1번, 2번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
SQL>
SET LINESIZE 400
COLUMN ACTION_TIME FORMAT A20
COLUMN ACTION FORMAT A20
COLUMN NAMESPACE FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN COMMENTS FORMAT A30
COLUMN BUNDLE_SERIES FORMAT A10
 
SELECT TO_CHAR(ACTION_TIME, 'YYYYMMDD HH24:MI:SS') AS ACTION_TIME
      ,ACTION
      ,NAMESPACE
      ,VERSION
      ,ID
      ,COMMENTS
      ,BUNDLE_SERIES
FROM   DBA_REGISTRY_HISTORY
ORDER BY ACTION_TIME
/
 
ACTION_TIME         ACTION          NAMESPACE           VERSION          ID COMMENTS                BUNDLE_SER
-------------------- -------------------- -------------------- ---------- ---------- ------------------------------ ----------
20180313 02:30:34    APPLY          SERVER           11.2.0.4        0 Patchset 11.2.0.2.0        PSU

db 패치가 적용되어있지 않음

 

 

사전작업

OPatch 버전 확인(1번, 2번 노드)

1
2
3
4
5
6
7
8
9
10
11
GRID OPatch
$ $GRID_HOME/OPatch/opatch version -oh $GRID_HOME
OPatch Version: 11.2.0.3.18
 
OPatch succeeded.
 
DB OPatch
$ $ORACLE_HOME/OPatch/opatch version -oh $ORACLE_HOME
OPatch Version: 11.2.0.3.4
 
OPatch succeeded.

grid opatch 버전은 .18, db opatch 버전은 .4임

 

 

최신 OPatch, PSU 파일 업로드(1번, 2번 노드)

1
2
3
4
5
6
$ ls -al /home/oracle/media/
total 1033124
drwxr-xr-x   2 oracle dba        87 Apr 17 08:10 .
drwxr-xr-x. 10 oracle dba      4096 Apr 17 08:10 ..
-rw-r--r--   1 oracle dba 935171743 Apr 17 08:02 p28429134_112040_Linux-x86-64.zip
-rw-r--r--   1 oracle dba 122739146 Apr 17 08:02 p6880880_112000_Linux-x86-64.zip

 

 

패치파일 unzip(1번, 2번 노드)

1
2
$ cd /home/oracle/media/
$ unzip p28429134_112040_Linux-x86-64.zip

 

 

OPatch 최신버전으로 교체(1번, 2번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
(1번, 2번 노드)
# cd $GRID_HOME
# mv OPatch/ OPatchbak
# unzip /home/oracle/media/p6880880_112000_Linux-x86-64.zip 
# chown -R oracle:dba OPatch/
# $GRID_HOME/OPatch/opatch version -oh $GRID_HOME
OPatch Version: 11.2.0.3.28
OPatch succeeded.
 
DB OPatch
$ cd $ORACLE_HOME
$ mv OPatch/ OPatchbak
$ unzip /home/oracle/media/p6880880_112000_Linux-x86-64.zip
$ $ORACLE_HOME/OPatch/opatch version -oh $ORACLE_HOME
OPatch Version: 11.2.0.3.28
 
OPatch succeeded.

OPatch version : 11.2.0.3.28 로 교체됨

 

 

기존 invalid object 확인(1번, 2번 노드)

1
2
3
4
5
SQL> select count(*from dba_objects where status!='VALID';
 
  COUNT(*)
----------
     0

 

 

db, grid 패치 버전 확인(1번, 2번 노드)

1
2
3
4
5
6
7
8
9
10
11
GRID
$ $GRID_HOME/OPatch/opatch lspatches -oh $GRID_HOME
18370031;Grid Infrastructure Patch Set Update : 11.2.0.4.x (gibugno)
 
OPatch succeeded.
 
DB
$ $ORACLE_HOME/OPatch/opatch lspatches -oh $ORACLE_HOME
There are no Interim patches installed in this Oracle Home "/app/oracle/product/11.2.0/db_1".
 
OPatch succeeded.

 

 

db 종료(1번 노드)

1
2
3
4
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

 

 

* em 이 설치되어있는 경우 em도 종료해야함

1
2
$ $ORACLE_HOME/bin/emctl stop dbconsole
OC4J Configuration issue. /app/oracle/product/11.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_rac1_RAC not found. 

 

 

grid 종료(1번 노드)

(positivemh.tistory.com/716 이슈로 인해 grid의 perl 을 이용함)

Linux 7 버전이 아니라면 /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock 명령만 사용해도됨

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# /app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /app/grid/product/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.ORADATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.ORAFRA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ORADATA.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ORAFRA.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /app/grid/product/11.2.0/grid

 

 

grid 패치(1번 노드)

OCW 패치(1번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
$ $GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local /home/oracle/media/28429134/27735020
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-43-21AM_1.log
 
Verifying environment and performing prerequisite checks...
 
--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...
 
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-43-26AM_1.log
 
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   27735020  
 
Do you want to proceed? [y|n]
y  <-- [y 입력]
User Responded with: Y
All checks passed.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
 
 
Is the local system ready for patching? [y|n]
y  <-- [y 입력]
User Responded with: Y
Backing up files...
Applying interim patch '27735020' to OH '/app/grid/product/11.2.0/grid'
 
Patching component oracle.crs, 11.2.0.4.0...
Patch 27735020 successfully applied.
Sub-set patch [18370031] has become inactive due to the application of a super-set patch [27735020].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-43-26AM_1.log
 
OPatch succeeded.

 

 

ACFS 패치(1번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
$ $GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local /home/oracle/media/28429134/28204939
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-44-49AM_1.log
 
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   28204939  
 
Do you want to proceed? [y|n]
y  <-- [y 입력]
User Responded with: Y
All checks passed.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
 
 
Is the local system ready for patching? [y|n]
y  <-- [y 입력]
User Responded with: Y
Backing up files...
Applying interim patch '28204939' to OH '/app/grid/product/11.2.0/grid'
 
Patching component oracle.usm, 11.2.0.4.0...
Patch 28204939 successfully applied.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-44-49AM_1.log
 
OPatch succeeded.

 

 

DB PSU 패치(1번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ $GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local /home/oracle/media/28429134/28204707
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-45-58AM_1.log
 
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   17478514  18031668  18522509  19121551  19769489  20299013  20760982  21352635  21948347  22502456  23054359  24006111  24732075  25869727  26609445  26392168  26925576  27338049  27734982  28204707  
.
.
Patching component oracle.sqlplus.rsf, 11.2.0.4.0...
 
OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
chmod: changing permissions of ??/app/grid/product/11.2.0/grid/bin/extjobO??: Operation not permitted
make: [iextjob] Error 1 (ignored)
 
 
Composite patch 28204707 successfully applied.
OPatch Session completed with warnings.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_10-45-58AM_1.log
 
OPatch completed with warnings.

에러가 발생했지만 패치가 완료됨

chmod~ 메세지는 버그임 무시해도됨(2265726.1)

 

 

grid 패치 버전 확인(1번 노드)

1
2
3
4
5
6
$ $GRID_HOME/OPatch/opatch lspatches -oh $GRID_HOME
28204707;Database Patch Set Update : 11.2.0.4.181016 (28204707)
28204939;ACFS Patch Set Update : 11.2.0.4.181016 (28204939)
27735020;OCW Patch Set Update : 11.2.0.4.181016 (27735020)
 
OPatch succeeded.

181016 패치(DB PSU, ACFS, OCW)가 정상적으로 적용됨

 

 

db 패치(1번 노드)

패치전 사전 확인(1번 노드)

1
2
/home/oracle/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/home/oracle/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh completed successfully.

정상

 

 

OCW 패치(1번 노드)

명령어 설명 : $ <ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_PSU_number>/<OCW Components_number>/custom/server/<OCW Components_number>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
$ $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /home/oracle/media/28429134/27735020/custom/server/27735020
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-17_10-56-54AM_1.log
 
Verifying environment and performing prerequisite checks...
 
--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...
 
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-17_10-57-00AM_1.log
 
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   27735020  
 
Do you want to proceed? [y|n]
y  <-- [y 입력]
User Responded with: Y
All checks passed.
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/db_1')
 
 
Is the local system ready for patching? [y|n]
y  <-- [y 입력]
User Responded with: Y
Backing up files...
Applying interim patch '27735020' to OH '/app/oracle/product/11.2.0/db_1'
 
Patching component oracle.rdbms, 11.2.0.4.0...
Patch 27735020 successfully applied.
Log file location: /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-17_10-57-00AM_1.log
 
OPatch succeeded.

 

 

DB PSU 패치(1번 노드)

명령어 설명 : $ <ORACLE_HOME>/OPatch/opatch apply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_PSU_number>/<DB_PSU_number>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ $ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local /home/oracle/media/28429134/28204707
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-17_10-58-27AM_1.log
 
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   17478514  18031668  18522509  19121551  19769489  20299013  20760982  21352635  21948347  22502456  23054359  24006111  24732075  25869727  26609445  26392168  26925576  27338049  27734982  28204707  
.
.
/bin/ld: warning: -z nolazyload ignored.
/bin/ld: warning: -z lazyload ignored.
/bin/ld: warning: -z nolazyload ignored.
 
 
Composite patch 28204707 successfully applied.
OPatch Session completed with warnings.
Log file location: /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-17_10-58-27AM_1.log
 
OPatch completed with warnings.

에러가 발생했지만 패치가 완료됨

 

 

패치후 확인(1번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
/home/oracle/media/28429134/27735020/custom/server/27735020/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME
Reading /app/oracle/product/11.2.0/db_1/install/params.ora..
Reading /app/oracle/product/11.2.0/db_1/install/params.ora..
Parsing file /app/oracle/product/11.2.0/db_1/bin/racgwrap
Parsing file /app/oracle/product/11.2.0/db_1/bin/srvctl
Parsing file /app/oracle/product/11.2.0/db_1/bin/srvconfig
Parsing file /app/oracle/product/11.2.0/db_1/bin/cluvfy
Verifying file /app/oracle/product/11.2.0/db_1/bin/racgwrap
Verifying file /app/oracle/product/11.2.0/db_1/bin/srvctl
Verifying file /app/oracle/product/11.2.0/db_1/bin/srvconfig
Verifying file /app/oracle/product/11.2.0/db_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/racgwrap
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/srvctl
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/srvconfig
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/diskmon.bin
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/lsnodes
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/osdbagrp
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/rawutl
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/srvm/admin/ractrans
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/srvm/admin/getcrshome
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/gnsd
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/crsdiag.pl
Postpatch completed successfully

정상

 

 

DB 패치버전 확인(1번 노드)

1
2
3
4
5
$ $ORACLE_HOME/OPatch/opatch lspatches -oh $ORACLE_HOME
28204707;Database Patch Set Update : 11.2.0.4.181016 (28204707)
27735020;OCW Patch Set Update : 11.2.0.4.181016 (27735020)
 
OPatch succeeded.

181016 패치(DB PSU, OCW)가 정상적으로 적용됨

 

 

패치 마무리(rootcrs.pl 스크립트 입력시 grid 기동됨)(1번 노드)

(rootadd_rdbms.sh 는 relink 작업을 해주는 스크립트)

1
2
3
4
5
# $GRID_HOME/rdbms/install/rootadd_rdbms.sh
/app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -patch
Using configuration parameter file: /app/grid/product/11.2.0/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

 

 

db 기동(1번 노드)

1
2
3
4
5
6
7
8
9
10
SQL> startup
ORACLE instance started.
 
Total System Global Area 1686925312 bytes
Fixed Size            2253864 bytes
Variable Size          486542296 bytes
Database Buffers     1191182336 bytes
Redo Buffers            6946816 bytes
Database mounted.
Database opened.

 

 

db 종료(2번 노드)

1
2
3
4
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

 

 

* em 이 설치되어있는 경우 em도 종료해야함

1
2
$ $ORACLE_HOME/bin/emctl stop dbconsole
OC4J Configuration issue. /app/oracle/product/11.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_rac2_RAC not found. 

 

 

grid 종료(2번 노드)

(positivemh.tistory.com/716 이슈로 인해 grid의 perl 을 이용함)

Linux 7 버전이 아니라면 /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock 명령만 사용해도됨

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
/app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /app/grid/product/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.ORADATA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.ORAFRA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2'
CRS-2677: Stop of 'ora.cvu' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac2'
CRS-2677: Stop of 'ora.ORADATA.dg' on 'rac2' succeeded
CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ORAFRA.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.rac2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1'
CRS-2676: Start of 'ora.rac2.vip' on 'rac1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2'
CRS-2677: Stop of 'ora.ons' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2'
CRS-2677: Stop of 'ora.net1.network' on 'rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

 

 

grid 패치(2번 노드)

grid 패치(로그 제외)(2번 노드)

1
2
3
4
5
6
7
8
OCW 패치
$ $GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local /home/oracle/media/28429134/27735020
 
ACFS 패치
$ $GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local /home/oracle/media/28429134/28204939
 
DB PSU 패치
$ $GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local /home/oracle/media/28429134/28204707

 

 

grid 패치 버전 확인(2번 노드)

1
2
3
4
5
6
$ $GRID_HOME/OPatch/opatch lspatches -oh $GRID_HOME
28204707;Database Patch Set Update : 11.2.0.4.181016 (28204707)
28204939;ACFS Patch Set Update : 11.2.0.4.181016 (28204939)
27735020;OCW Patch Set Update : 11.2.0.4.181016 (27735020)
 
OPatch succeeded.

181016 패치(DB PSU, ACFS, OCW)가 정상적으로 적용됨

 

db 패치(2번 노드)

패치전 사전 확인(2번 노드)

1
2
/home/oracle/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/home/oracle/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh completed successfully.

정상

 

 

db 패치(2번 노드)

1
2
3
4
5
6
7
OCW 패치
설명 : $ <ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_PSU_number>/<OCW Components_number>/custom/server/<OCW Components_number>
$ $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /home/oracle/media/28429134/27735020/custom/server/27735020
 
DB PSU 패치
설명 : $ <ORACLE_HOME>/OPatch/opatch apply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_PSU_number>/<DB_PSU_number>
$ $ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local /home/oracle/media/28429134/28204707

 

 

패치후 확인(2번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
/home/oracle/media/28429134/27735020/custom/server/27735020/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME
Reading /app/oracle/product/11.2.0/db_1/install/params.ora..
Reading /app/oracle/product/11.2.0/db_1/install/params.ora..
Parsing file /app/oracle/product/11.2.0/db_1/bin/racgwrap
Parsing file /app/oracle/product/11.2.0/db_1/bin/srvctl
Parsing file /app/oracle/product/11.2.0/db_1/bin/srvconfig
Parsing file /app/oracle/product/11.2.0/db_1/bin/cluvfy
Verifying file /app/oracle/product/11.2.0/db_1/bin/racgwrap
Verifying file /app/oracle/product/11.2.0/db_1/bin/srvctl
Verifying file /app/oracle/product/11.2.0/db_1/bin/srvconfig
Verifying file /app/oracle/product/11.2.0/db_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/racgwrap
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/srvctl
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/srvconfig
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/diskmon.bin
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/lsnodes
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/osdbagrp
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/rawutl
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/srvm/admin/ractrans
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/srvm/admin/getcrshome
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/gnsd
Reapplying file permissions on /app/oracle/product/11.2.0/db_1/bin/crsdiag.pl
Postpatch completed successfully

정상

 

 

DB 패치버전 확인(2번 노드)

1
2
3
4
5
$ $ORACLE_HOME/OPatch/opatch lspatches -oh $ORACLE_HOME
28204707;Database Patch Set Update : 11.2.0.4.181016 (28204707)
27735020;OCW Patch Set Update : 11.2.0.4.181016 (27735020)
 
OPatch succeeded.

181016 패치(DB PSU, OCW)가 정상적으로 적용됨

 

 

패치 마무리(rootcrs.pl 스크립트 입력시 grid 기동됨)(2번 노드)

(rootadd_rdbms.sh 는 relink 작업을 해주는 스크립트)

1
2
3
4
5
# $GRID_HOME/rdbms/install/rootadd_rdbms.sh
/app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -patch
Using configuration parameter file: /app/grid/product/11.2.0/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

 

 

db 기동(2번 노드)

1
2
3
4
5
6
7
8
9
10
SQL> startup
ORACLE instance started.
 
Total System Global Area 1686925312 bytes
Fixed Size            2253864 bytes
Variable Size          486542296 bytes
Database Buffers     1191182336 bytes
Redo Buffers            6946816 bytes
Database mounted.
Database opened.

 

 

수정(패치)된 sql 파일을 db에도 적용 및 invalid object 검사(한쪽 노드에서만)

1
2
SQL> @?/rdbms/admin/catbundle.sql psu apply
SQL> @?/rdbms/admin/utlrp.sql

 

 

패치 내역 확인(1번, 2번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
SQL>
SET LINESIZE 400
COLUMN ACTION_TIME FORMAT A20
COLUMN ACTION FORMAT A20
COLUMN NAMESPACE FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN COMMENTS FORMAT A30
COLUMN BUNDLE_SERIES FORMAT A10
 
SELECT TO_CHAR(ACTION_TIME, 'YYYYMMDD HH24:MI:SS') AS ACTION_TIME
      ,ACTION
      ,NAMESPACE
      ,VERSION
      ,ID
      ,COMMENTS
      ,BUNDLE_SERIES
FROM   DBA_REGISTRY_HISTORY
ORDER BY ACTION_TIME
/
 
ACTION_TIME          ACTION               NAMESPACE            VERSION            ID COMMENTS                       BUNDLE_SER
-------------------- -------------------- -------------------- ---------- ---------- ------------------------------ ----------
20180313 02:30:34    APPLY                SERVER               11.2.0.4            0 Patchset 11.2.0.2.0            PSU
20210417 22:31:31    APPLY                SERVER               11.2.0.4       181016 PSU 11.2.0.4.181016            PSU
 
2 rows selected.

db에도 정상적으로 패치가 적용됨

 

 

invalid object 확인

1
2
3
4
5
SQL> select count(*from dba_objects where status!='VALID';
 
  COUNT(*)
----------
     0

 

 

OPatch 에서도 패치내역 확인

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
GRID
$ $GRID_HOME/OPatch/opatch lsinventory -oh $GRID_HOME
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-04-17_23-05-30PM_1.log
 
Lsinventory Output file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2021-04-17_23-05-30PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
 
Installed Top-level Products (1): 
 
Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.
 
 
Interim patches (3) :
 
Patch  28204707     : applied on Sat Apr 17 10:46:43 EDT 2021
Unique Patch ID:  22491169
Patch description:  "Database Patch Set Update : 11.2.0.4.181016 (28204707)"
   Created on 19 Sep 201803:24:06 hrs PST8PDT
.
.
 
DB
$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-04-17_23-04-47PM_1.log
 
Lsinventory Output file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2021-04-17_23-04-47PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
 
Installed Top-level Products (1): 
 
Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.
 
 
Interim patches (2) :
 
Patch  28204707     : applied on Sat Apr 17 10:59:35 EDT 2021
Unique Patch ID:  22491169
Patch description:  "Database Patch Set Update : 11.2.0.4.181016 (28204707)"
   Created on 19 Sep 201803:24:06 hrs PST8PDT
.
.

 

 

grid, db 상태 확인(1번, 2번 노드)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
ora.OCR_VOTE.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
ora.ORADATA.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
ora.ORAFRA.dg
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
ora.asm
               ONLINE  ONLINE       rac1                     Started             
               ONLINE  ONLINE       rac2                     Started             
ora.gsd
               OFFLINE OFFLINE      rac1                                         
               OFFLINE OFFLINE      rac2                                         
ora.net1.network
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
ora.ons
               ONLINE  ONLINE       rac1                                         
               ONLINE  ONLINE       rac2                                         
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                                         
ora.cvu
      1        ONLINE  ONLINE       rac2                                         
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                         
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                         
ora.racdb.db
      1        ONLINE  ONLINE       rac1                     Open                
      2        ONLINE  ONLINE       rac2                     Open                
ora.scan1.vip
      1        ONLINE  ONLINE       rac2 

모두 정상

 

 

------------------------------

롤링으로 하지않고 인스턴스 다 내리고 동시 진행 방법

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
db 종료(1번, 2번 노드)
SQL> shutdown immediate
 
em 종료
emctl stop dbconsole
 
grid 종료(1번 끝나고 2번 노드)
/app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock
 
ocw 패치(1번 끝나고 2번 노드)
$ $GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local /app/media/28429134/27735020/
 
ACFS 패치(1번 끝나고 2번 노드)
$ $GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local /app/media/28429134/28204939/
 
GRIDHOME DB PSU 패치(1번 끝나고 2번 노드)
$ $GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local /app/media/28429134/28204707
 
GRID 패치 버전 확인(1번, 2번 노드)
$ $GRID_HOME/OPatch/opatch lspatches -oh $GRID_HOME
28204707;Database Patch Set Update : 11.2.0.4.181016 (28204707)
28204939;ACFS Patch Set Update : 11.2.0.4.181016 (28204939)
27735020;OCW Patch Set Update : 11.2.0.4.181016 (27735020)
 
OPatch succeeded.
181016 패치(DB PSU, ACFS, OCW)가 정상적으로 적용됨
 
DB 패치(1번, 2번 노드)
패치전 사전 확인(1번, 2번 노드)
/app/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/app/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh completed successfully.
 
OCW 패치(1번 끝나고 2번 노드)
$ $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /app/media/28429134/27735020/custom/server/27735020
 
DB PSU 패치(1번 끝나고 2번 노드)
$ $ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local /app/media/28429134/28204707
 
패치후 확인(1번, 2번 노드)
/app/media/28429134/27735020/custom/server/27735020/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME
 
DB 패치버전 확인(1번, 2번 노드)
$ $ORACLE_HOME/OPatch/opatch lspatches -oh $ORACLE_HOME
 
패치 마무리(rootcrs.pl 스크립트 입력시 grid 기동됨)(1번 끝나고 2번 노드)
(rootadd_rdbms.sh 는 relink 작업을 해주는 스크립트)
# $GRID_HOME/rdbms/install/rootadd_rdbms.sh
/app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -patch
 
db 기동(1번, 2번 노드)
SQL> startup
 
수정(패치)된 sql 파일을 db에도 적용 및 invalid object 검사(한쪽 노드에서만)
SQL> @?/rdbms/admin/catbundle.sql psu apply
SQL> @?/rdbms/admin/utlrp.sql
 

롤링으로 하지않고 인스턴스 다 내리고 동시 진행 방법

------------------------------

 

 

참조 : 28429134, 28204707 readme 문서, 2217053.1, 2265726.1, 1641136.1

https://positivemh.tistory.com/772

 

Oracle 11g R2 RAC PSU 롤백 가이드

OS환경 : Oracle Linux 7.4 (64bit) DB 환경 : Oracle Database 11.2.0.4 RAC 방법 : Oracle 11g R2 RAC PSU 롤백 가이드 Oracle 11g R2 RAC PSU 패치 가이드(https://positivemh.tistory.com/715) 패치 후..

positivemh.tistory.com

https://positivemh.tistory.com/769

positivemh.tistory.com/175

wisdom16.tistory.com/39

wisdom16.tistory.com/24

blog.naver.com/hanajava/220734801954

blog.goodusdata.com/116

pat98.tistory.com/817

positivemh.tistory.com/716

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwaix/relinking-oracle-grid-infrastructure-for-a-cluster-binaries.html#GUID-11A2832B-A1F1-4454-888F-0CF1CF45EB16

https://positivemh.tistory.com/771

https://positivemh.tistory.com/770