Oracle 11g R2 RAC PSU 롤백 가이드

OS환경 : Oracle Linux 7.4 (64bit)

 

DB 환경 : Oracle Database 11.2.0.4 RAC

 

방법 : Oracle 11g R2 RAC PSU 롤백 가이드

Oracle 11g R2 RAC PSU 패치 가이드(https://positivemh.tistory.com/715) 패치 후

롤링으로 롤백하는 방법에 대해 설명함

 

 

기존 패치내역 확인

SQL>
SET LINESIZE 400
COLUMN ACTION_TIME FORMAT A20
COLUMN ACTION FORMAT A20
COLUMN NAMESPACE FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN COMMENTS FORMAT A30
COLUMN BUNDLE_SERIES FORMAT A10
 
SELECT TO_CHAR(ACTION_TIME, 'YYYYMMDD HH24:MI:SS') AS ACTION_TIME
      ,ACTION
      ,NAMESPACE
      ,VERSION
      ,ID
      ,COMMENTS
      ,BUNDLE_SERIES
FROM   DBA_REGISTRY_HISTORY
ORDER BY ACTION_TIME
/
 
ACTION_TIME         ACTION          NAMESPACE           VERSION          ID COMMENTS                BUNDLE_SER
-------------------- -------------------- -------------------- ---------- ---------- ------------------------------ ----------
20180313 02:30:34    APPLY          SERVER           11.2.0.4        0 Patchset 11.2.0.2.0        PSU
20211020 07:52:45    APPLY          SERVER           11.2.0.4       181016 PSU 11.2.0.4.181016        PSU
 
2 rows selected.

11.2.0.4 181016 패치가 존재함

 

OPatch 에서도 패치내역 확인

GRID
$ $GRID_HOME/OPatch/opatch lsinventory -oh $GRID_HOME
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-04-13AM_1.log
 
Lsinventory Output file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2021-10-20_09-04-13AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
 
Installed Top-level Products (1): 
 
Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.
 
 
Interim patches (3) :
 
Patch  28204707     : applied on Wed Oct 20 06:15:50 EDT 2021
Unique Patch ID:  22491169
Patch description:  "Database Patch Set Update : 11.2.0.4.181016 (28204707)"
.
.
 
DB
$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-05-33AM_1.log
 
Lsinventory Output file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2021-10-20_09-05-33AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
 
Installed Top-level Products (1): 
 
Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.
 
 
Interim patches (2) :
 
Patch  28204707     : applied on Wed Oct 20 07:08:01 EDT 2021
Unique Patch ID:  22491169
Patch description:  "Database Patch Set Update : 11.2.0.4.181016 (28204707)"
   Created on 19 Sep 2018, 03:24:06 hrs PST8PDT
.
.

 

 

db 종료(1번 노드)

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

 

 

* em 이 설치되어있는 경우 em도 종료해야함

$ $ORACLE_HOME/bin/emctl stop dbconsole
OC4J Configuration issue. /app/oracle/product/11.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_rac1_RAC not found. 

 

 

grid 종료(1번 노드)

(positivemh.tistory.com/716 이슈로 인해 grid의 perl 을 이용함)
Linux 7 버전이 아니라면 /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock 명령만 사용해도됨

# /app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /app/grid/product/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.ORADATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.ORAFRA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.ORADATA.dg' on 'rac1' succeeded
CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ORAFRA.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /app/grid/product/11.2.0/grid

 

 

GRID 패치 롤백(1번 노드)

OCW 패치 롤백(1번 노드)

$ $GRID_HOME/OPatch/opatch rollback -oh $GRID_HOME -local -id 27735020
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-13-53AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
 
--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...
 
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-13-57AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
The following patch(es) will be rolled back: 27735020  
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 27735020...
 
RollbackSession rolling back interim patch '27735020' from OH '/app/grid/product/11.2.0/grid'
 
Patching component oracle.crs, 11.2.0.4.0...
RollbackSession removing interim patch '27735020' from inventory
Inactive sub-set patch [18370031] has become active due to the rolling back of a super-set patch [27735020].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-13-57AM_1.log
 
OPatch succeeded.

 

 

ACFS 패치 롤백(1번 노드)

$ $GRID_HOME/OPatch/opatch rollback -oh $GRID_HOME -local -id 28204939
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-16-00AM_1.log
 
 
Patches will be rolled back in the following order: 
   28204939
The following patch(es) will be rolled back: 28204939  
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 28204939...
 
RollbackSession rolling back interim patch '28204939' from OH '/app/grid/product/11.2.0/grid'
 
Patching component oracle.usm, 11.2.0.4.0...
RollbackSession removing interim patch '28204939' from inventory
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-16-00AM_1.log
 
OPatch succeeded.

 

 

GRIDHOME DB 패치 롤백(1번 노드)

$ $GRID_HOME/OPatch/opatch rollback -oh $GRID_HOME -local -id 28204707
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-16-55AM_1.log
 
 
Patches will be rolled back in the following order: 
   28204707   27734982   27338049   26925576   26392168   26609445   25869727   24732075   24006111   23054359   22502456   21948347   21352635   20760982   20299013   19769489   19121551   18522509   18031668   17478514
The following patch(es) will be rolled back: 28204707  27734982  27338049  26925576  26392168  26609445  25869727  24732075  24006111  23054359  22502456  21948347  21352635  20760982  20299013  19769489  19121551  18522509  18031668  17478514  
Sub-patches of a composite series are being rolled back. The system will be returned to a state where all subpatches are rolled back.
The following bug fixes will be removed: 19429927,18674465,23140259,17019345,25914276,20856766,17551674,22594718,18272672,19891090,26679352,17518652,18604692,17359610,26078387,17279227,26631046,25775213,21842740,18166013,26667032,17551699,21656630,24790914,26039623,17484731,18610915,24975421,25635149,17291347,21063322,17570606,23115139,28254374,17019086,25879984,17019356,14176370,20273319,26245237,19692824,19207156,19718981,13960236,18682983,18740837,17231779,22901797,25634317,20725343,16354467,16410570,18685892,17484762,28364007,25555252,22730454,26243698,27870645,27534509,23725036,24348685,18308268,27825893,20294666,28100487,27374796,27441326,22351572,17392698,24563422,19487147,18604493,18774543,27086138,26910644,21641760,24570598,22809871,22465352,21142837,16756406,27053456,21174504,23003979,20513399,25897615,21795111,26569225,21354456,25764020,20671094,20717359,21629064,18090142,24835538,21380789,21422580,19475971,24717859,18948177,20860659,20598042,21172913,20031873,20324049,23007241,25165496,19174430,20828947,19490948,19285025,24652769,20869721,26746894,26667015,24719736,23315889,20475845,20569094,25654936,21281607,25947799,26474853,20144308,17008068,23294548,25489607,21429602,23713236,18849970,22551446,19272701,24624166,24662775,26667023,17903598,23065323,23026585,21532755,19211433,21132297,26198926,26544823,26575788,22168163,19461270,19777862,26030218,20879889,25423453,14521849,19207117,19578350,25823754,20387265,19469538,25655390,19888853,25600421,13871092,16685417,17365043,19689979,25879656,25957038,16832076,18899974,18259031,18043064,20509482,25427662,25505394,25369547,24842886,25505407,24560906,24411921,25505371,25093656,25505382,22905130,18440095,22243719,22148226,22657942,24316947,6599380,24476265,24476274,24528741,24433711,23330119,22507234,23177648,22380919,22296366,23330124,22507210,23536835,16596890,22502493,19721304,22683212,22353199,20596234,22893153,21756661,18093615,22683225,21756677,21286665,21868720,19258504,21983325,21387964,21343838,21756699,20861693,22195492,22321756,21787056,21330264,17655240,21847223,21517440,22195457,22195465,22195448,22195485,21526048,21668627,22195441,19032867,17551063,21343775,22321741,21972320,21168487,21453153,22092979,21343897,21351877,22195477,16065166,20925795,21538558,21538485,21352646,20448824,21538567,20441797,20004021,21067387,18260550,19699191,18331812,21179898,21051833,20004087,21051852,18440047,20331945,20657441,20334344,20803583,21051840,21051862,20558005,21051858,20067212,20777150,17348614,19788842,16870214,17346091,17082983,17957017,19013183,19358317,18619917,18316692,18317531,17649265,17816865,18856999,17634921,17889549,20296213,17852463,16571443,18000422,18868646,18084625,17201159,16194160,17848897,16903536,16668584,16091637,17798953,16579084,18607546,18193833,18384537,18325460,18430495,13829543,17877323,17811789,14015842,17360606,18456514,18723434,14692762,18362222,18091059,17246576,18384391,18704244,17889583,12982566,20506715,20631274,18331850,12611721,17945983,19393542,17156148,17694209,20506699,18765602,17762296,16306373,20299015,14368995,18762750,18051556,8322815,19915271,18155762,18783224,20506706,17853456,13837378,17853498,16384983,19972566,14034426,19972568,17835048,18471685,19972569,18508861,18244962,17258090,12816846,15990359,19615136,18235390,17274537,18098207,19854503,17922254,15913355,16595641,18334586,17282229,18441944,18315328,19309466,18202441,19972570,19769489,16901385,19680952,18828868,14354737,11883252,18282562,16777840,17936109,12364061,17835627,18641461,18306996,16538760,17912217,18482502,18554763,17184721,20074391,18614015,19197175,16934803,17851160,18092127,18356166,16422541,18436647,18674047,14106803,17302277,20142975,19006849,13558557,17891943,17308789,16344544,17025461,16833527,18273830,18641451,19827973,18228645,18189036,18339044,17215560,18964939,19730508,18135678,19972564,18293054,18747196,17571039,18674024,16042673,14245531,12747740,18264060,19211724,19121551,19727057,17449815,19544839,18673304,19554106,18436307,17982555,17586955,18673342,14657740,18009564,17232014,9756271,19271443,17842825,13955826,17237521,17042658,17040527,18262334,17951233,17267114,17390160,18641419,16494615,18191164,17570240,17787259,17299889,17036973,14764829,17891946,19463893,17587063,19289642,19463897,14084247,18230522,17343514,17824637,18973907,18280813,18328509,17242746,17705023,18673325,16198143,19584068,17006183,11733603,16524926,14285317,18199537,19466309,19458377,14829250,17037130,16692232,17596908,17174582,17648596,17011832,13645875,18096714,17752995,13951456,17655634,16992075,17786518,17393915,13498382,13609098,15861775,16360112,17571306,17477958,16392068,17006570,14458214,14602788,17397545,16268425,17165204,14565184,17883081,18522509,18018515,17672719,16315398,13853126,18554871,17767676,17588480,16542886,17227277,16855292,14054676,17786278,17311728,18247991,18277454,15979965,18681862,18996843,17754782,17892268,17614134,16314254,17478145,18061914,19049453,17238511,17600719,17381384,18744139,17297939,17783588,17716305,16180763,17545847,18203838,17393683,16863422,17389192,17394950,16731148,17761775,10136473,17726838,17186905,17441661,17325413,18203835,13364795,17050888,16472716,17622427,17546761,17468141,17040764,14010183,16837842,16228604,17205719,17284817,17346671,18180390,17235750,14133975,17721717,17082359,17602269,17071721,16613964,16929165,16043574,17080436,17265217,17865671,14852021,16399083,16943711,17375354,17546973,16785708,16956380,18094246,17614227,18031668,17313525,16912439,17385178,16618694,16220077,17016369,18203837,17027426,17341326,18139690,17437634,16875449,17465741,17752121,17344412,17296856,17390431,14338435,17323222,16850630,17551709,13944971,17811447,13866822,17811429,16069901,16721594,17443671,17478514,17612828,17610798,17239687,17501491,17446237,16450169,17811438,17288409,17811456,12905058,17088068,16285691,17332800
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
.
.
OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
chmod: changing permissions of ??/app/grid/product/11.2.0/grid/bin/extjobO??: Operation not permitted
make: [iextjob] Error 1 (ignored)
 
 
OPatch Session completed with warnings.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-16-55AM_1.log
 
OPatch completed with warnings.

 

 

DB 패치 롤백(1번 노드)

패치전 사전 확인(1번 노드)

$ /app/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/app/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh completed successfully.

정상

 

 

OCW 패치 롤백(1번 노드)

$ $ORACLE_HOME/OPatch/opatch rollback -oh $ORACLE_HOME -local -id 27735020
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-22-14AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
 
--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...
 
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-22-19AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
The following patch(es) will be rolled back: 27735020  
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/db_1')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 27735020...
 
RollbackSession rolling back interim patch '27735020' from OH '/app/oracle/product/11.2.0/db_1'
 
Patching component oracle.rdbms, 11.2.0.4.0...
RollbackSession removing interim patch '27735020' from inventory
Log file location: /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-22-19AM_1.log
 
OPatch succeeded.

 

 

DB 패치 롤백(1번 노드)

$ $ORACLE_HOME/OPatch/opatch rollback -oh $ORACLE_HOME -local -id 28204707
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-23-42AM_1.log
 
 
Patches will be rolled back in the following order: 
   28204707   27734982   27338049   26925576   26392168   26609445   25869727   24732075   24006111   23054359   22502456   21948347   21352635   20760982   20299013   19769489   19121551   18522509   18031668   17478514
The following patch(es) will be rolled back: 28204707  27734982  27338049  26925576  26392168  26609445  25869727  24732075  24006111  23054359  22502456  21948347  21352635  20760982  20299013  19769489  19121551  18522509  18031668  17478514  
Sub-patches of a composite series are being rolled back. The system will be returned to a state where all subpatches are rolled back.
The following bug fixes will be removed: 19429927,18674465,23140259,17019345,25914276,20856766,17551674,22594718,18272672,19891090,26679352,17518652,18604692,17359610,26078387,17279227,26631046,25775213,21842740,18166013,26667032,17551699,21656630,24790914,26039623,17484731,18610915,24975421,25635149,17291347,21063322,17570606,23115139,28254374,17019086,25879984,17019356,14176370,20273319,26245237,19692824,19207156,19718981,13960236,18682983,18740837,17231779,22901797,25634317,20725343,16354467,16410570,18685892,17484762,28364007,25555252,22730454,26243698,27870645,27534509,23725036,24348685,18308268,27825893,20294666,28100487,27374796,27441326,22351572,17392698,24563422,19487147,18604493,18774543,27086138,26910644,21641760,24570598,22809871,22465352,21142837,16756406,27053456,21174504,23003979,20513399,25897615,21795111,26569225,21354456,25764020,20671094,20717359,21629064,18090142,24835538,21380789,21422580,19475971,24717859,18948177,20860659,20598042,21172913,20031873,20324049,23007241,25165496,19174430,20828947,19490948,19285025,24652769,20869721,26746894,26667015,24719736,23315889,20475845,20569094,25654936,21281607,25947799,26474853,20144308,17008068,23294548,25489607,21429602,23713236,18849970,22551446,19272701,24624166,24662775,26667023,17903598,23065323,23026585,21532755,19211433,21132297,26198926,26544823,26575788,22168163,19461270,19777862,26030218,20879889,25423453,14521849,19207117,19578350,25823754,20387265,19469538,25655390,19888853,25600421,13871092,16685417,17365043,19689979,25879656,25957038,16832076,18899974,18259031,18043064,20509482,25427662,25505394,25369547,24842886,25505407,24560906,24411921,25505371,25093656,25505382,22905130,18440095,22243719,22148226,22657942,24316947,6599380,24476265,24476274,24528741,24433711,23330119,22507234,23177648,22380919,22296366,23330124,22507210,23536835,16596890,22502493,19721304,22683212,22353199,20596234,22893153,21756661,18093615,22683225,21756677,21286665,21868720,19258504,21983325,21387964,21343838,21756699,20861693,22195492,22321756,21787056,21330264,17655240,21847223,21517440,22195457,22195465,22195448,22195485,21526048,21668627,22195441,19032867,17551063,21343775,22321741,21972320,21168487,21453153,22092979,21343897,21351877,22195477,16065166,20925795,21538558,21538485,21352646,20448824,21538567,20441797,20004021,21067387,18260550,19699191,18331812,21179898,21051833,20004087,21051852,18440047,20331945,20657441,20334344,20803583,21051840,21051862,20558005,21051858,20067212,20777150,17348614,19788842,16870214,17346091,17082983,17957017,19013183,19358317,18619917,18316692,18317531,17649265,17816865,18856999,17634921,17889549,20296213,17852463,16571443,18000422,18868646,18084625,17201159,16194160,17848897,16903536,16668584,16091637,17798953,16579084,18607546,18193833,18384537,18325460,18430495,13829543,17877323,17811789,14015842,17360606,18456514,18723434,14692762,18362222,18091059,17246576,18384391,18704244,17889583,12982566,20506715,20631274,18331850,12611721,17945983,19393542,17156148,17694209,20506699,18765602,17762296,16306373,20299015,14368995,18762750,18051556,8322815,19915271,18155762,18783224,20506706,17853456,13837378,17853498,16384983,19972566,14034426,19972568,17835048,18471685,19972569,18508861,18244962,17258090,12816846,15990359,19615136,18235390,17274537,18098207,19854503,17922254,15913355,16595641,18334586,17282229,18441944,18315328,19309466,18202441,19972570,19769489,16901385,19680952,18828868,14354737,11883252,18282562,16777840,17936109,12364061,17835627,18641461,18306996,16538760,17912217,18482502,18554763,17184721,20074391,18614015,19197175,16934803,17851160,18092127,18356166,16422541,18436647,18674047,14106803,17302277,20142975,19006849,13558557,17891943,17308789,16344544,17025461,16833527,18273830,18641451,19827973,18228645,18189036,18339044,17215560,18964939,19730508,18135678,19972564,18293054,18747196,17571039,18674024,16042673,14245531,12747740,18264060,19211724,19121551,19727057,17449815,19544839,18673304,19554106,18436307,17982555,17586955,18673342,14657740,18009564,17232014,9756271,19271443,17842825,13955826,17237521,17042658,17040527,18262334,17951233,17267114,17390160,18641419,16494615,18191164,17570240,17787259,17299889,17036973,14764829,17891946,19463893,17587063,19289642,19463897,14084247,18230522,17343514,17824637,18973907,18280813,18328509,17242746,17705023,18673325,16198143,19584068,17006183,11733603,16524926,14285317,18199537,19466309,19458377,14829250,17037130,16692232,17596908,17174582,17648596,17011832,13645875,18096714,17752995,13951456,17655634,16992075,17786518,17393915,13498382,13609098,15861775,16360112,17571306,17477958,16392068,17006570,14458214,14602788,17397545,16268425,17165204,14565184,17883081,18522509,18018515,17672719,16315398,13853126,18554871,17767676,17588480,16542886,17227277,16855292,14054676,17786278,17311728,18247991,18277454,15979965,18681862,18996843,17754782,17892268,17614134,16314254,17478145,18061914,19049453,17238511,17600719,17381384,18744139,17297939,17783588,17716305,16180763,17545847,18203838,17393683,16863422,17389192,17394950,16731148,17761775,10136473,17726838,17186905,17441661,17325413,18203835,13364795,17050888,16472716,17622427,17546761,17468141,17040764,14010183,16837842,16228604,17205719,17284817,17346671,18180390,17235750,14133975,17721717,17082359,17602269,17071721,16613964,16929165,16043574,17080436,17265217,17865671,14852021,16399083,16943711,17375354,17546973,16785708,16956380,18094246,17614227,18031668,17313525,16912439,17385178,16618694,16220077,17016369,18203837,17027426,17341326,18139690,17437634,16875449,17465741,17752121,17344412,17296856,17390431,14338435,17323222,16850630,17551709,13944971,17811447,13866822,17811429,16069901,16721594,17443671,17478514,17612828,17610798,17239687,17501491,17446237,16450169,17811438,17288409,17811456,12905058,17088068,16285691,17332800
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/db_1')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 28204707...
.
.
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ins_emagent.mk:113: warning: overriding recipe for target `nmosudo'
ins_emagent.mk:52: warning: ignoring old recipe for target `nmosudo'
/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk:113: warning: overriding recipe for target `nmosudo'
/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk:52: warning: ignoring old recipe for target `nmosudo'
/bin/ld: warning: -z lazyload ignored.
/bin/ld: warning: -z nolazyload ignored.
 
 
OPatch Session completed with warnings.
Log file location: /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-23-42AM_1.log
 
OPatch completed with warnings.

 

 

GRID, DB opatch 패치정보 확인(1번 노드)

$ $GRID_HOME/OPatch/opatch lspatches -oh $GRID_HOME
18370031;Grid Infrastructure Patch Set Update : 11.2.0.4.x (gibugno)
OPatch succeeded.
 
$ $ORACLE_HOME/OPatch/opatch lspatches -oh $ORACLE_HOME
There are no Interim patches installed in this Oracle Home "/app/oracle/product/11.2.0/db_1".
 
OPatch succeeded.

 

 

1번 노드 패치 마무리(rootcrs.pl 스크립트 입력시 grid 기동됨)

(rootadd_rdbms.sh 는 relink 작업을 해주는 스크립트)

# $GRID_HOME/rdbms/install/rootadd_rdbms.sh
# /app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -patch

 

 

db 기동(1번 노드)

SQL> startup
ORACLE instance started.
 
Total System Global Area 1686925312 bytes
Fixed Size            2253864 bytes
Variable Size          486542296 bytes
Database Buffers     1191182336 bytes
Redo Buffers            6946816 bytes
Database mounted.
Database opened.

 

 

db 종료(2번 노드)

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

 

 

* em 이 설치되어있는 경우 em도 종료해야함

$ $ORACLE_HOME/bin/emctl stop dbconsole
OC4J Configuration issue. /app/oracle/product/11.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_rac1_RAC not found. 

 

 

grid 종료(2번 노드)

# /app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /app/grid/product/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.ORADATA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.ORAFRA.dg' on 'rac2'
CRS-2677: Stop of 'ora.cvu' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac2'
CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ORAFRA.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ORADATA.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac1'
CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2676: Start of 'ora.rac2.vip' on 'rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2'
CRS-2677: Stop of 'ora.ons' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2'
CRS-2677: Stop of 'ora.net1.network' on 'rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /app/grid/product/11.2.0/grid

 

 

GRID 패치 롤백(2번 노드)

OCW 패치 롤백(2번 노드)

$ $GRID_HOME/OPatch/opatch rollback -oh $GRID_HOME -local -id 27735020
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-36-58AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
 
--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...
 
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-37-02AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
The following patch(es) will be rolled back: 27735020  
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 27735020...
 
RollbackSession rolling back interim patch '27735020' from OH '/app/grid/product/11.2.0/grid'
 
Patching component oracle.crs, 11.2.0.4.0...
RollbackSession removing interim patch '27735020' from inventory
Inactive sub-set patch [18370031] has become active due to the rolling back of a super-set patch [27735020].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-37-02AM_1.log
 
OPatch succeeded.

 

 

ACFS 패치 롤백(2번 노드)

$ $GRID_HOME/OPatch/opatch rollback -oh $GRID_HOME -local -id 28204939
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-37-39AM_1.log
 
 
Patches will be rolled back in the following order: 
   28204939
The following patch(es) will be rolled back: 28204939  
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 28204939...
 
RollbackSession rolling back interim patch '28204939' from OH '/app/grid/product/11.2.0/grid'
 
Patching component oracle.usm, 11.2.0.4.0...
RollbackSession removing interim patch '28204939' from inventory
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-37-39AM_1.log
 
OPatch succeeded.

 

 

GRIDHOME DB 패치 롤백(2번 노드)

$ $GRID_HOME/OPatch/opatch rollback -oh $GRID_HOME -local -id 28204707
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/grid/product/11.2.0/grid
Central Inventory : /app/oraInventory
   from           : /app/grid/product/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-38-57AM_1.log
 
 
Patches will be rolled back in the following order: 
   28204707   27734982   27338049   26925576   26392168   26609445   25869727   24732075   24006111   23054359   22502456   21948347   21352635   20760982   20299013   19769489   19121551   18522509   18031668   17478514
The following patch(es) will be rolled back: 28204707  27734982  27338049  26925576  26392168  26609445  25869727  24732075  24006111  23054359  22502456  21948347  21352635  20760982  20299013  19769489  19121551  18522509  18031668  17478514  
Sub-patches of a composite series are being rolled back. The system will be returned to a state where all subpatches are rolled back.
The following bug fixes will be removed: 19429927,18674465,23140259,17019345,25914276,20856766,17551674,22594718,18272672,19891090,26679352,17518652,18604692,17359610,26078387,17279227,26631046,25775213,21842740,18166013,26667032,17551699,21656630,24790914,26039623,17484731,18610915,24975421,25635149,17291347,21063322,17570606,23115139,28254374,17019086,25879984,17019356,14176370,20273319,26245237,19692824,19207156,19718981,13960236,18682983,18740837,17231779,22901797,25634317,20725343,16354467,16410570,18685892,17484762,28364007,25555252,22730454,26243698,27870645,27534509,23725036,24348685,18308268,27825893,20294666,28100487,27374796,27441326,22351572,17392698,24563422,19487147,18604493,18774543,27086138,26910644,21641760,24570598,22809871,22465352,21142837,16756406,27053456,21174504,23003979,20513399,25897615,21795111,26569225,21354456,25764020,20671094,20717359,21629064,18090142,24835538,21380789,21422580,19475971,24717859,18948177,20860659,20598042,21172913,20031873,20324049,23007241,25165496,19174430,20828947,19490948,19285025,24652769,20869721,26746894,26667015,24719736,23315889,20475845,20569094,25654936,21281607,25947799,26474853,20144308,17008068,23294548,25489607,21429602,23713236,18849970,22551446,19272701,24624166,24662775,26667023,17903598,23065323,23026585,21532755,19211433,21132297,26198926,26544823,26575788,22168163,19461270,19777862,26030218,20879889,25423453,14521849,19207117,19578350,25823754,20387265,19469538,25655390,19888853,25600421,13871092,16685417,17365043,19689979,25879656,25957038,16832076,18899974,18259031,18043064,20509482,25427662,25505394,25369547,24842886,25505407,24560906,24411921,25505371,25093656,25505382,22905130,18440095,22243719,22148226,22657942,24316947,6599380,24476265,24476274,24528741,24433711,23330119,22507234,23177648,22380919,22296366,23330124,22507210,23536835,16596890,22502493,19721304,22683212,22353199,20596234,22893153,21756661,18093615,22683225,21756677,21286665,21868720,19258504,21983325,21387964,21343838,21756699,20861693,22195492,22321756,21787056,21330264,17655240,21847223,21517440,22195457,22195465,22195448,22195485,21526048,21668627,22195441,19032867,17551063,21343775,22321741,21972320,21168487,21453153,22092979,21343897,21351877,22195477,16065166,20925795,21538558,21538485,21352646,20448824,21538567,20441797,20004021,21067387,18260550,19699191,18331812,21179898,21051833,20004087,21051852,18440047,20331945,20657441,20334344,20803583,21051840,21051862,20558005,21051858,20067212,20777150,17348614,19788842,16870214,17346091,17082983,17957017,19013183,19358317,18619917,18316692,18317531,17649265,17816865,18856999,17634921,17889549,20296213,17852463,16571443,18000422,18868646,18084625,17201159,16194160,17848897,16903536,16668584,16091637,17798953,16579084,18607546,18193833,18384537,18325460,18430495,13829543,17877323,17811789,14015842,17360606,18456514,18723434,14692762,18362222,18091059,17246576,18384391,18704244,17889583,12982566,20506715,20631274,18331850,12611721,17945983,19393542,17156148,17694209,20506699,18765602,17762296,16306373,20299015,14368995,18762750,18051556,8322815,19915271,18155762,18783224,20506706,17853456,13837378,17853498,16384983,19972566,14034426,19972568,17835048,18471685,19972569,18508861,18244962,17258090,12816846,15990359,19615136,18235390,17274537,18098207,19854503,17922254,15913355,16595641,18334586,17282229,18441944,18315328,19309466,18202441,19972570,19769489,16901385,19680952,18828868,14354737,11883252,18282562,16777840,17936109,12364061,17835627,18641461,18306996,16538760,17912217,18482502,18554763,17184721,20074391,18614015,19197175,16934803,17851160,18092127,18356166,16422541,18436647,18674047,14106803,17302277,20142975,19006849,13558557,17891943,17308789,16344544,17025461,16833527,18273830,18641451,19827973,18228645,18189036,18339044,17215560,18964939,19730508,18135678,19972564,18293054,18747196,17571039,18674024,16042673,14245531,12747740,18264060,19211724,19121551,19727057,17449815,19544839,18673304,19554106,18436307,17982555,17586955,18673342,14657740,18009564,17232014,9756271,19271443,17842825,13955826,17237521,17042658,17040527,18262334,17951233,17267114,17390160,18641419,16494615,18191164,17570240,17787259,17299889,17036973,14764829,17891946,19463893,17587063,19289642,19463897,14084247,18230522,17343514,17824637,18973907,18280813,18328509,17242746,17705023,18673325,16198143,19584068,17006183,11733603,16524926,14285317,18199537,19466309,19458377,14829250,17037130,16692232,17596908,17174582,17648596,17011832,13645875,18096714,17752995,13951456,17655634,16992075,17786518,17393915,13498382,13609098,15861775,16360112,17571306,17477958,16392068,17006570,14458214,14602788,17397545,16268425,17165204,14565184,17883081,18522509,18018515,17672719,16315398,13853126,18554871,17767676,17588480,16542886,17227277,16855292,14054676,17786278,17311728,18247991,18277454,15979965,18681862,18996843,17754782,17892268,17614134,16314254,17478145,18061914,19049453,17238511,17600719,17381384,18744139,17297939,17783588,17716305,16180763,17545847,18203838,17393683,16863422,17389192,17394950,16731148,17761775,10136473,17726838,17186905,17441661,17325413,18203835,13364795,17050888,16472716,17622427,17546761,17468141,17040764,14010183,16837842,16228604,17205719,17284817,17346671,18180390,17235750,14133975,17721717,17082359,17602269,17071721,16613964,16929165,16043574,17080436,17265217,17865671,14852021,16399083,16943711,17375354,17546973,16785708,16956380,18094246,17614227,18031668,17313525,16912439,17385178,16618694,16220077,17016369,18203837,17027426,17341326,18139690,17437634,16875449,17465741,17752121,17344412,17296856,17390431,14338435,17323222,16850630,17551709,13944971,17811447,13866822,17811429,16069901,16721594,17443671,17478514,17612828,17610798,17239687,17501491,17446237,16450169,17811438,17288409,17811456,12905058,17088068,16285691,17332800
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid/product/11.2.0/grid')
.
.
Patching component oracle.rdbms.rman, 11.2.0.4.0...
RollbackSession removing interim patch '17478514' from inventory
 
OPatch found the word "error" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
chmod: changing permissions of ??/app/grid/product/11.2.0/grid/bin/extjobO??: Operation not permitted
make: [iextjob] Error 1 (ignored)
 
 
OPatch Session completed with warnings.
Log file location: /app/grid/product/11.2.0/grid/cfgtoollogs/opatch/opatch2021-10-20_09-38-57AM_1.log
 
OPatch completed with warnings.

 

 

DB 패치 롤백(2번 노드)

패치전 사전 확인(2번 노드)

$ /app/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/app/media/28429134/27735020/custom/server/27735020/custom/scripts/prepatch.sh completed successfully.

정상

 

 

OCW 패치 롤백(2번 노드)

$ $ORACLE_HOME/OPatch/opatch rollback -oh $ORACLE_HOME -local -id 27735020
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-41-01AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
 
--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...
 
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-41-06AM_1.log
 
 
Patches will be rolled back in the following order: 
   27735020
The following patch(es) will be rolled back: 27735020  
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/db_1')
 
 
Is the local system ready for patching? [y|n]
y
User Responded with: Y
 
Rolling back patch 27735020...
 
RollbackSession rolling back interim patch '27735020' from OH '/app/oracle/product/11.2.0/db_1'
 
Patching component oracle.rdbms, 11.2.0.4.0...
RollbackSession removing interim patch '27735020' from inventory
Log file location: /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-41-06AM_1.log
 
OPatch succeeded.

 

 

DB 패치 롤백(2번 노드)

$ $ORACLE_HOME/OPatch/opatch rollback -oh $ORACLE_HOME -local -id 28204707
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.
 
 
Oracle Home       : /app/oracle/product/11.2.0/db_1
Central Inventory : /app/oraInventory
   from           : /app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-42-31AM_1.log
 
 
Patches will be rolled back in the following order: 
   28204707   27734982   27338049   26925576   26392168   26609445   25869727   24732075   24006111   23054359   22502456   21948347   21352635   20760982   20299013   19769489   19121551   18522509   18031668   17478514
The following patch(es) will be rolled back: 28204707  27734982  27338049  26925576  26392168  26609445  25869727  24732075  24006111  23054359  22502456  21948347  21352635  20760982  20299013  19769489  19121551  18522509  18031668  17478514  
Sub-patches of a composite series are being rolled back. The system will be returned to a state where all subpatches are rolled back.
The following bug fixes will be removed: 19429927,18674465,23140259,17019345,25914276,20856766,17551674,22594718,18272672,19891090,26679352,17518652,18604692,17359610,26078387,17279227,26631046,25775213,21842740,18166013,26667032,17551699,21656630,24790914,26039623,17484731,18610915,24975421,25635149,17291347,21063322,17570606,23115139,28254374,17019086,25879984,17019356,14176370,20273319,26245237,19692824,19207156,19718981,13960236,18682983,18740837,17231779,22901797,25634317,20725343,16354467,16410570,18685892,17484762,28364007,25555252,22730454,26243698,27870645,27534509,23725036,24348685,18308268,27825893,20294666,28100487,27374796,27441326,22351572,17392698,24563422,19487147,18604493,18774543,27086138,26910644,21641760,24570598,22809871,22465352,21142837,16756406,27053456,21174504,23003979,20513399,25897615,21795111,26569225,21354456,25764020,20671094,20717359,21629064,18090142,24835538,21380789,21422580,19475971,24717859,18948177,20860659,20598042,21172913,20031873,20324049,23007241,25165496,19174430,20828947,19490948,19285025,24652769,20869721,26746894,26667015,24719736,23315889,20475845,20569094,25654936,21281607,25947799,26474853,20144308,17008068,23294548,25489607,21429602,23713236,18849970,22551446,19272701,24624166,24662775,26667023,17903598,23065323,23026585,21532755,19211433,21132297,26198926,26544823,26575788,22168163,19461270,19777862,26030218,20879889,25423453,14521849,19207117,19578350,25823754,20387265,19469538,25655390,19888853,25600421,13871092,16685417,17365043,19689979,25879656,25957038,16832076,18899974,18259031,18043064,20509482,25427662,25505394,25369547,24842886,25505407,24560906,24411921,25505371,25093656,25505382,22905130,18440095,22243719,22148226,22657942,24316947,6599380,24476265,24476274,24528741,24433711,23330119,22507234,23177648,22380919,22296366,23330124,22507210,23536835,16596890,22502493,19721304,22683212,22353199,20596234,22893153,21756661,18093615,22683225,21756677,21286665,21868720,19258504,21983325,21387964,21343838,21756699,20861693,22195492,22321756,21787056,21330264,17655240,21847223,21517440,22195457,22195465,22195448,22195485,21526048,21668627,22195441,19032867,17551063,21343775,22321741,21972320,21168487,21453153,22092979,21343897,21351877,22195477,16065166,20925795,21538558,21538485,21352646,20448824,21538567,20441797,20004021,21067387,18260550,19699191,18331812,21179898,21051833,20004087,21051852,18440047,20331945,20657441,20334344,20803583,21051840,21051862,20558005,21051858,20067212,20777150,17348614,19788842,16870214,17346091,17082983,17957017,19013183,19358317,18619917,18316692,18317531,17649265,17816865,18856999,17634921,17889549,20296213,17852463,16571443,18000422,18868646,18084625,17201159,16194160,17848897,16903536,16668584,16091637,17798953,16579084,18607546,18193833,18384537,18325460,18430495,13829543,17877323,17811789,14015842,17360606,18456514,18723434,14692762,18362222,18091059,17246576,18384391,18704244,17889583,12982566,20506715,20631274,18331850,12611721,17945983,19393542,17156148,17694209,20506699,18765602,17762296,16306373,20299015,14368995,18762750,18051556,8322815,19915271,18155762,18783224,20506706,17853456,13837378,17853498,16384983,19972566,14034426,19972568,17835048,18471685,19972569,18508861,18244962,17258090,12816846,15990359,19615136,18235390,17274537,18098207,19854503,17922254,15913355,16595641,18334586,17282229,18441944,18315328,19309466,18202441,19972570,19769489,16901385,19680952,18828868,14354737,11883252,18282562,16777840,17936109,12364061,17835627,18641461,18306996,16538760,17912217,18482502,18554763,17184721,20074391,18614015,19197175,16934803,17851160,18092127,18356166,16422541,18436647,18674047,14106803,17302277,20142975,19006849,13558557,17891943,17308789,16344544,17025461,16833527,18273830,18641451,19827973,18228645,18189036,18339044,17215560,18964939,19730508,18135678,19972564,18293054,18747196,17571039,18674024,16042673,14245531,12747740,18264060,19211724,19121551,19727057,17449815,19544839,18673304,19554106,18436307,17982555,17586955,18673342,14657740,18009564,17232014,9756271,19271443,17842825,13955826,17237521,17042658,17040527,18262334,17951233,17267114,17390160,18641419,16494615,18191164,17570240,17787259,17299889,17036973,14764829,17891946,19463893,17587063,19289642,19463897,14084247,18230522,17343514,17824637,18973907,18280813,18328509,17242746,17705023,18673325,16198143,19584068,17006183,11733603,16524926,14285317,18199537,19466309,19458377,14829250,17037130,16692232,17596908,17174582,17648596,17011832,13645875,18096714,17752995,13951456,17655634,16992075,17786518,17393915,13498382,13609098,15861775,16360112,17571306,17477958,16392068,17006570,14458214,14602788,17397545,16268425,17165204,14565184,17883081,18522509,18018515,17672719,16315398,13853126,18554871,17767676,17588480,16542886,17227277,16855292,14054676,17786278,17311728,18247991,18277454,15979965,18681862,18996843,17754782,17892268,17614134,16314254,17478145,18061914,19049453,17238511,17600719,17381384,18744139,17297939,17783588,17716305,16180763,17545847,18203838,17393683,16863422,17389192,17394950,16731148,17761775,10136473,17726838,17186905,17441661,17325413,18203835,13364795,17050888,16472716,17622427,17546761,17468141,17040764,14010183,16837842,16228604,17205719,17284817,17346671,18180390,17235750,14133975,17721717,17082359,17602269,17071721,16613964,16929165,16043574,17080436,17265217,17865671,14852021,16399083,16943711,17375354,17546973,16785708,16956380,18094246,17614227,18031668,17313525,16912439,17385178,16618694,16220077,17016369,18203837,17027426,17341326,18139690,17437634,16875449,17465741,17752121,17344412,17296856,17390431,14338435,17323222,16850630,17551709,13944971,17811447,13866822,17811429,16069901,16721594,17443671,17478514,17612828,17610798,17239687,17501491,17446237,16450169,17811438,17288409,17811456,12905058,17088068,16285691,17332800
 
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/db_1')
.
.
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ins_emagent.mk:113: warning: overriding recipe for target `nmosudo'
ins_emagent.mk:52: warning: ignoring old recipe for target `nmosudo'
/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk:113: warning: overriding recipe for target `nmosudo'
/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk:52: warning: ignoring old recipe for target `nmosudo'
/bin/ld: warning: -z lazyload ignored.
/bin/ld: warning: -z nolazyload ignored.
 
 
OPatch Session completed with warnings.
Log file location: /app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2021-10-20_09-42-31AM_1.log
 
OPatch completed with warnings.

 

 

GRID, DB opatch 패치정보 확인(2번 노드)

$ $GRID_HOME/OPatch/opatch lspatches -oh $GRID_HOME
18370031;Grid Infrastructure Patch Set Update : 11.2.0.4.x (gibugno)
 
OPatch succeeded.
 
$ $ORACLE_HOME/OPatch/opatch lspatches -oh $ORACLE_HOME
There are no Interim patches installed in this Oracle Home "/app/oracle/product/11.2.0/db_1".
 
OPatch succeeded.

 

 

2번 노드 패치 마무리(rootcrs.pl 스크립트 입력시 grid 기동됨)

(rootadd_rdbms.sh 는 relink 작업을 해주는 스크립트)

# $GRID_HOME/rdbms/install/rootadd_rdbms.sh
# /app/grid/product/11.2.0/grid/perl/bin/perl /app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -patch
 

 

 

db 기동(2번 노드)

SQL> startup
ORACLE instance started.
 
Total System Global Area 1686925312 bytes
Fixed Size            2253864 bytes
Variable Size          486542296 bytes
Database Buffers     1191182336 bytes
Redo Buffers            6946816 bytes
Database mounted.
Database opened.

 

 

롤백 sql 실행

수정(롤백)된 sql 파일을 db에도 적용 및 invalid object 검사(한쪽 노드에서만)

SQL> @?/rdbms/admin/catbundle_PSU_RACDB_ROLLBACK.sql
SQL> @?/rdbms/admin/utlrp.sql

 

 

롤백 내역 확인

SQL>
SET LINESIZE 400
COLUMN ACTION_TIME FORMAT A20
COLUMN ACTION FORMAT A20
COLUMN NAMESPACE FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN COMMENTS FORMAT A30
COLUMN BUNDLE_SERIES FORMAT A10
 
SELECT TO_CHAR(ACTION_TIME, 'YYYYMMDD HH24:MI:SS') AS ACTION_TIME
      ,ACTION
      ,NAMESPACE
      ,VERSION
      ,ID
      ,COMMENTS
      ,BUNDLE_SERIES
FROM   DBA_REGISTRY_HISTORY
ORDER BY ACTION_TIME
/
 
ACTION_TIME         ACTION          NAMESPACE           VERSION          ID COMMENTS                BUNDLE_SER
-------------------- -------------------- -------------------- ---------- ---------- ------------------------------ ----------
20180313 02:30:34    APPLY          SERVER           11.2.0.4        0 Patchset 11.2.0.2.0        PSU
20211020 07:52:45    APPLY          SERVER           11.2.0.4       181016 PSU 11.2.0.4.181016        PSU
20211020 09:51:33    ROLLBACK          SERVER           11.2.0.4       181016 PSU 11.2.0.4.181016        PSU
20211020 09:51:33    APPLY          SERVER           11.2.0.4        0 Patchset 11.2.0.2.0        PSU
 
4 rows selected.

정상적으로 롤백됨

 

 

참조 : 28429134, 28204707 readme 문서, 2217053.1, 2265726.1, 1641136.1

https://positivemh.tistory.com/715

Oracle 11g R2 RAC PSU 패치 가이드

https://positivemh.tistory.com/769

positivemh.tistory.com/175

wisdom16.tistory.com/39

wisdom16.tistory.com/24

blog.naver.com/hanajava/220734801954

blog.goodusdata.com/116

pat98.tistory.com/817

positivemh.tistory.com/716

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwaix/relinking-oracle-grid-infrastructure-for-a-cluster-binaries.html#GUID-11A2832B-A1F1-4454-888F-0CF1CF45EB16

https://positivemh.tistory.com/771

https://positivemh.tistory.com/770