Friday, January 7, 2011

HOW TO APPLY ORACLE PATCH (OPATCH)

HOW TO APPLY ORACLE PATCH (OPATCH)
====================================================



To apply Opatch conditions are db and listener both must be down as opatch will update your current ORACLE_HOME with patches.

in single instance its not possible.

but for RAC instance its possible.
as in RAC there will be two seperate oracle home and two seperate instances running once instance on each oracle_home


use this command:

opatch napply -skip_subset -skip_duplicate -local -oh $ORACLE_HOME



when using -local parameter and -oh $ORACLE_HOME this means this patch session will only apply patch to current sourced ORACLE_HOME.



steps before applying patch:
----------------------------

1) check the database status.
wch_db.sql
-----------
select name,
open_mode,
database_name,
created,
log_mode,
platform_name
from v$database;



2) Check the object's invalid.

user_inv.sql
============
SELECT owner,
COUNT(*)
FROM dba_objects
WHERE status = 'INVALID'
GROUP BY owner;


count_inv.sql
-------------
select count(*)
from dba_objects
WHERE status ='INVALID';


3) Take backup of invalid's

create table bk_inv_ as select * from dba_objects
where status='INVALID';



4) check opatch version using
opatch -v
if opatch version is not compatible check the readme file and
download the latest version and uncompress
in $ORACLE_HOME.



5) check oraInst.loc file pointing to your current $ORACLE_HOME or not.
cat /etc/oraInst.loc

inventory_loc=/u01/app/oracle/10.2.0/GlobalOraInv
inst_group=dba



if your server have more then one $ORACLE_HOME then comment the other $ORACLE_HOME and
uncomment the current $ORACLE_HOME

inventory must point to the current $ORACLE_HOME which is getting patched.



6) check free space on $ORACLE_HOME
df -h $ORACLE_HOME



7) chek the utilities like
which ld
which ar
which make
etc as per readme file.



8) unzip the patch
unzip -d /loc_2_unzip p.zip


9) Go the patch directory
cd /loc_2_unzip/patch_number


10) Bring down the listner.
cd $ORACLE_HOME/bin
lsnrctl stop


11) Bring down the database
Shutdown immediate.


12) export opatch
export PATH=$PATH:$HOME:$ORACLE_HOME/OPatch:/bin

13) Start the patch

skip_duplicate
Skips patches to be applied that are duplicates of other patches installed in the Oracle home. Two patches are duplicates if they fix the same set of bugs.


skip_subset
Skips patches to be applied that are subsets of other patches installed in the Oracle home. One patch is a subset of another patch if the former fixes a subset of bugs fixed by the latter.



opatch napply -skip_subset -skip_duplicate

for RAC database then database can be up
as it may be having more then one instance
so you can bring down one instance and listener and apply the patch and open it
and then do the same on another node.
like this db will be up and no user will face issue in outage also.



to apply opatch in RAC instance (We can use -skip_subset -skip_duplicate with Opatch napply only? And is it right that if we are installing patch for the first time on a oracle software we will generally use opatch apply. consider I have applied october cpu patch and I am applying November patch. so while applting Oct patch I shall just use opatch apply and while applying Nov patch i shall use opatch napply -skip_subset -skip_duplicate)

///////////***************************************************


The readme stats whether you should use opatch apply or opatch napply.

I am not sure whether november CPU patch is a bundle of multiple patch or single patch. If it is bundle of multiple patch then you should be using opatch napply irrespective of whether you applied any patch on your software previously. apply and naply does not depends on previous patches that are appled on your database. It just depend onl current opatch session. If you need to install just one patch currently use opatch apply. And if you want to install multiple patch in a single session use opatch napply.


And regarding -skip_subset and -skip_duplicate you are correct it shoudl be used only with opatch napply.



For more info use below command
opatch apply -help
opatch napply -help

**********************************************************

opatch napply -skip_subset -skip_duplicate -local -oh $ORACLE_HOME

when using -local parameter and
-oh $ORACLE_HOME this means this patch session will only apply patch to current ORACLE_HOME only.

--------------------------------------------------------
. All-Node Patch

. Shutdown all Oracle instances on all nodes
. Apply the patch to all nodes
. Bring all nodes up

. Minimum downtime

. Shutdown the Oracle instance on node 1
. Apply the patch to the Oracle instance on node 1
. Shutdown the Oracle instance on node 2
. Apply the patch to the Oracle instance on node 2
. Shutdown the Oracle instance on node 3
. At this point, instances on nodes 1 and 2 can be brought up
. Apply the patch to the Oracle instance on node 3
. Startup the Oracle instance on node 3

. (no downtime)

. Shutdown the Oracle instance on node 1
. Apply the patch to the Oracle instance on node 1
. Start the Oracle instance on node 1
. Shutdown the Oracle instance on node 2
. Apply the patch to the Oracle instance on node 2
. Start the Oracle instance on node 2
. Shutdown the Oracle instance on node 3
. Apply the patch to the Oracle instance on node 3
. Start the Oracle instance on node 3
-------------------------------------------------------------


14) Once patch installation is completed need to do post patching steps.

a) starup the instance
startup
b) Loading modified sqlfiles into the database.
@$ORACLE_HOME/rdbms/admin/catbundle.sql cpu apply

to check the logs generated
catbundle_CPU__APPLY_.log
catbundle_CPU__GENERATE_.log


c) Recompiling Views in the Database
shutdown immediate
startup upgrade
@$ORACLE_HOME/cpu/view_recompile/view_recompile_jan2008cpu.sql
shutdown immediate
startup

If it is a RAC instance.
shutdown
startup nomount
alter database set cluster database=false scope=spfile;
shutdown
startup upgrade
@?/cpu/view_recompile/view_recompile_jan2008cpu.sql
shutdown
startup
alter database set cluster database=true scope=spfile;


restart the database.
cd $CRS_HOME/bin
srvctl start database -d
15) If any invalid objects were reported, run the utlrp.sql script as follows



user_inv.sql
============
SELECT owner,
COUNT(*)
FROM dba_objects
WHERE status = 'INVALID'
GROUP BY owner;

count_inv.sql
-------------
select count(*)
from dba_objects
WHERE status ='INVALID';

if any new invalids seen then again take backup of invalid objects and compile it.

create table bk_inv_ as select * from dba_objects
where status='INVALID';

@?/rdbms/admin/utlrp.sql --- to compile the invalid objects.


16) Confirm that patch has been applied successfully or not at db level also.


post_patch.sql
--------------
col action_time for a40
col action for a15
col namespace for a15
col version for a15
col comments for a40
set pages 1000
set lines 170
select * from registry$history ;



*************************************************************



Output given below as >>>>>


cd $HOME/DBA/patches
unzip p6239052_10203_SOLARIS64.zip
cd 6239052
export OBJECT_MODE=32_64
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch apply

Invoking OPatch 10.2.0.3.0

Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..

Oracle Home : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.3.0
OUI version : 10.2.0.3.0
OUI location : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-17-48AM.log

ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
Invoking fuser to check for active processes.
Invoking fuser on "/u01/app/oracle/product/db10.2.0/bin/oracle"

OPatch detected the node list and the local node from the inventory.
OPatch will patch the local system then propagate the patch to the remote nodes.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'node2' 'node3'
Local node: 'node1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')

Is the local system ready for patching?

Do you want to proceed? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '6239052' for restore. This might take a while...
Backing up files affected by the patch '6239052' for rollback. This might take a while...

Patching component oracle.rdbms, 10.2.0.3.0...
Updating archive file "/u01/app/oracle/product/db10.2.0/lib/libserver10.a"
with "lib/libserver10.a/kjbm.o"
ApplySession adding interim patch '6239052' to inventory

Verifying the update...
Inventory check OK: Patch ID 6239052 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 6239052 are present in Oracle Home.
Running make for target ioracle

The local system has been patched. You can restart Oracle instances on it.

Patching in rolling mode.

Remaining nodes to be patched:
'node2' 'node3'
What is the next node to be patched?
node2
You have selected 'node2' from 'node2' 'node3'

The node 'node2' will be patched next.

Please shutdown Oracle instances running out of this ORACLE_HOME on 'node2'.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')
Is the node ready for patching?

Do you want to proceed? [y|n]
y
User Responded with: Y
Updating nodes 'node2'
Apply-related files are:
FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"

Instantiating the
file "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated"
by replacing $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
with actual path.
Propagating files to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt" with actual path.
Running command on remote node 'node2': cd /u01/app/oracle/product/db10.2.0/rdbms/lib;
/usr/ccs/bin/make -f ins_rdbms.mk
ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2

The node 'node2' has been patched. You can restart Oracle instances on it.

The node 'node3' will be patched next.

Please shutdown Oracle instances running out of this ORACLE_HOME on 'node3'.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')
Is the node ready for patching?

Do you want to proceed? [y|n]
y
User Responded with: Y
Updating nodes 'node3'
Apply-related files are:
FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"

Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file
"/u01/app/oracle/product/db10.2.0/..../rac/make_cmds.txt.instantiated" by replacing
$ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
Running command on remote node 'node3': cd /u01/app/oracle/product/db10.2.0/rdbms/lib;
/usr/ccs/bin/make -f ins_rdbms.mk
ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2

The node 'node3' has been patched. You can restart Oracle instances on it.

There were relinks on remote nodes.
Remember to check the binary size and timestamp on the nodes 'node2' 'node3' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/db10.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk
ioracle $ORACLE_HOME=/u01/app/oracle/product/db10.2.0'

OPatch succeeded.




***********************************




Ensure the new patch appears in the inventory by running the following command:

opatch lsinventory

Invoking OPatch 10.2.0.3.0

Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..

Oracle Home : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.3.0
OUI version : 10.2.0.3.0
OUI location : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/....log

Lsinventory Output file location : /u01/app/oracle/product/db10.2.0/....txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 2 10.2.0.3.0
There are 2 products installed in this Oracle Home.

Interim patches (5) :

Patch 6239052 : applied on Tue Nov 20 02:18:58 GMT 2007
Created on 31 Oct 2007, 08:01:04 hrs PST8PDT
Bugs fixed:
6239052

Patch 4655998 : applied on Fri May 04 10:23:04 BST 2007
Created on 13 Apr 2007, 00:31:22 hrs PST8PDT
Bugs fixed:
4655998

Patch 5749953 : applied on Fri Apr 20 16:13:34 BST 2007
Created on 15 Feb 2007, 02:45:16 hrs US/Pacific
Bugs fixed:
5749953

Patch 5556081 : applied on Mon Apr 16 10:18:22 BST 2007
Created on 9 Nov 2006, 22:20:50 hrs PST8PDT
Bugs fixed:
5556081

Patch 5557962 : applied on Mon Apr 16 10:17:33 BST 2007
Created on 9 Nov 2006, 23:23:06 hrs PST8PDT
Bugs fixed:
4269423, 5557962, 5528974

Rac system comprising of multiple nodes
Local node = node1
Remote node = node2
Remote node = node3

--------------------------------------------------------------------------------

OPatch succeeded




<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>



TROUBLESHOOTING : >>>>

* If the OPatch is NOT applied to the primary node in the cluster then the following error output will be produced:
o ACTION: Restart the patch process on the primary node.

Invoking OPatch 10.2.0.3.0

Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..

Oracle Home : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.3.0
OUI version : 10.2.0.3.0
OUI location : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-16-36AM.log

ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/db10.2.0. Possible #
causes are:
No read or write permission to ORACLE_HOME/.patch_storage
Central Inventory is locked by another OUI instance
No read permission to Central Inventory
The lock file exists in ORACLE_HOME/.patch_storage
The Oracle Home does not exist in Central Inventory

ApplySession failed: ApplySession failed to prepare the system. No detail available.
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73

* If the crs_stat -t output doesn't show all items in an ONLINE state after a suitable amount of time, e.g. 15 mins, the cluster may require restarting completely.
o ACTION: On each node, ONE NODE AT A TIME, use the following commands:

su -

sync;sync;reboot

No comments:

Post a Comment