APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later

Information in this document applies to any platform.

This document has been generated related to NCC analysis

SYMPTOMS

o After applying 11.2.0.4.5GIPSU, oraagent started to leak memory and use high cpu.

o 'ps' and 'svmon' shows the memory used by the process is increasing over the time. The CPU usage is much higher than other processes.

zzz ***Wed May 20 11:00:25 CDT 2015
    USER      PID     PPID PRI  %CPU  %MEM   VSZ   RSS    WCHAN S     ELAPSED        TIME COMMAND
  oracle 13041828        1  60   1.6   9.0 3972632 4278256    * A  5-08:39:08  1-08:26:35 /opt/oracrs/gridsw/grid11204/bin/oraagent.bin
  oracle 12124326        1  60   0.1   1.0 73824   379592     * A  5-08:40:28    01:02:51 /opt/oracrs/gridsw/grid11204/bin/oraagent.bin
....
zzz ***Wed May 20 11:22:27 CDT 2015
    USER      PID     PPID PRI  %CPU  %MEM   VSZ   RSS    WCHAN S     ELAPSED        TIME COMMAND
  oracle 13041828        1  60   1.6   6.0 2552380 2858060    * A  5-09:01:10  1-08:34:41 /opt/oracrs/gridsw/grid11204/bin/oraagent.bin
  oracle 12124326        1  60   0.1   1.0 73824   379592     * A  5-09:02:30    01:03:02 /opt/oracrs/gridsw/grid11204/bin/oraagent.bin

note that oraagent with PID 13041828 (from CRSD) is growing in memory and the cumulative CPU usage is considerable higher  1-08:34:41, compared with the other oraagent with PID 12124326 (from OHASD).

svmon output:

Wed May 20 00:01:08 CDT 2015
     Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
13041828 oraagent.bin   1659692    83538        0  1582356      Y     Y     Y

Wed May 20 11:32:20 CDT 2015
     Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
13041828 oraagent.bin   1781494    83538        0  1703011      Y     Y     Y

o 'svmon' shows the additional memory 'work text data BSS heap' segments created in oraagent.

o The log for oraagent shows the following messages constantly:

InstConnection:connectInt connected
In DedicatedThread::generateLocalListener
DbAgent::DedicatedThread::generateLocalListener getLocalVipAddress
CrsCmd::ClscrsCmdData::stat entity 1 statflag 1 useFilter 0
CrsCmd::ClscrsCmdData::destroy
CrsCmd::ClscrsCmdData::stat entity 1 statflag 1 useFilter 0
CrsCmd::ClscrsCmdData::destroy
CrsCmd::ClscrsCmdData::stat entity 1 statflag 1 useFilter 0
CrsCmd::ClscrsCmdData::destroy
Local VIP address is lsidp4822.co.com
CrsCmd::destroy 
vipVector[0]=lsidp4822.co.com
m_crsHome = /opt/oracrs/gridsw/grid11204
DbAgent::DedicatedThread::generateLocalListener getResNameByTypes
CrsCmd::ClscrsCmdData::destroy
DbAgent::DedicatedThread::generateLocalListener getAttrValueFromResource
CrsCmd::ClscrsCmdData::stat entity 5 statflag 48 useFilter 0
CrsCmd::ClscrsCmdData::destroy
CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
CrsCmd::ClscrsCmdData::destroy
ORACLE_HOME = , START_DEPENDENCIES = hard(type:ora.cluster_vip_net1.type) pullup(type:ora.cluster_vip_net1.type) for ora.LISTENER.lsnr
CrsCmd::destroy

o The log information shows oraagent is checking the local listener, but also reveals ORACLE_HOME is not set.  

CHANGES

Customer updated from 11.2.0.3 to 11.2.0.4 and applied the GIPSU 11.2.0.4.5

CAUSE

Before updating to 11.2.0.4, customer had modified the listener resource, removing the attribute ORACLE_HOME:

crsctl modify res ora.LISTENER.lsnr -attr "ORACLE_HOME= "

 

SOLUTION

Oracle Clusterware resources 'ora.* ' must not be modified, this is an unsupported operation.

For this particular situation, add back the ORACLE_HOME, running:

    

crsctl modify resource ora.LISTENER.lsnr -attr "ORACLE_HOME=%CRS_HOME%"

After setting back the ORACLE_HOME, oraagent stopped using higher memory and cpu.