The below shows an "ORA-27102 out of memory" error on startup for an Oracle RAC database however this error can occur in single instances also.
$ srvctl start database -d MYDB
PRCR-1079 : Failed to start resource ora.MYDB.db
CRS-5017: The resource action "ora.MYDB.db start" encountered the following error:
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument
. For details refer to "(:CLSN00107:)" in
"/app/11.2.0/grid/log/myserver01/agent/crsd/oraagent_oracle/oraagent_oracle.log".
In alert log:
WARNING: The system does not seem to be configured
optimally. Creating a segment of size 0x000000009b000000
failed. Please change the shm parameters so that
a segment can be created for this size. While this is
not a fatal issue, creating one segment may improve
performance
9b000000 hex = 2400M = MEMORY_TARGET and MEMORY_MAX_TARGET settings for this particular instance.
Check the shared memory max in the project.
Databases are being started by the 'grid' user for RAC however single instances will usually be 'oracle':
Shared memory maximum allocated to the project is 4G. If the sum of all SGA/PGAs allocated for instances is more than 4G within the project an instance will encounter the error either on startup or even during operation.
To fix either increase the maximum shared memory limit (OS resources permitting) or decrease the SGA/PGA in individual instances to under the limit.
The following metalink note describes the problem and solutions:
Database Startup On Solaris 10 Fails With Ora-27102 Out Of Memory Error [ID 399895.1]
Why does Logical Standby say an applied log is 'CURRENT' instead of 'YES' when the Standby is up to date? Reason is because there's at least one session on the Primary that hasn't yet committed or rolled back. Until a commit or rollback occurs on the Primary the Standby keeps the logs in a 'CURRENT' state and on the Standby server filesystem as they may still be required to apply transactions.
(output below formatted to fit)
column file_name heading 'Archive Log' format a80
column applied heading 'Applied' format a30
select file_name, sequence#, timestamp,
decode(applied,'NO',applied||' - Not Applied Logs!','YES',applied||' - OK',applied) applied
from dba_logstdby_log
order by sequence#;
Archive Log SEQUENCE# TIMESTAMP Applied
-------------------------------------------------------------------------------- ---------- -------------------- ------------------------------
/oradata/MYDB/arch/logapplyarch/arch1_1247463_699468119.log 1247463 10.AUG.2012 16:16:33 CURRENT
/oradata/MYDB/arch/logapplyarch/arch1_1247464_699468119.log 1247464 10.AUG.2012 16:17:04 CURRENT
...
...
The 10.2 documentation doesn't give many details:
"APPLIED, Indicates whether each archive log has been applied (YES) or not (NO)"
To resolve:
On the Primary look for a session that has a lock requested around the time when the first (oldest) 'CURRENT' archive log appeared (16:16:33 in the above).
set linesize 500
column program format a40
column event format a30
column username format a15
column osuser format a15
column lock_requested format a30 heading 'Lock Requested'
column object_name format a30
select
s.sid,
s.serial#,
s.username,
s.osuser,
s.program,
s.event,
l.type,
decode(l.lmode,
0, 'NONE',
1, 'NULL',
2, 'ROW SHARE',
3, 'ROW EXCLUSIVE',
4, 'SHARE',
5, 'SHARE ROW EXCLUSIVE',
6, 'EXCLUSIVE', '?') "Mode",
decode(l.request,
0, 'NONE',
1, 'NULL',
2, 'ROW SHARE',
3, 'ROW EXCLUSIVE',
4, 'SHARE',
5, 'SHARE ROW EXCLUSIVE',
6, 'EXCLUSIVE', '?') "Request",
o.object_name,
to_char(sysdate-l.ctime / 86400,'DD-MON-YYYY HH24:MI:SS') lock_requested
from
v$lock l, dba_objects o, v$session s
where l.id1 = o.object_id (+)
and l.sid = s.sid
order by l.sid, l.type
/
SID SERIAL# USERNAME OSUSER PROGRAM EVENT TY Mode Request OBJECT_NAME Lock Requested
--------- ----------------------- ------ ------------------------- ------------------------------ -- ------------------- ------------- ------------------ -----------------------------
...
113 34978 SCOTT scott prog@tigerserv (TNS V1-V3) SQL*Net message from client TM ROW EXCLUSIVE NONE scott25X 10-AUG-2012 16:16:32
113 34978 SCOTT scott prog@tigerserv (TNS V1-V3) SQL*Net message from client TM ROW EXCLUSIVE NONE scott25J 10-AUG-2012 16:16:32
113 34978 SCOTT scott prog@tigerserv (TNS V1-V3) SQL*Net message from client TM ROW EXCLUSIVE NONE scott25I 10-AUG-2012 16:16:32
113 34978 SCOTT scott prog@tigerserv (TNS V1-V3) SQL*Net message from client TX EXCLUSIVE NONE
...
...
Above there was a DML lock at 16:16:32 on the Primary. Investigate why the session has uncommitted transactions. There is nothing necessarily wrong but if the Standby has been uncharacteristically building up 'CURRENT' applied logs for hours or the filesystem is starting to fill up with shipped archive logs then maybe there is an SQL performance problem or it could be something as simple as a user has not exited their session.
"Error
Connection to {hostname} as user {username} failed."
In the agent home on the host:
AGENT_HOME/sysman/log/emagent.trc:
ERROR util.fileops: error: file AGENT_HOME/bin/nmo is not a setuid file
WARN Authentication: nmo binary in current oraHome doesn't have setuid privileges !!!
ERROR Authentication: altNmo binary doesn't exist ... reverting back to nmo
Check that the root.sh was run when the agent product was installed, if not then UNIX admin need to run root.sh to set the proper permissions for the Oracle binaries. This should be under AGENT_HOME/root.sh
Try and set preferred/default credetials again in Grid Control.
"Information
Credentials successfully verified for {hostname}."
The DBMS_RCVMAN.num2displaysize procedure can be used to display human readable byte sizes.
The DBMS_RCVMAN.Sec2DisplayTime procedure can display a time format given seconds.
The drawback is the DBMS_RCVMAN package is only available to SYS.
SQL> select sys.dbms_rcvman.num2displaysize(1024) sz from dual;
SZ
-------------------------------------------------------------------------------
1.00K
SQL> select sys.dbms_rcvman.num2displaysize(235236346) sz from dual;
SZ
-------------------------------------------------------------------------------
224.34M
SQL> select sys.dbms_rcvman.num2displaysize(1024*1024*1024*1024) sz from dual;
SZ
-------------------------------------------------------------------------------
1.00T
Given a number of seconds display in time format:
SQL> select dbms_rcvman.Sec2DisplayTime(600) dt from dual;
DT
------------------------------
00:10:00
SQL> select dbms_rcvman.Sec2DisplayTime(90035) dt from dual;
DT
------------------------------
25:00:35
SQL>
