Copying files to ASM using ASMCMD

When manually copying a file from the filesystem to ASM using ASMCMD the file will be placed under a generic ASM directory regardless of specifying the destination. Below example a file is copied to +DATA/mynewdb1/ however file this creates an alias to +DATA/ASM/DATAFILE/

ASMCMD> cp /app/oracle/backup/mydb1/DF.2105.2105.784545123 +DATA/mynewdb1/datafile/DF.2105
copying /app/oracle/backup/mydb1/DF.2105.2105.784545123 -> +DATA/mynewdb1/datafile/DF.2105
ASMCMD> cd +DATA/mynewdb1/datafile/
ASMCMD> ls -al
WARNING:option 'a' is deprecated for 'ls'
please use 'absolutepath'
Type Redund Striped Time Sys Name
...
N DF.2105 => +DATA/ASM/DATAFILE/DF.2105.401.784554241
...
ASMCMD> pwd
+DATA/mynewdb1/datafile

According to Oracle this is expected behaviour.

However files transferred to ASM via the database ONNN background process set the context to which to store the files and are placed in the "correct" location. For example using RMAN connected to the target database:

RMAN> restore controlfile from '/app/oracle/backup/db1/Current.312.769340147';
Starting restore at 23-SEP-12
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=242 instance=MYNEWDB1 device type=DISK
channel ORA_DISK_1: copied control file copy
output file name=+DATA/mynewdb1/control01.ctl
output file name=+DATA/mynewdb1/control02.ctl
Finished restore at 23-SEP-12
RMAN>

To copy a file without it ending up being an alias to the generic location use RMAN which will communicate with ASM via the ONNN process and place it in the appropriate location for the database.

11.2 Background Processes "Onnn, ASM Connection Pool Process, Maintains a connection to the ASM instance for metadata operations, Onnn slave processes are spawned on demand. These processes communicate with the ASM instance."

On a semi-related note, why did Oracle consider "ls -absolutepath" was better than "ls -al"? "ls -al" had fewer keystrokes.

ASMCMD> ls -al
WARNING:option 'a' is deprecated for 'ls'
please use 'absolutepath'

ORA-27504: IPC error creating OSD context, ORA-27302: failure occurred at: skgxpcini3

CRS-5017: The resource action "ora.mydb.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:check if failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini3
ORA-27303: additional information: requested interface vnet5:1 interface not up
 _disable_interface_checking = TRUE to disable this check for single instance cluster. 
Check output from ifconfig co

Alert log error for instance:

SKGXP: ospid 10823: network interface with IP address 169.254.20.147 is DOWN

The error message mentions network interface vnet5:1 In this case vnet5 and vnet6 are private interconnects using HAIP hence the 169.254.x.x address (Link Local Address). On SERVER01, vnet5 is UP, no 169. address bound to it:

vnet5: flags=1000843 mtu 9000 index 6
        inet 10.0.84.194 netmask fffffff0 broadcast 10.0.84.207

On SERVER02, vnet5 not UP, has 169.254.20.147 bound to both vnet5 and vnet6!

vnet5: flags=1000842 mtu 9000 index 8
        inet 10.0.84.195 netmask fffffff0 broadcast 10.0.84.207
vnet5:1: flags=1000842 mtu 9000 index 8
        inet 169.254.20.147 netmask ffff8000 broadcast 169.254.127.255
vnet6: flags=1000843 mtu 9000 index 7
        inet 10.0.84.211 netmask fffffff0 broadcast 10.0.84.223
vnet6:1: flags=1000843 mtu 9000 index 7
        inet 169.254.164.136 netmask ffff8000 broadcast 169.254.255.255
vnet6:2: flags=1000843 mtu 9000 index 7
        inet 169.254.20.147 netmask ffff8000 broadcast 169.254.127.255

Fix was to reboot SERVER02 as it turned out the UNIX administrator had performed some work on the blades concerning MAC addresses for vnet5. Somewhere vnet5 and vnet6 got the link-local address mixed up.

ORA-27102: out of memory

The below shows an "ORA-27102 out of memory" error on startup for an Oracle RAC database however this error can occur in single instances also.

$ srvctl start database -d MYDB
PRCR-1079 : Failed to start resource ora.MYDB.db
CRS-5017: The resource action "ora.MYDB.db start" encountered the following error:
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument
. For details refer to "(:CLSN00107:)" in 
"/app/11.2.0/grid/log/myserver01/agent/crsd/oraagent_oracle/oraagent_oracle.log".

In alert log:

WARNING: The system does not seem to be configured
optimally. Creating a segment of size 0x000000009b000000
failed. Please change the shm parameters so that
a segment can be created for this size. While this is
not a fatal issue, creating one segment may improve
performance

9b000000 hex = 2400M = MEMORY_TARGET and MEMORY_MAX_TARGET settings for this particular instance. Check the shared memory max in the project. Databases are being started by the 'grid' user for RAC however single instances will usually be 'oracle':

> projects -l

user.oracle
...
                 project.max-shm-memory=(priv,4294967295,deny)
user.grid
...
                 project.max-shm-memory=(priv,4294967295,deny)

Shared memory maximum allocated to the project is 4G. If the sum of all SGA/PGAs allocated for instances is more than 4G within the project an instance will encounter the error either on startup or even during operation. To fix either increase the maximum shared memory limit (OS resources permitting) or decrease the SGA/PGA in individual instances to under the limit. The following metalink note describes the problem and solutions: Database Startup On Solaris 10 Fails With Ora-27102 Out Of Memory Error [ID 399895.1]