Connect to RMAN over SQL*Net fails with ORA-27211: Failed to load Media Management Library

Connecting to RMAN works locally (target /) but fails over SQL*Net with "ORA-27211: Failed to load Media Management Library". Media manager software in this case was EMC Data Domain Boost.

Encountered this error when RMAN cloning (DUPLICATE) an RAC target database. Solution was to set the LD_LIBRARY_PATH in the LISTENER.ORA for the target instance to reference the ORACLE_HOME/lib essentially, media manager libraries. If the LD_LIBRARY_PATH is not required to be set for your Media Management software there are other things to also check, read on.

Allocating a channel from a local connection OK:

$ export LD_LIBRARY_PATH=/app/oracle/product/11.2.0/db_1/lib:${LD_LIBRARY_PATH}
$ rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Mon Jun 24 16:52:02 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: REMODB (DBID=3746514001)

RMAN> run
2> {
3> allocate channel ch1 device type sbt TRACE 2 PARMS 
'SBT_LIBRARY=/app/oracle/product/11.2.0/db_1/lib/libddobk.so,BLKSIZE=1048576,ENV=(STORAGE_UNIT=rmanbackups,
BACKUP_HOST=bkdom01,ORACLE_HOME=/app/oracle/product/11.2.0/db_1)';
4> }

using target database control file instead of recovery catalog
allocated channel: ch1
channel ch1: SID=85 instance=REMODB1 device type=SBT_TAPE
channel ch1: Data Domain Boost API
released channel: ch1

Allocate a channel via an SQL*Net connection results in error:

$ rman target sys@remodb

Recovery Manager: Release 11.2.0.3.0 - Production on Mon Jun 24 16:52:56 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

target database Password:
connected to target database: REMODB (DBID=3746514001)

RMAN> run
2> {
3> allocate channel ch1 device type sbt TRACE 2 PARMS 
'SBT_LIBRARY=/app/oracle/product/11.2.0/db_1/lib/libddobk.so,BLKSIZE=1048576,ENV=(STORAGE_UNIT=rmanbackups,
BACKUP_HOST=bkdom01,ORACLE_HOME=/app/oracle/product/11.2.0/db_1)';
4> }

using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ch1 channel at 06/24/2013 16:53:18
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 3977

First check - Process of elimination. Since the target is a RAC instance, connecting to a cluster instance could end up on any node. Eliminate the possbility of inconsistent media manager installations across nodes by creating a tnsnames entry (setup TNS_ADMIN) and connect only to one instance for testing. Then try another instance to test any differences until all the instances are fine.

Second check - Since connections over SQL*Net create a dedicated server process as the same user of the running listener, and in this case the listener was running under Grid Infrastructure (Clusterware) as the 'grid' user, verify the media manager shared objects are readable by the listener "owner", 'grid' in this case under the database product (ORACLE_HOME of target database). You can test by creating a seperate listener for the instance on a different port as the 'oracle' user (or whichever user owns the database product), create a new tnsnames entry (like in the first check) and test. After trying this, in our case the same error still occurred but eliminated another possibility.

Third check - Does LD_LIBRARY_PATH get used over SQL*NET when setting as an environment variable? The ENVS parameter can be set in the LISTENER.ORA for the target instance. As it turns out LD_LIBRARY_PATH set locally does not apply over SQL*Net.

listener.ora

LISTENER_DUP =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = myserver01)(PORT = 1522))
)
)

SID_LIST_LISTENER_DUP =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = REMODB)
(ORACLE_HOME = /app/oracle/product/11.2.0/db_1)
(SID_NAME = REMODB1)
(ENVS="LD_LIBRARY_PATH=/app/oracle/product/11.2.0/db_1/lib")
)
)

oracle@:/home/oracle/dba/keskins/tns_admin$ lsnrctl start listener_dup

LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 05-JUL-2013 15:07:51

Copyright (c) 1991, 2011, Oracle. All rights reserved.

Starting /app/oracle/product/11.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
...
The command completed successfully

oracle@:/home/oracle/dba/keskins/tns_admin$ tnsping remodb_dup
...
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = remodb01)
(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = REMODB)))
OK (30 msec)

Now test:

oracle@:/home/oracle/dba/keskins/tns_admin$ rman target sys@remodb_dup

Recovery Manager: Release 11.2.0.3.0 - Production on Fri Jul 5 15:08:03 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

target database Password:
connected to target database: REMODB (DBID=3746514001)

RMAN> run
2> {
3> allocate channel ch1 device type sbt TRACE 2 PARMS 
'SBT_LIBRARY=/app/oracle/product/11.2.0/db_1/lib/libddobk.so,BLKSIZE=1048576,ENV=(STORAGE_UNIT=rmanbackups,
BACKUP_HOST=bkdom01,ORACLE_HOME=/app/oracle/product/11.2.0/db_1)';
4> }

using target database control file instead of recovery catalog
allocated channel: ch1
channel ch1: SID=118 instance=REMODB1 device type=SBT_TAPE
channel ch1: Data Domain Boost API
released channel: ch1

RMAN> 

Solution - set ENVS with LD_LIBRARY_PATH in LISTENER.ORA for target instance. LD_LIBRARY_PATH does not get loaded over SQL*Net when set locally at the SHELL.

One last thing I came across in testing, loading a dependant media manager library first (libDDBoost.so) which is not a "libobk.so" object, and then libobk library allocated a channel OK. Even without setting LD_LIBRARY_PATH RMAN can load shared objects by force using this workaround - wouldn't call it supported.

RMAN> run
2> {
3> allocate channel ch1 device type sbt TRACE 2 PARMS 
'SBT_LIBRARY=/app/oracle/product/11.2.0/db_1/lib/libDDBoost.so,BLKSIZE=1048576,ENV=(STORAGE_UNIT=rmanbackups,
BACKUP_HOST=bkdom01,ORACLE_HOME=/app/oracle/product/11.2.0/db_1)';
4> }

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ch1 channel at 06/24/2013 16:53:34
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27212: some entrypoints in Media Management Library are missing
SVR4 Error: 2: No such file or directory

RMAN> run
2> {
3> allocate channel ch1 device type sbt TRACE 2 PARMS 
'SBT_LIBRARY=/app/oracle/product/11.2.0/db_1/lib/libddobk.so,BLKSIZE=1048576,ENV=(STORAGE_UNIT=rmanbackups,
BACKUP_HOST=bkdom01,ORACLE_HOME=/app/oracle/product/11.2.0/db_1)';
4> }

allocated channel: ch1
channel ch1: SID=69 instance=REMODB2 device type=SBT_TAPE
channel ch1: Data Domain Boost API
released channel: ch1