-
Title
Error "ld.so.1: sp_cop: fatal: libsporacle.so: open failed", after failover to 2nd node in a Unix -
Description
While failing over to the second node in a Unix cluster environment, the following error is observed when attempting to launch sp_cop:
$ sp_cop &
[1] 19387
[wposbpk-db1:oracle:ora10g] $ ld.so.1: sp_cop: fatal: libsporacle.so: open failed: No such file or directory -
Cause
Can be a result of incorrect steps undertaken when installing Shareplex in a Unix cluster environment resulting in library errors.
-
Resolution
Workaround 1:
The error points to some inconsistency in the way the libraries are linked or the absence of some libraries altogether. The command "ldd -s sp_cop" was issued to find out the library dependencies for sp_cop as follows:
$ ldd -s sp_cop
find object=libsporacle.so; required by sp_cop
search path=/opt/bpk-3/ora10g/lib32:/usr/ccs/lib:/opt/messageq/lib (LD_LIBR
ARY_PATH)
ignore path=/opt/bpk-3/ora10g/lib32 (insecure directory name)
ignore path=/usr/ccs/lib (insecure directory name)
ignore path=/opt/messageq/lib (insecure directory name)
search path=/var/adm/.splex/V-5.3/lib100 (RPATH from file sp_cop)
trying path=/var/adm/.splex/V-5.3/lib100/libsporacle.so
search path=/usr/lib (default)
trying path=/usr/lib/libsporacle.so
libsporacle.so => (file not found).
.
.
It shows that the links to libsporacle.so are missing as do the links for a number of other libraries. On further attempting to probe the hidden directory .splex (that contains links to some Oracle libraries), it is also observed that the directory does not exist in the second node. As a workaround one can copy the directory .splex from the first node, make sure that the soft links to Oracle libraries are created and then add license to the second node and then start Shareplex. The adding of license is required as this is a different node. This should the issue.Workaround 2:
Ideally one should follow the guidelines as given in the Shareplex Installation Guide. The following portions are pertinent to our situation:
Chapter 2 "Unix Installation and Setup" section "Preinstallation Checklist"
SharePlex can be used in a system cluster environment. Within a cluster,
SharePlex uses a central, shared disk that stores the binaries and the replication
environment for the cluster, including the SharePlex data queues and license keys.
During failover, the adoptive node mounts the shared disk so that replication continues.
Create a non-raw disk that can be mounted to all nodes in the cluster as part of the
failover procedure (not the same disk as the Oracle files or data). You will be running
the SharePlex installation script on the primary node of the cluster and installing the
SharePlex product and variable-data directories onto the shared disk.
The SharePlex queues reside in the rim sub-directory of the variable-data directory
and grow dynamically as data is replicated. The maximum potential size of the variable-
data directory determines the size that your shared disk needs to be. For more
information about sizing the variable-data directory, see checklist step 15.
Note that the /var/adm/.splex directory is on your local system. You can either create
this directory on your secondary system or move it to your shared drive, and change
the /var/adm/.splex a symbolic link.Chapter 2 "Unix Installation and Setup" section "Post-installation procedures for UNIX cluster environments"
Creating symbolic links to the /var/adm/.splex directory
The following instructions help you create links from each cluster node to a central
/var/adm/.splex directory, which contains the library files necessary for SharePlex to
run on each node
1 On each node in the cluster, use the following syntax to create a directory named
/var/adm/.splex and then symbolically link this directory to the shared disk where
you installed the SharePlex binaries and directories.
Syntax:
# ln -s /shared_drive/.splex /var/adm/.splexFollowing the above guidelines would ensure that the cluster failover is seamless and at least Shareplex does not cause any additional delay, whether it be a planned or unplanned failover.
Additional information:
The following briefly touches on the advantages of shared libraries as well as what exactly are libraries.
What is the advantage of using shared library:
The single biggest advantage of using shared libraries is smaller executables and hence less disk space requirement.
Metalink note # Note:61628.1 may be very useful to understand what is a shared library. The following are excerpt from that note:
What are Symbols and Libraries?
-----------------------------------When you create a 3GL program, you have external references, global and local functions, and static and global variables. After compiling and producing the object file, you would have external, global, and local symbols. The linker assigns addresses to these variable and function symbols. Therefore, the symbols become names for memory addresses and are derived from the named functions.
Object files are created when you compile your function source files. These functions can be linked with a main program to produce an executable, or bundled into one archived file called a library. So, the library becomes a repository for these object files referencing the defined symbols.
Static vs. Shared Libraries
-------------------------------Static libraries have an extension .a which means the library is archived using the UNIX 'ar' utility. When linking with a static library, a physical copy of the library is attached to the executable program. That is, the object files and the libraries are linked together before calling the executable. This makes your executable size fairly large and may present resource (disk or memory)
concerns.With a shared library, on the other hand, the library is sharable among many versions of different executable files. The entire copy of the library is not attached to the executable. Think of it as a pointer from the executable to the shared library.
The shared library is dynamically linked to an executable at run time. The shared or dynamic link has advantages over the static link. The size of the code to be executed can be considerably smaller. You may gain memory efficiency if many applications are to be run at the same time, and one copy of the shared library is used by all of them.
Regarding determining library dependencies with ldd, one may want to refer to this excellent document by Sandra Henry titled "Viewing library dependencies with ldd ", the link to which is: