NVBU fails to start on Solaris Global Zone - "Fatal error during address bind (Port 20032)" (4034849)

NVBU fails to start on Solaris Global Zone - "Fatal error during address bind (Port 20032)"

NVBU service on a Solaris 10 client is unable to start.
Trace shows NetVault Process Manager (NVPMGR) spawns its first process Communication Manager (NVCMGR), but receives an "unexpected network error" when it tries to talk to it via the loopback address 127.0.0.1 on port 20032. Port 20032 is however free.
Since NetVault cannot communicate with its Start of the Day processes (SOD), it stops.

Trace shows:

NVPMGR.log:
2 PMGR    :5338  142   0 142222 Started process 2
4 NET     :5338   53   0 142222 Connect to 7f000001:20032
0 MESSAGE :5338  261   1 142222 Unexpected network error
0 PMGR    :5338  112   0 142222 Failed to connect process manager to comms manager

NVCMGR.log:
0 NET     :5343   17 126 142222 Fatal error during address bind: Cannot assign requested address
0 CMGRLOOP:5343   42   0 142222 Failed to bind address to listen socket
0 CMGRLOOP:5343   24   0 142222 Failed to initialize communications manager
0 CMGR    :5343   25   0 142222 Failed to start message loop

ifconfig -a shows multiple loopback interfaces:

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
       inet 10.10.13.49 netmask ff000000
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
       zone dark.zone
       inet 127.0.0.1 netmask ff000000
lo0:2: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
       zone basin.zone
       inet 10.10.13.49 netmask ff000000

The loopback interface NetVault needs to connect to for IPCs with its processes, is the one corresponding to 127.0.0.1 (hexadecimal value of 7f000001).
On a normal setup 127.0.0.1 is typically bound to lo0 on Solaris, but in this example it is bound to the lo0:1 adapter from the zone called 'dark'.
When Solaris Zones (aka Containers) are in use, you may endup with multitple loopback interfaces and if configured improperly, they may cause issues with NVBU starting.

1- If the Solaris Zones/Containers feature is in use, ensure NVBU has been installed in the Global Zone.
Installing NVBU in a non-global zone is not supported.

2- Is port 20032 really in use?  Run 'netstat -a' to verify whether port 20032 is in use. If in use you will have to determine which process is holding this port so you can stop it for use by NVBU. The process of mapping a port to a process in Solaris may not be easy. LSOF can help but this program may not be installed by default on Solaris 10. The following link offers a useful script which will map port to PID and list the result http://www.sybaseteam.com/lsof-alternative-in-solaris-t-339.html

3- If it was determined that port 20032 is free for use by NVBU, then you will have to investigate possible issues with the loopback interface.  Run 'ifconfig -a' to check how many loopback interfaces the machine has. The interface names should all start with 'lo0:x', 'lo0' being the first one.

3- If the machine has multiple loopback interfaces, ensure only 'lo0' is bound to 127.0.0.1 as this is the interface and address expected by NVBU.
If another loopback interface (other than 'lo0') already is bound to 127.0.0.1, then that alternate loopback interface must be taken down so 127.0.01 is assigned to lo0.
a) Take both loopback interfaces down:
example:
ifconfig lo0 down
ifconfig lo0:3 down
b) Bring both interfaces up with the correct IP addresses:
ifconfig lo0 plumb  127.0.0.1 up
ifconfig lo0:3 plumb  127.0.0.2 up
c) Start NVBU which should start successfully.