AIX Client fails to start "Fatal error during address bind (Port 20031) Address already in use" (4034022)

AIX Client fails to start "Fatal error during address bind (Port 20031) Address already in use"

Following successful NVBU Client installation on an AIX 6.1 server, the NetVault service fails to start.
Enabling trace while attempting to start NetVault shows that the NetVault Process Manager "nvpmgr" exits as the nvnmgr (Network Manager) process died so the application shuts down:
The nvnmgr trace itself shows that this process stopped because it cannot bind port 20031 which is already in use:
3   NETLOOP   ???   34341332   102   0   75   308.801679 Attach to 0x0:20031
6   NET   ???   34341332   119   0   75   308.801687 bind(7, 2ff224f8, 16) 0:20031
0   NET   ???   34341332   17   67   75   308.801713 Fatal error during address bind: Address already in use
6   SYSTEM   ???   34341332   20   0   75   308.801725 Logging 'Fatal error during address bind (Port 20031) Address already in use' to system logs
0   NETLOOP   ???   34341332   125   0   75   308.802015 Failed to open datagram socket on port 20031
2   NETLOOP   ???   34341332   205   0   75   308.802123 Failed to initialize network manager system
5   PROCESS   ???   34341332   13   0   75   308.802131 ProcCleanup(1)

NetVault uses port 20031 UDP for IPCs (Inter Process Communications) via the loopback address to speak to its processes,
but also for communications between other NetVault clients.
If this port cannot be bound to NetVault the application cannot remain started as nvpmgr cannot communicate with the other processes.
Using system tools one can determine which other application already has this port and take steps to free it for use by NetVault.

To determine which process already uses port 20031 preventing NetVault from starting, perform the following:

1) Make sure the process already locking port 20031 is not a NetVault process left after an unclean shutdown (zombie process).
If NetVault is not stopped correctly, nvpmgr may exit but leave the rest of the other process.
To be sure no netvault processes are running use:

ps -ef | grep nv

this should not return any nv* process.

2) If there are no zombie NetVault processes, then another application is locking that port.
In an attempt to locate which process  is holding it you could seach for it this way:

netstat -Aan |grep 2003

This should return the ouput in this format:

f1000e0035f9b600 udp4 0 0 127.0.0.1.20030 *.*
f1000e0044215000 udp4 0 0 127.0.0.1.20035 *.*

3) Now use the long number associated with the 20031 port with the rmsock command this way:

rmsock f1000e0035f9b600 inpcb

It should show you the process id associated with this socket and 20031.
example:

The socket 0x3665e408 is being held by proccess 8912968 (oracle).

In this example we know Oracle is already using this port.

4) Now using the process id shown in the previous command, you can determine the exact process that is using the port:
ps -ef | grep 8912968 

Port 20031 is a necessary port for NetVault to function, therefore one should schedule a time when the application already using this port can be shutdown, so NetVault can successfully start, then bring up the application which should use any other port.
As a last resort, there is a possibility to change de NetVault default port, but this option should only be contemplated in extreme cases where for instance a production database must at all cost use 20031 and no other port.
The following solution explains how to change the NetVault default port but this is not recommended as it would imply also changing the default port on all other NetVault machines (Server and Clients) so they can successfully communicate and allow backup and restore.

https://support.quest.com/SolutionDetail.aspx?id=SOL78546&pr=NetVault%20Backup