When adding a library it will start a process called nvdevicescan on the SmartClient where the library is attached. It will establish a communication channel between the NVBU server and this process on the SmartClient. In most cases this error is due to a firewall blocking this communication. This can be the case even if the firewall test in client management passes.
Some people open their firewall by port range and others allow access per process. There are some windows scripts that open firewalls based on the NetVault processes, but these do not include the nvdevicescan process. If this is how you have your firewall configured, you will need to manually create a rule for this process.
If the firewall is opened by port range, you need to be sure that that port range is configured in the firewall settings for that client and that the client is set to outside firewall. If outside firewall is not selected in client management for the client, it will not use the configured port range. Without this setting, it will use a very high numbered random port.