VMware backup error: The host name used for the connection does not match the subject name on the (4045154)

Product & Version: NVBU
OS Version: Any
Module & Version: VMware Plugin
Application version: ESX



Symptoms:
When trying to backup a VM machine, the following is observed in the binary log:

Information 2011/03/24 23:22:00 21301 Data Plugin VCBACK Starting Program 'C:\Program Files\VMware\VMware Consolidated Backup Framework\vcbMounter.exe'
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.356 'App' 3876 info] Current working directory: C:\Program Files\BakBone Softwa...
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.356 'BaseLibs' 3876 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 2 numThrea...
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.356 'BaseLibs' 3876 info] HOSTINFO: This machine has 1 physical CPUS, 2 total c...
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.653 'BaseLibs' 3876 info] Using system libcrypto, version 90709F
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.684 'BaseLibs' 3876 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch:...
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.684 'BaseLibs' 3876 warning] SSLVerifyCertAgainstSystemStore: The remote host c...
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK * The host name used for the connection does not match the subject name on the host certificate
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK * A certificate in the host's chain is not time valid
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK * A certificate in the host's chain is based on an untrusted root.
Information 2011/03/24 23:22:01 21301 Data Plugin VCBACK [2011-03-25 00:22:00.684 'BaseLibs' 3876 warning] SSLVerifyIsEnabled: failed to read registry value....
Information 2011/03/24 23:22:01 21301 Data Plugin VC

The "host name used for the connection..." issue is a matter of getting the name you're using to connect to the host to match the certificate on ESX. You can see the actual name in the ESX server certificate by going to https://esxervername from a web browser and then looking at the certificate that's presented once the web page is displayed (i.e. in IE7 click on the toolbar where it has the red X shield and says "Certificate Error", or in Firefox 3 you'll see a page displayed that says "The certificate is only valid for ..." and will show the server name in the cert. You can either change the DNS name you use to access the ESX box to match that in the cert, or you can regenerate the cert on the ESX box to match the DNS name you want to use (see doc link below on how to regenerate the SSL cert).
The untrusted root issue has a few possible solutions, all of which involve generating a new SSL certificate request on your ESX hosts. Once you've done that you can:

If you have an internal PKI already setup which is trusted by your VC server then submit the certificate requests to the internal CA
If you don't have an internal PKI then submit the certificate requests to a trusted 3rd party CA such as Verisign, Entrust, etc.

Once you get the certificate back from the CA then install it on each of the ESX boxes.

http://www.vmware.com/pdf/vi_vcserver_certificates.pdf gives a good rundown on all of this in a lot more detail on how to regenerate certs, etc.