VM backup task fails when the “Enable guest quiescing” option is checked but is successful if this option is left unchecked.
On ESX 4.0 hosts the VSS component is installed by default when Vmware tools is installed.
To verify that is installed go to Start->Control Panel->Administrative Tools->Component Services.
Click to open Component Services and then double click Computers->MyComputer->COM+ Applications.
You should then see the VMware Snapshot Provider which confirms that VSS is installed successfully from inside the Guest OS.