Protocol error from VMX appears during Quiesced snapshot in vRanger and/or vCenter.
Host ESX 3.5.0 Update 5
VM with a single disk
Quiesced snaps had worked in the past, but started failing. This points to VMware tools on the guest in some cases.
VMware tools are causing the issue for the VM.
Restart VMware tools on guest. The following VMware KB should be able to help as well. Http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1009558&sliceId=1&docTypeID=DT_KB_1_1&dialogID=211428299&stateId=1%200%20211430678
There is a similar KB for servers with 7 or more disks, but this happens on guests with 1 disk.