This can happen if the Data Execution Prevention is enabled for all processes on the system. It will occur when you open Recovery Manager for Exchange, before the MMC opens, you will see this error. If you click ok, the MMC opens, but when you try to kick off any operation, the MMC will disappear.
Please Disable DEP and then Restart the server, the issue should now be resolved.
Right Click on My Computer, go to Properties, and then click on the Advanced Tab.
Click on the Data Execution Prevention tab.
Set DEP to the top option which is Turn on DEP for essential Windows Programs and services only.
You will then be asked to reboot, if you are not asked to reboot, reboot anyway.
Open RME and the issue should be cleared up.