Recently came across this mysterious error message when trying to install VMWare Server 2.0 on a freshly built Windows Server 2003 standalone machine, after running the installer, an error message would pop up saying;

“Software restriction policies prevent this installation.”

Which was rather strange since this machine was not part of a domain and there were no local software restriction policies in place. The first workaround was to create a local software restriction policy;

1. Open Local Security Policy Settings
2. Select Software Restriction Policies
3. Right click the Software Restriction Policies node and select New Software Restriction Policy
4. Double click Enforcement
5. Select “All users except local administrators”
6. Click OK

You might need a reboot after this, alternatively, running gpupdate may also work.

However, unsatisfied with the workaround, I checked the event logs, searched some more and found knowledge base article http://support.microsoft.com/kb/925336.

In summary;

“This problem occurs if the Windows Installer process has insufficient contiguous virtual memory to verify that the .msi package or the .msp package is correctly signed.”

I undid the software restriction policy installed the hotfix and the VMWare Server installation ran fine.