The DIagnostic Server is automatically capped at about 2GB.
With large number of connections, each of these requires a certain amount of memory before anything happens.
The balance is required for the rest of the work that the DS does.
If the amount of memory available to the DS is reduced by capping it at a lower amount, and the DS for whatever reason should then require more memory than the cap allows, it will result in the DS crashing.
Depending on the circumstances, this may be a once only event, or it may be a regular event.
Capping it further is not recommended.