You're basically right about X. If the server is doing what it's supposed to be then you certainly don't need X to be running - and if you're not going to be running it there's little point in installing it - it's a big puppy.
If things do go awry, then commandline access is all you need to get at all the diagnostic utilities needed to get things back up again - so that makes terminal access a lot simpler as well (since only text needs to be piped back and forth).
If you do want to use a Linux GUI, you could have it on a separate machine, and connect to the server as and when needed.