Short answer: yes.
Longer answer: You're getting URLs and paths mixed up. You're wanting a URL - the path would be relative to the root of your site's directory tree (wherever you've got it) - what you're calling the "server path", not relative to your entire server's filesystem.
If the file you're wanting them to download is outside your site's directory tree, and you don't want to move it so that it's inside, then you can have links to a script that uses [man]readfile[/man] or suchlike function to fetch the file and pass it through.
<a href="download.php&file=".urlencode($file_download).">
IMPORTANT!
In that download.php script, YOU MUST CHECK $_GET['file'] (which is where the filename that is to be downloaded is stored) to make sure that it's safe! Otherwise I could ask for any file on your system. This, not incidentally, is exactly why your method doesn't work.