If you're sending them all to the fastest server, it won't be the fastest for long. Meaning, the fastest server will vary and there's no way to know without pinging each one and timing the response time, or some other hackery.
Your best bet is probably either a round-robin style forwarding, or forwarding based on geographical proximity.
If you're doing this in code, then round-robin should be pretty simple. For geo-based mirror selection you could take a look at this package here:
http://pear.php.net/package/Net_Geo
Or you could take a look at the php.net code (see http://cvs.php.net/phpweb/ ) to see how they do it. You may be able to do it, depending on how much control over your server you have, you may be able to do it via an Apache module or other daemon-level tool.
Hope this helps.
Cheers,
Lux