Checking if files exist on another server. Need a faster and more reliable way!

[atokatim]

I own an Mp3 search site that has over 480,000 mp3s in the Db and i need to do a massive clean up! I need some kind of script that i can get the URL from the DB and check to see if it exists. I have tried several different ways, but it just is not working fast enough for me and some of the songs are marked as missing and they actually are not. Could someone help please?

[dagon]

grab urls do a loop

get_headers() check its 200

could take several seconds per file so do it in batches.

[atokatim]

will that work if the URL for the song is a redirect to somewhere else and the song does exist?

[dagon]

how about testing it and finding out,

[MarkR]

Are these 480k all referring to the same server? If so you can simply do an appropriate "ls -lR" or "find" command on that server, then write a script which goes through the output finding out which of the 480k still exist.

If they are on several servers, you should get the admin of each server to send you a list (in a standard format) of exactly which files exist (perhaps you distribute your own tool to those servers to do this), then write a program to compare the contents of the database with those.

Mark