[Resolved] Speeding up URL Validation

Skull

Hi,

currently to check if a url is correct i'm simple using fopen() then checking if its TRUE or FALSE to determine wether the URL works or not.

The problem is when checking this in a mass, it runs pretty slow. Is there any other way that this could be sped up by some sort of function? I've checked the PHP manual and can't find anything (tried things such as is_link,is_file e.t.c..) and done a search so any help would be appreciated

Thanks in advance

rehfeld

im not sure what is the fastest, i would compare a few. try file_exists() and fsockopen()

but its going to be slow no matter what because your having to request the file from remote servers.

you may want to consider caching the results. like using a database to store each link, and have a flag for whether its valid or not.

set up another script via cron and run it every 30 min or whatever interval is suitable, and have that script update the db.

that way you can just query your db instead of doing an http request.

Skull

I would set up a database but its a "Spider Simulator", where the user types in their site URL and then they get shown how a spider would view their site (which includes showing all the links on their website) so that couldn't work.

I tried file_exists() and it didn't work (would return false when the file did actually exist)

I'll check out fsockopen() now.

rehfeld

gotcha.

in that case, i would def use fsockopen() because then you can use stream_set_timeout() so your script doesnt sit there and wait forever for the server to respond. that way you can specify a limit on how long php will wait before aborting.

Skull

Thanks, tried it out and it worked to a certain extent (and quickly🙂)

Problem is, it works if its just the domain name e.g "www.linksuk.com", but if its something like "www.linksuk.com/index.php" then it returns an error

rehfeld

take a look at the example given in the manual again

where you see this

$out = "GET / HTTP/1.1\r\n";

make it

$out = "GET /index.php HTTP/1.1\r\n";

Skull

Thanks, works fine now