Fast Spiders

laviavigdor

I've tried several methods in PHP to speed up the process of getting some data from a page on the web.
FOPEN, CURL, and sockets.

The best timing I got was 3 sites per second (just for the fopen action of a small html page).

How may I speed this up, any suggestions?

Lavi.

MarkR

You're not going to get a faster speed unless you use multiple threads or nonblocking stuff.

I think you need to review what you're trying to do, and try to find a more sensible way of doing it - for example, you could cache the results of your http request somewhere locally.

If you're genuinely writing a spider, then you're presumably making a CLI PHP application, in which case you don't need to worry about it doing 3 sites per second.

In fact you should probably put delays in to make it slower to be nice to the site owners (if all the pages are on the same site anyway).

Mark

mcalpine

i have a spider and i sometimes get good speed out of it, but most times dont...

please post your code so i can see how you are getting 3 a second... maybe i can offer some help too...

your code may give mine the boost...

thanks
cameron

justsomeone

PHP Accelerator can work wonders.

Like MarkR says you should also look into multi threading.

MarkR

Presumably you're doing this in an offline spider. So you could just start up a number of PHP processes, which could do various sites in parallel. They'd have to have some communication of course.

That might be better than multithreading.

Or you could create a custom SAPI which runs several PHP programs in multiple threads? (Which could all be the same program really)?

Mark

mcalpine

i cant seem to find any info in the PHP manual about threading...

please post an example...

thanks
cameron

MarkR

I don't think PHP contains any support for creating/destroying threads. At least, not that's API-visible.

Some SAPIs use threads though (ISAPI, Apache2 if it's threaded). This means that PHP should at least be thread-aware.

If you want to use threads then you'd probably have to create a custom SAPI which creates / destroys threads etc on PHP's behalf (it could provide functions to do so).

I haven't done this - this is entirely speculation.

Instead of threads, you could use multiple processes though, this would be much easier. Just create a master process which starts as many children as necessary (Presumably both in the CLI SAPI). You can do this with the normal PHP API.

The processes would need to communicate in some sensible way - but this could be done by a mixture of (for example), pipes, sockets and a shared database, for example.

Mark

mcalpine

ok... thanks

the problem with getting a php app to create other php processes is that the master cant close until the others have completed.

but any way...

the problem im having with the sockets is that when spidering a dynamic page (no content-length is sent), the socket functions sit open, waiting for data even thouht the server has completed the send.... they wait until they time out... which is a waste of time....

is there something im doing wrong?

socket non blocking also just loops and returns no data. obviously cos the server takes a few seconds to respond.

MarkR

mcalpine wrote:
the problem with getting a php app to create other php processes is that the master cant close until the others have completed.

Is that a problem? You're running this all under the CLI SAPI anyway, so it shouldn't matter. Nothing will timeout, it should take as long as it takes.

the problem im having with the sockets is that when spidering a dynamic page (no content-length is sent), the socket functions sit open, waiting for data even thouht the server has completed the send.... they wait until they time out... which is a waste of time....

is there something im doing wrong?

If you're writing your own HTTP implementation (DO NOT DO THIS), then this means that your HTTP implementation is incorrect. Probably you've written an incorrect HTTP 1.1 implementation.

Don't write your own HTTP implementation. It will be buggy and complicated. Use PHP's built-in one, which works perfectly well. I've used it to fetch static and dynamic pages, and never experienced this problem.

Do not use "curl", which is now semi-deprecated anyway because PHP's had its own, better integrated, HTTP implementation in for ages.

If you want to do (for example) POSTs or set other headers, you can use stream_context_create (or whatever) and set stream context options to allow all necessary things to happen. Then just fopen your URL and read it in with fread()

Mark