How to do an Remote/Local Image Diff ... has image been updated?

dalecosp

We store our own copies of customers' images for their products. On a "regular" basis, software looks at their sites to see if a product are still being offered, if the price has changed, etc.

I'm tasked now with trying to figure out if the image has been updated. Here's my thoughts/choices as I see 'em:

Try a remote call for mtime() or somesuch. But I don't think that PHP's filemtime() will do HTTP.
Try and read headers from a GET request to the image. Etag? Last-Modified? Right now the system isn't using cURL for images ... just a quick system call to fetch/wget.
Compute a hash for my copy of the image, grab the new image via HTTP, and hash that and THEN compare them ... sounds terribly expensive.
Erase my copy and just fetch a new one from them.

Anyone done anything like this before? (Pretty sure I've read and talked with some of you about similar stuff ... the search page seems to fail me today...)

NogDog

My initial thoughts are either 3 or 4, or possibly pulling down the image and seeing if it has any embedded metadata (exif?) you could use to compare?

Though I suppose if you have to pull the image down in any case (for either #3 or the exif stuff), you might as well just do number 4 and be done with it. 🙂

dalecosp

NogDog;11055175 wrote:
My initial thoughts are either 3 or 4, or possibly pulling down the image and seeing if it has any embedded metadata (exif?) you could use to compare?

Though I suppose if you have to pull the image down in any case (for either #3 or the exif stuff), you might as well just do number 4 and be done with it. 🙂

Yeah, but I'm concerned about taxing my system, so I wondered if there was something similar to 1 or 2 that might be elegant. Maybe I should read up on browser caches ... they do something like this, right?

laserlight

I would go with either #3 or #4. If you have a secondary server doing the hashing and downloading in the background, then I would go with #3, otherwise #4.

dalecosp

Hmm. I have a "secondary server", but it doesn't have any idea of the images as they exist on our system (IOW: modify that class and a few extras, probably). It would certainly be "lower hanging fruit" (the tech VP's favorite meal) to do #4, then. What advantage #3? I guess less load on our customer's servers ....

sneakyimp

Why bother with #3? You've already fetched the image, might as well go ahead and replace the local copy rather than calculating its hash.

Seems like #1 and #2 would depend on apache/iis/nginx settings on the remote machine -- unless you can get a PHP script on the remote machine to calculate the hash over there?

I've done something sort of similar, but I'm pretty sure we just assume that if the url is identical that the file has not changed. The fancy thing we did bother to set up was a multi-threaded system that can fetch a bunch of images all at once. Fetching hundreds of thousands (millions?) of images sequentially was just going to take too long. There's a post on the multi-threadifying here.

Weedpacket

You don't need to retrieve the resource if you only want to see if it has changed: you can ask for it with an "If-modified-since" request; You make the request a HEAD instead of a GET and decide from the response if it's worth checking further...

dalecosp

sneakyimp;11055183 wrote:
Why bother with #3? You've already fetched the image, might as well go ahead and replace the local copy rather than calculating its hash.

Seems like #1 and #2 would depend on apache/iis/nginx settings on the remote machine -- unless you can get a PHP script on the remote machine to calculate the hash over there?

I've done something sort of similar, but I'm pretty sure we just assume that if the url is identical that the file has not changed. The fancy thing we did bother to set up was a multi-threaded system that can fetch a bunch of images all at once. Fetching hundreds of thousands (millions?) of images sequentially was just going to take too long. There's a post on the multi-threadifying here.

I do multi-threading in some parts of the system; we adjust the thread count based on the target's capacity and whether or not the Sales Dept reports their clients are crying about it ;-)

For this portion, so far the image count is < 100K. The mechanism in question is part of an API of sorts, and can be called from more than one caller, which is how we'd want to implement threading in this context. The machine can't really be tasked to run much multi-threading as it's a production box that already hosts a number of our "important" web properties. Based on experience so far, I don't think the bottleneck (if one exists) is in the image fetching API.

Weedpacket wrote:
You don't need to retrieve the resource if you only want to see if it has changed: you can ask for it with an "If-modified-since" request; You make the request a HEAD instead of a GET and decide from the response if it's worth checking further...

I'll look into this idea. Initial test here doesn't seem to indicate that I can do this (HEAD) via file_get_contents? (Actually getting some confusing signals ... sometimes get full page, sometimes empty string ....)

NogDog

The cURL functions would allow you to just get the headers. (And for that matter, to get the file itself without having to worry about PHP configs that might disable file_get_contents() from using HTTP.)

sneakyimp

Looks like you should look into [man]curl_setopt[/man] and in particular CURLOPT_NOBODY.

dalecosp

Yeah, I know and use cURL when necessary ... it just feels too heavy for some things (I should probably try and research that, because "feeling" != "proof"), not to mention having to modify a working system. We're trying to give an answer to a client who says "what happens if we change the image but keep the filename". I'll certainly use cURL if it means we can't sell him otherwise, but if he's not gonna pay us I'm not gonna modify the whole system just for him right now 😉

sneakyimp

I love curl and use it by default. As NogDog pointed out, this sidesteps PHP configs that don't permit allow_fopen_url or whatever. curl does seem quite verbose when one uses it, so I tend to wrap my curl operations in a function or class so the code that uses it stays simple and readable.

Regarding client who wants a change, let them choose: either they change (no changes to images unless you also change the filename!) or they pay you for you to change. Choice lets clients feel like they are in charge. Even the illusion of choice can help keep clients happy.

Weedpacket

With file_get_contents() the empty string would be because the file contents got consists of the body of the response - no body, no content. Dunno why the full-page response in the other case (perhaps the target server mistakenly handles the HEAD as a GET).

I don't see a stream context option to not strip headers from an HTTP response; the next step in that direction would be down a level to fsockopen() or stream_socket_client(). Which would mean chasing redirects ourselves because we left those upstairs at the HTTP level....

dalecosp

Weedpacket;11055209 wrote:
With file_get_contents() the empty string would be because the file contents got consists of the body of the response - no body, no content. Dunno why the full-page response in the other case (perhaps the target server mistakenly handles the HEAD as a GET).

I don't see a stream context option to not strip headers from an HTTP response; the next step in that direction would be down a level to fsockopen() or stream_socket_client(). Which would mean chasing redirects ourselves because we left those upstairs at the HTTP level....

Yeah, sounds a pain for sure, 😛 If I decide to read headers, I'll just reimplement the fetch method in cURL. I think, for now, I'll leave it as is. I modified it to just check mtime on our copy and re-fetch if it exceeds a threshold. Like I say, if the prospective client communicates that this is such as concern as to not do business with us, I'll do what's needed to make 'em happy.

Sneakyimp, thanks for the comments regarding control. That might be an overlooked fact in our business plan, and worth remembering next time I'm involved in discussions with the Sales guys ;-)

sneakyimp

dalecosp;11055213 wrote:
Sneakyimp, thanks for the comments regarding control. That might be an overlooked fact in our business plan, and worth remembering next time I'm involved in discussions with the Sales guys ;-)

I don't know if you watch the HBO show Silicon Valley but a recent episode pointed out something I never really noticed: sales people like selling something that's easy to sell. If what they are selling is too hard, they'll find something easier to sell. It stands to reason that your sales people will want to promise your clients the moon for a song if they can. If you want better working conditions, you'll have to keep them in check.