Cache a webpage in PHP?

[andy75180]

Hi, can anybody help with this one?

I'm trying to design a PHP script that caches a webpage. Similar to what Yahoo does in it's search result list.

It's easy to turn a URL into text I can then save using:

$lines = file('http://www.mysite.com/');

and equally as simple to then echo it on a script to print the page out again:

foreach ($lines as $line_num => $line) 
{
echo $line;
}

But the problem is how to save all the images, external style sheets etc as well. So when loading page back in from the script it doesn't receive anything from external internet files what-so-ever?

Anyone any advice with this sort of thing?

Andy[/COLOR]

[bradgrafelman]

Well, the easiest way to get the file would be to use [man]file_get_contents[/man]. Then, match all instances of 'src=' inside <>'s and retrieve those files as well. Don't forget, some websites use relative paths, some use absolute paths, so when you get an array of all of the files you need to download, check each file to make sure it begins with "http://" and, if not, add the directory from which you grabbed the page from to the beginning of the location.

Also, you'll have to make sure your script can handle the '..' notation, so if you grabbed a file from http://mysite.com/save/this/page.html and your script starts saving files, if it runs into ../../images/test.jpg, it should know to get rid of the '../../' and instead go back two directories so that it retrieves http://mysite.com/images/test.jpg.

Rather complicated, so you're script is going to have to be comprehensive as there are different ways to go about doing things.

[andy75180]

Yes I see. I've got an idea in my head how i'm going to do it now. :rolleyes:

But totally forgot about the ./ and ../ This makes is more difficult. :eek:

I would also need to save any external javascript files and external css style sheet files.

Can you think of anymore external files the average webpage uses? Don't have to worry about PHP files of course because they will be parsed before outputed. Good old PHP!! :o

Thankyou very much for your reply. It is most appricated.. 😃

[bradgrafelman]

Yep, there's more. 2 more just for CSS, that I can think of.

• CSS is loaded with a different attribute than JS (it's not src=), as in <link rel="stylesheet" type="text/css" href="CSS/mycss.css">, so you'd probably need a regexp such as:

$pattern = '/<link.*?href=["\']?(.*?)["\']?>/si';

Also, inside of <style> tags, you can use the CSS @import( str fileName ) to attach a CSS stylesheet to the page, so again, you'd need a regexp to detect @import() within the style tags.

The '../' handling really could be done quite easily, at least in my mind. Once you have the target you're trying to save, trim off the http(s):// from the full address ( http://mysite.com/etc/etc/etc.php ) and then, using [man]explode/man, break that string into pieces on every '/'. Then, make a function with a while() loop or something similar, loop through the external file location in question (each reference within the HTML that we find to other files that we've been talking about) until there is no '../' left in the string. If it is found during the loop, use [man]array_pop/man to remove one directory from the end of the address that you exploded above, and then remove the first occurence of '../' (ONLY one of them, don't just use [man]str_replace[/man] as that will replace ALL instances!) and then let the loop continue until '../' does not appear in the external file's location.

Confusing enough? 😉

[MarkR]

This is extremely nontrivial. I recommend that you find a tool that will do it for you, rather than trying to write a PHP app to do it.

It requires capturing external objects (images, css etc), saving them into local filenames, which may need to be changed from the remote server, and modifying all the HTML accordingly to point to local versions of these items.

There are some apps which can do that. I recommend you find one, rather than trying to make your own - you will probably not succeed very well if you try (it requires a LARGE amount of work - there are a lot of different syntaxes for including external objects; relative URLs are tricky to work out. It has to deal (as best it can)with incorrect HTML as well).

If you were to write such a tool yourself, you'd need to account for:

• Images
• Scripts
• Frames
• Iframes
• CSS
• Stylesheet includes
• Inline stylesheet includes
• Handling redirects correctly
• Handling relative and absolute URLs correctly (root-relative as well as absolute ones)

Plus probably lots of other things I can't think of right now.

You still wouldn't be able to detect and capture any resources that were referenced programmatically by Javascript.

Firefox has a feature "save web page as..." which takes all the dependent resources and saves them out too (optionally). I think MSIE also has something "Make available offline" which does something similar.

Mark

[andy75180]

God this is starting to sound very complicated!! 🙁

Actually I don't need any links in the chached page to work. I just need a 'snapshot' of the page visually.

Anyone any idea where I can find a script on the web to do this. I'll have a look myself in the meantime.

Thanks, andy

[bradgrafelman]

There's some kind of software out there than actually can save image files of what pages look like, but can't remember what exactly it is. Sorry!