caching pages... curl? ...

jmueller0823

Here's the situation.

We subscribe to newsfeeds that are delivered by js.

Sometimes there is a delay with the feeds that causes our home page to hang, waiting for the js to display.

I've gone around and around with a javascript solution (i.e. launching the feeds after the page load (onload), etc.) and I don't see that as an option.

I'm thinking this.

A few times a day, I'd like to launch the newsfeed js and write the resulting url's to a file. Then, we'll maintain our own static copy.

Make sense? If so, does anyone have an example of what I need?
Thanks!

bretticus

If you have shell access, the simpliest method that I can think of would be to use wget to download the link from the remote Website and setup a cron job to do it automatically 3 times per day. It could as simple as:

15 6,12,16 * wget -O /var/www/mysite/whatiwannacallit.html http://thelink.com/thepage.html

Good Luck.

jmueller0823

Okay.

I ran one of the js url's and wget returned something like this:

document.write('The Headline Text Appears Here');

So, we're halfway there.

I could use the resulting wget file (above), but need to parse the headline text.

Ideas? I want to keep this simple, but looks like a two step process, yes? What's your recommendation?

Thanks.

drew010

the easiest thing to do would be to save it to a .js file, and on your site put <script src="http://www.yoursite.com/file.js"> and it will be output just as it appears on their end, with no need to parse out the headline, and other stuff. but if you dont want to do that there are ways to get the headline data.

jmueller0823

drew010, thanks for the reply.

The point of using wget is to create a static copy of the feed. What I don't want to do is load the js file on the site (often the feed is slow and delays our page load time).

Am I making sense?

My goal is to store a static copy of the js output on our site. Then, display from that static text.

Jim

drew010

i know, but it you make a static copy of the exact same file on your site, you can just load it from your site, therefore it will still be fast cause you are referencing it from the saved version on your server.

jmueller0823

Hey Drew

i know, but if you make a static copy of the exact same file on your site, you can just load it
from your site, therefore it will still be fast cause you are referencing it from the saved version
on your server.

I'm not communicating.

This is the process:
#1 Run wget
15 6,12,16 * wget -O /var/www/mysite/whatiwannacallit.html http://thelink.com/newsfeed1.js

#2 Static file is created
This file contains: document.write('The Headline Text Appears Here');

#3 Use the above static file
Need to parse (remove js stuff) and display the text.

The 'saved version on the server' is the text (document.write)above-- not an executable file.

Or am I not understanding you?

drew010

if you put
<script src="http://www.yoursite.com/thefile.js"></script>
on your front page somewhere, you can tell it to use the saved version of the file. It will all be output properly, just like the original js file from the remote server, the only difference is it will be using the saved local copy so you wont have your site loading slow cause its coming from a slow remote site. I guess you could go out of the way to strip the javascript stuff and save it to a text file, buts its practically the same using a php include or something to bring in the contents as text and to use javascript to include a local file via script src.

jmueller0823

ohhhhhhhhhhh. Okay.
Very simple.

I'll give it a shot. Thanks for your patience.

Do you have knowledge of wget?
We use wget extensively, but since this feed can be problematic, I'm curious about timeout.

I assume wget has a standard timeout value... ?

drew010

i think the read timeout is 900 seconds by default. you can also set the switch -t int to tell it how many times to retry if the download fails

i wrote a script in php a while back to fetch an html template from a site, and save all the images to a folder on mine, here is the part for the html, it may be useful

<?php

$DATA_URL = "http://www.site.com/file.js";
$OUTFILE = "localfile.js";
while ( ($html = file_get_contents($DATA_URL) ) == FALSE) { sleep(5); }
//sleep 5 seconds if the download times out and then try again

$fp = fopen($OUTFILE, "w+");
flock($fp, LOCK_EX);
fwrite($fp, $html);
flock($fp, LOCK_UN);
fclose($fp);

?>

you can use cron to trigger that script to run as well, even optionally put a call to mail() in there to let you know every time it runs.

jmueller0823

Excellent. Thank you/