Web Fetching

w0rM

I have a page that I want to grab some stuff from in real time so I can provide up to the minute traffic info on a web site (yes, I have permission). Normally this would be no big deal but the HTML is a little odd so grabbing what I need and stripping out what I don't is pretty hard. Take a look at the HTML for this page.

What I want to do is to pull each accident from the site, strip all the BS, count the number of accidents and display the info on a page. The problem I'm having is trying to tell when one accident ends and anothe begins. As you can see in the HTML, there's no common way for a line to end. I can fetch the whole thing but splitting up the accidents is kicking my butt!

I know this can be done but I'm at a loss as to how to do it. Any ideas or suggestions?

AstroTeg

Well, each accident ends by a carriage return. Lets take a look:

__________________________________________________________________________________________________

11:34  ACCIDENT                P     3000  AIRLINE HW                    I-110
11:18  ACCIDENT                P     3100  HARDING BL                    VETERANS MEMORIAL/HOWEL
10:33  HIT AND RUN             P      700  WEST I-10 HW                  ACADIAN/KCS RR UNDERPAS
10:25  ACCIDENT W/INJ         S FE   7600  PRESCOTT RD                   MARIBEL/LANIER

I'd recommend working on just getting about this much data into a variable. Once you have this data seperated out, you've "divided and conquered" and taking the next stepp will be a bit easier.

The next step I might take would be to split the lines on the carriage return (or the "\n" character). A simple explode("\n", $string) should work (where $string contains the text chunk). Now its a matter of looping through the resulting array and ditching the first couple lines (maybe ditch each line until the first character is a number).

Now you can actually start handling each individual line. You'll see the data is column delimited. So you can use [man]substr[/man] to read the individual parts of the line. When done, you move onto the next line. I might also recommend doing [man]strip_tags[/man] to ensure you don't have any HTML junk mixed in (see the last line).

Basically, start by narrowing your data down into something you can manage. Then narrow it down again. Rinse and repeat. Not the best way to do it all the time, but for this problem it will do fine.

drawmack

Here is the code that does the bull work, note that it is tied heavily to the html input so if they change their site layout it may need to be reworked

<?php
$url = "http://ci.baton-rouge.la.us/reports/public/cdactiv.html";
$page = GetHTML($url,$a,true);

//find the start of the preformated text
$trim_to = strpos($page['html'],'<pre>');
//find the end of the line that begins the preformated text
$trim_to = strpos($page['html'],"\n",$trim_to);
//find the start of the city name
while(!eregi('[a-z]',substr($page['html'],$trim_to,1))) {$trim_to++;}
//find the end of the city name
$stop_at = strpos(strtolower($page['html']),' area',$trim_to);
//calculate the length of the city name
$length = $stop_at - $trim_to;
//pull out the name of the city
$city = substr($page['html'],$trim_to,$length);

$stop = strpos($page['html'],"\n",$trim_to)-$trim_to;
//find the first line containing traffic accident data
while(!eregi('^[0-9]{2}:[0-9]{2}',substr($page['html'],$trim_to,$stop))) {
  $trim_to += $stop;
  $stop = strpos($page['html'],"\n",$trim_to) - $trim_to + strlen("\n");
}

//find the end of the traffic accident data
$stop = strpos($page['html'],'</style>',$trim_to);
$length = $stop - $trim_to;

//create a string of just the accidents
$text = substr($page['html'],$trim_to,$length);
$text = htmlentities($text); 

//turn the acidents into an array
$text = explode("\n",$text);

//capitalize city properly
$city = explode(" ",$city);
$fmt_city = '';
foreach($city as $part) {$fmt_city .= ucfirst(strtolower($part));}

echo $city . " had " . count($text) . " accidents recently.<br /><br />";
foreach($text as $accident) {echo $accident . "<br />";}
?>

and for completness here is the code for GetHTML

<?php
function GetHTML ($url, &$delta, $cut = false, $complete = true) { 
  $url_stuff = parse_url($url); 

  if(empty($url_stuff['host'])) {return FALSE;}
  if(empty($url_stuff['path'])) {$url_stuff['path'] = '/';}
  if(empty($url_stuff['query'])) {$url_stuff['query'] = '';}
  else {$url_stuff['query'] = '?' . $url_stuff['query'];}

  $fp = fsockopen ($url_stuff['host'], 80, $errno, $errstr, 30); 
  if (!$fp) {exit;}
  else { 
     $header = "GET " . $url_stuff['path'] . "?" . $url_stuff['query'] ; 
     $header = $header . " HTTP/1.0\r\nHost: " . $url_stuff['host'] . "\r\n\r\n"; 
     fputs ($fp, $header); 

 $header = ''; 
 $body = ''; 
 $act = false; 
 $end = false; 
 while (!feof($fp)) { 
     $line = fgets ($fp,1024); 
     if (!$act) { 
         if (strpos($line, "\r\n", 0) == 0) { 
             $header .= $line; 
             if (!$complete) $end = true; 
             $act = true; 
         } else { 
             $header .= $line; 
         } //end if 
     } else { 
         $body = $body . $line; 
     } //end if
 } //end while

 fclose ($fp); 
  } //end if 


  return array('headers'=>$header,'html'=>$body); 
} 
?>

w0rM

Wow! Thanks guys! I wasn't expecting all that! I really can't thank you enough! When I get a raise at work I'm treating you both to dinner. LOL

w0rM

This works just perfectly! I'm so thrilled, I can't tell you...

In all seriousness, it's sites like this one and ppl like you two that make the internet such a great place! Thank you both for going out of your way to help me. You have my undying grattitude.