Extracting Data from HTML Code...

Dixen

I'm rather in need of some help with a script I need to create. Most of what I have done with PHP is insertion into DBs and extraction and display, with some calculations and manipulation of variables mixed in... What I need is this:

The script I need to build would be one that takes set html code and extracts approx 55 numbers or Text strings from the code and inserts them into appropriate table fields for extraction and display at a later time. The format of the html will be the same every time, but obviously the information that would need to be extracted would not be.

Since I have never done anything like this before I'm not even sure where to start. I look over the other information avail on PHP Builder and am still unsure where I should begin.

Any help or points in right directions would be appreciated.

Thanks.

TheIceman5

heres a function that will return the html of a webpage that you put in the function. Just call this function and pass in the url you want the html code for. This script can then be modified to extract what ever you wanted from it with some other functions.
Is this what you were looking for or am i on wrong track?

<? 
define('SHOW_HEADERS',false); 

function get_source($url,$show_headers=0) 
{ 
    if(preg_match('!^[url]http://[/url]!',$url)) 
        $url=substr($url,7,strlen($url)); 

if(strpos($url,'/')) 
    list($url,$uri)=explode('/',$url); 
else 
    $uri=''; 

$fp=fsockopen($url,80,$errno,$errstr,4); 
if(!$fp) 
{ 
    echo "<b><font color=\"red\">Unable to connect to: $url</font></b>"; 
    return false; 
} 
else 
{ 
    $buffer=''; 
    $headers=''; 

    fputs($fp,"GET /$uri HTTP/1.0\r\n"); 
    fputs($fp,"Host: $url\r\n"); 
    //fputs($fp,"Referer: [url]http://www.plebian.com\r\n[/url]"); 
    fputs($fp,"User-Agent: sourcegetter\r\n"); 
    //fputs($fp,"Cookie: x=y;a=b\r\n"); 
    fputs($fp,"Connection: close\r\n"); 

    fputs($fp,"\r\n"); 
    while(!feof($fp)) 
    { 
        if(!isset($end_of_headers)) 
        { 
            $header=fgets($fp,4096); 
            if($header=="\r\n") 
                $end_of_headers=1; 
            $headers.=$header; 
        } 
        else 
        { 
            $buffer.=fgets($fp,4096); 
        } 
    } 

    fclose($fp); 

    if($show_headers) 
    { 
        $headers=htmlentities($headers); 
        $headers=nl2br($headers); 

        echo $headers; 
    } 

    $buffer=htmlentities($buffer); 
    $buffer=nl2br($buffer); 

    echo $buffer; 

    return true; 
} 
} 
?>

Dixen

Uhm, wow... nice script. 🙂 But I think you went a bit beyond what I actually need to do. Users would view the source of the html page, copy the entire source, then paste it into a form. Then from this forum the script I need to build would then grab the necessary information and insert it into a table I would have prepared for it.

Does that better explain what I am trying to do? I appreciate the help, thanks.

Pig

You would have to use regular expressions to find that information. Do you have control over this page? It would be much easier if you could put in markers for the expressions to look for. Beyond pointing you towards regular expressions, I don't think anyone could help you without know EXACTLY what you are trying to do.

Dixen

Okay, here is an actual example of the html code I am going to be working with. The format will always be the same, but obviously the values I need to grab and store will change. I do not have control over the page, but instead would have users copy and paste the source code (which is the example below) into a text box or form and then submit it for the script to do it's thing.

I actually counted and there are 51 values that would need to be extracted and stored in table fields. Each time the script would run it would produce a single record in the table in question.

I colored the lines that I would be extracting from in Red, my comments added to the code are in Blue.

<html><head><title>Earth: 2025 Standard</title>
<style type="text/css">
BODY, TABLE {font-family: verdana,arial; font-size: 12px}
.small {font-size: 10px}
.title {color: #ffeecc; background: #500000; font: bold}
.bluetitle {color: #ffeecc; background: #000050; font: bold}
.y {color: #ffeecc; font: bold}
</style>
</head>
<body BGCOLOR=#000000 TEXT=#FFFFFE LINK=#FFEECC VLINK=#FFDDBB

<center>
<script LANGUAGE="JavaScript1.1" SRC="http://network.realmedia.com/RealMedia/ads/adstream_jx.ads/swirve/ros/ron/ent/ss/a@Position1"></script>
<script LANGUAGE="JavaScript">

</SCRIPT>
<noscript><a HREF="http://network.realmedia.com/RealMedia/ads/click_nx.ads/swirve/ros/ron/ent/ss/a@Position1?x"><img SRC="http://network.realmedia.com/RealMedia/ads/adstream_nx.ads/swirve/ros/ron/ent/ss/a@Position1"></a></noscript>

 
<table CELLSPACING=5><tr bgcolor=#500000><td>Turns: 75</td>
<td>Money: $25,000</td>

<td>Food: 100 Bushels</td>
<td>Networth: $7867</td>
<tr></table>
<table BORDER=1 CELLSPACING=0 BGCOLOR=#222222>
<caption>The Status of your Country as of 06/08/2003</caption>

/* From the line above the Date would need to be extracted and stored; 06/08/2003.

<tr><th NOWRAP>Country Name
<td NOWRAP COLSPAN=3>Legio Spartan (#2010)
<th NOWRAP>Government<td NOWRAP>Monarchy

/* From these two lines 2 values would need to be extracted and stored; Legio Spartan (#2010) and Monarchy. It is possible that I would want to also store the 4 digit # as an additional seperate field as well.

<tr VALIGN=TOP><th NOWRAP>Turns Left Turns Taken Turns Stored Rank NetWorth</th>

<td ALIGN=RIGHT NOWRAP>75 0 37 11,528 $7867</td>

/* The line above has 5 vaules to be extracted and stored; 75, 0, 11,528, 7867.

<th NOWRAP>Money Food Oil Population Land Per Capita Income</th>
<td ALIGN=RIGHT NOWRAP>$25,000 100 bushels 0 barrels 1000 170 Acres $20.00</td>

/* The line above has 6 values to be extracted and stored; 25,000, 100, 0, 1000, 170, 20.00.

<th NOWRAP>Tax Rate Taxes Expenses Food Production Food Consumption</th>
<td ALIGN=RIGHT NOWRAP>15% $3000 $656 68 bushels 21 bushels</td>

/* In the line above there are 5 vaules that would need to be extracted and stored; 15, 3000, 656, 68, and 21.

</tr>
</table>
<table BORDER=1 CELLSPACING=0 BGCOLOR=#222222>
<tr><th COLSPAN=2>Expense Breakdown
<tr><th>Military Alliances Land</td>

<td ALIGN=RIGHT>$12 $0 $644</td>

/* In the line above there are 3 vaules that would need to be extracted and stored in table fields, the 12, 0, and 644.

</table>
<table BORDER=1 CELLSPACING=0 CELLPADDING=4 BGCOLOR=#222222>
<tr><th NOWRAP COLSPAN=2>Land Distribution
<th NOWRAP COLSPAN=2>Military Forces<th NOWRAP COLSPAN=2>Technology Ratings
<tr VALIGN=TOP><td NOWRAP>
Enterprise Zones 
Residences 
Industrial Complexes 
Military Bases 

Research Labs 
Farms 
Oil Rigs 
Construction Sites 
Unused Lands 
<td NOWRAP ALIGN=RIGHT>
1 
2 
3 

4 
5 
6 
7 
8 
9 

/* These 9 numbers would be extracted individually and placed into 9 different named fields.

<td NOWRAP>
Spies 
Troops 

Jets 
Turrets 
Tanks 
Nuclear Missiles 
Chemical Missiles 
Cruise Missiles 
<td NOWRAP ALIGN=RIGHT>
1 
2 

3 
4 
5 
6 
7 
8 

/* These 8 numbers would be extracted individually and placed into 8 different named fields.

<td NOWRAP>
Military 
Medical 

Business 
Residential 
Agricultural 
Warfare 
Military Strategy 
Weapons 
Industrial 
Spy 
SDI 

<td NOWRAP ALIGN=RIGHT>
1 
2 
3 
4 
5 
6 
7 
8 

9 
10 
11 

/* These 11 numbers would be extracted individually and placed into 11 different named fields.

</table>
 <table><tr align=center><td>
<iframe src="http://leader.linkexchange.com/513/X191221/showiframe?" width=468 height=60 marginwidth=0 marginheight=0 hspace=0 vspace=0 frameborder=0 scrolling=no>
<a target=blank href="http://leader.linkexchange.com/513/X191221/clickle"><img
width=468 height=60 border=0 ismap alt="" src="http://leader.linkexchange.com/513/X191221/showle?"></a></iframe> 
<a href="http://leader.linkexchange.com/513/X191221/clicklogo" target="top"><img
src="http://leader.linkexchange.com/513/X191221/showlogo?" width=468 height=16 border=0 ismap alt="LinkExchange Network"></a> 
</td></tr></table>
</body></html>

TheIceman5

what a nightmare. i dont even wanna think about it.

Dixen

That bad, eh? Hmmm... okay. New plan. What about picking up the entire table (so that the ad banners and such would be left behind) and just inserting it into a table field? I would also like to remove all of the "size-x" in the table as it tends to mess the font sizes up when you look at it that way.

I would need to grab 1 line up from the first marked red and 1 line beyond the last marked blue.

Please tell me that would be easier?

Dixen

Bump, anyone have anything further to add...

feeling frustrated

Thanks.

benkillin

well, I dont think you can get specific lines, but to return the entire source, just do this

<?
$url = "http://www.phpbuilder.com/";
$source = show_source($url);
echo $source;
?>

Dixen

Hmmm... that would grab it straight from the url. But, my users would be copying and pasting the source from the website into a text box for me...

Thanks for the input though... 🙂

This is going to be harder than I thought... 🙂

benkillin

not really, if you want int in a text box then do this:

<?
$url = "http://www.phpbuilder.com/";
$source = show_source($url);
?>

<textarea><? echo $source; ?></textarea>

Dixen

Actually, what I meant is that Users will copy the source of a webpage (like that I gave above as the example) and they will paste it into a textarea... then from the text area I want the script to extract the info I need from the source that the user provides.

Does that better explain?

benkillin

not really...

Dixen

Hmmm, okay... let me see if I can explain this. I admit, it's not easy.

Okay, assume you are a user of my site. and you have the html source that I gave above. You would come onto my site and access the page in question. You past the source code above into a text area provided and click a submit button... then the script I need would parse that html code and find the information I need (colored in red in the example above) and then insert it into a mySQL Table.

I hope this is better.

benkillin

well, you can put it into a mysql database easy enough, but its gonna be very difficult to do it by line numbers, you cant do it by finding specific tags to the best of my abilities to understand PHP...