Screen Scraping

[reddymade]

Could you please tell me, how do you do the screen scraping, for a webpage:

I would like to get the address information plus the persons name etc.

I have the table Tag structure, which has the address information and also the persons name.
How can i do the scraping to get the information.
Please help with any ideas atleast.
Thank you very much for the information.

      

<TABLE cellSpacing="1" cellPadding="1" Width="600" Border="0">

    <TR>
     <TD VAlign="top">
      <TABLE cellSpacing="0" cellPadding="2" width="450" border="1" BorderColor="black" >
       <TR>
        <TD WIDTH = "98" Align="Left">
         <SPAN CLASS = "RedBodyCopyBold9">
          Site Address
         </SPAN>
        </TD>

        <TD Align="Left">
         <SPAN CLASS = "BodyCopyBold9">
          1544, sw 34th Street, Miami,FL                         
           </SPAN>
        </TD>
       </TR>
       <TR>
        <TD Align="Left" VAlign = "Top">
         <SPAN CLASS = "RedBodyCopyBold9">
          Property Owner
        </TD>
        <TD Align="Left" VAlign = "Top">
         <SPAN CLASS = "BodyCopyBold9">
          JUAN HERNANDEZ
         </SPAN>
        </TD>
       </TR>
      </TABLE>

[bpat1434]

[man]preg_match[/man] or [man]preg_match_all[/man] is what you need....
That and one good regular expression...

[TF_TheMoon]

I'm not an expert with regular expressions, far far far from it, and i'm sure there is a much better way of doing it, but here's mine for that.

// $txt is the HTHML that you pasted above

$pattern ="!\"[Red]*BodyCopyBold9\">[ ]*[\\r\\n\\t\\x0B]*[ ]*(.*)[ ]*[\\r\\n\\t\\x0B]*[ ]*(</SPAN>|</TD>)!isU";

preg_match_all( $pattern, $txt, $matches);

$Matches[1] then contains a list of the matches, though you'll need to do a trim() to remove the additional whitespaces.

I can't remember the correct method for branching without using the (), adn for removing the white spaces.

Personally my 'scraping' code uses a loop and strpos as it has to do a lot more than I can work out the regular expression for. (plus I wrote it before i looked into them) :p

[bpat1434]

Perhaps this could be faster/better:

$text = '';  // The code you posted above

$pattern = "~[Red]*BodyCopyBold9\">([^<]*)[</TD>|</SPAN>]~si";
preg_match_all($pattern, $text, $matches);
array_shift($matches);
echo '<pre>';
print_r($matches);

That gets the following:

Array
(
    [0] => Array
        (
            [0] => 
              Site Address

        [1] => 
          1544, sw 34th Street, Miami,FL                         

        [2] => 
          Property Owner

        [3] => 
          JUAN HERNANDEZ

    )

)

[reddymade]

Hello Pat,
Thank you very much.
Can you please tell me, how should i mention the url link, which need s to be scraped,

this is my url:
http://www.bcpa.net/Includes/Inc_RecInfo.cfm?URL_Folio=514012345678

That information, i need to assign to $text variable.

Please help, Thank you very much.

[MarkR]

Screen scraping should be considered a last resort when you can't get the data any other way.

It's a brittle technique which could fail at any moment, due to the page owners changing the structure in some way.

For best results, you should use a "real" HTML parser, like PHP's DOM->loadHTML, however, this is still not guaranteed to survive changes in structure, particularly if you're just going on the general structure of the page.

Screen-scraping without the site owner's consent is of course very naughty and they might block you (or worse, send you deliberately misleading content). Scraping with the owner's consent isn't much better, as unless you have a contract with them, they have no particular incentive not to change the page structure.

preg_match is a lousy HTML parser.

Mark

[bpat1434]

Yes, screen scraping is a lousy thing to do. Yes it's "naughty" but sometimes it's just what has to be done.

To get the contents of the file into a string, use [man]file_get_contents/man.

[reddymade]

Hello Pat,
This is really my last resort, to get the information, i have to go through the screen scraping process.
And my company does have the authorization to scrape the info. from there site.
I am able to get all the content of the page. to the $text variable. but that content does have lot of <Tables> in it. which are all using the same <span>class through out the file.
So i am really interested in a particualr cell which gives <td>site address</td><td>3456 SW 21 street, Miami, FL</td>

Please give me an idea, how to get the site address information, which has (SW 21 street, Miami, FL)
That will be a greatest help for me.
Thank you very much for the information.

       <TABLE cellSpacing="1" cellPadding="1" Width="600" Border="0">

    <TR>
     <TD VAlign="top">
      <TABLE cellSpacing="0" cellPadding="2" width="450" border="1" BorderColor="black" >
       <TR>
        <TD WIDTH = "98" Align="Left">
         <SPAN CLASS = "RedBodyCopyBold9">
          Site Address
         </SPAN>
        </TD>

        <TD Align="Left">
         <SPAN CLASS = "BodyCopyBold9">
              3456 SW 21 street, Miami, FL             
         </SPAN>
        </TD>
       </TR>
       <TR>
        <TD Align="Left" VAlign = "Top">
         <SPAN CLASS = "RedBodyCopyBold9">
          Property Owner
        </TD>
        <TD Align="Left" VAlign = "Top">
         <SPAN CLASS = "BodyCopyBold9">
          Cruz,FERNANDO & Cruz,Elizabeth
         </SPAN>
        </TD>
       </TR>
       <TR>
        <TD Align="Left" VAlign = "Top">
         <SPAN CLASS = "RedBodyCopyBold9">
          Mailing Address
         </SPAN>
        </TD>
        <TD Align="Left" VAlign = "Top">
         <SPAN CLASS = "BodyCopyBold9">
           3456 SW 21 street, Miami, FL
         </SPAN>
        </TD>
       </TR>
      </TABLE>
     </TD>

[bpat1434]

$pattern = "@<TD WIDTH = \"98\" Align=\"Left\">.*<SPAN CLASS = \"RedBodyCopyBold9\">.*Site Address.*</SPAN>[^<]*</TD>[^<]*<TD Align=\"Left\">[^<]*<SPAN CLASS = \"BodyCopyBold9\">([^<]*)</SPAN>@si";

preg_match_all($pattern, $string, $matches);
array_shift($matches);  // get rid of the first element of array

echo trim($matches[0][0]);

[reddymade]

Pat, Thank you very much for the help.

[reddymade]

Hello Pat,

Could you please help, i want to build a pattern to get the data which is in the following Div tag id="subtext",

I want to get the Name, Address, and telephone number. in the array.

$pattern = "~[Red]*BodyCopyBold9\">([^<]*)[</TD>|</SPAN>]~si";

*******************************************************
<div id="subtext" style="padding-top: 8px;">
<span style="line-height:18pt;">
	<strong>Frank&#160;Sanchez</strong><br>
</span>
12345 SW 21st St<br>
Miromar, FL  33321-4286<br>
[phone]<br>
<a href="/10668/track/10126/search/FindNeighbors?search_id=10311351887581679174&amp;element_id=0">Find Neighbors</a>
<br>
</div>
********************************

Thank you very much.

[bpat1434]

Try using this as your pattern:

"@<div id=\"subtext\".*>.*<strong>([a-zA-Z ]*)</strong>.*</span>([a-zA-Z0-9 ]*)<br>([a-zA-Z0-9, -]*)<br>([a-zA-Z0-9\(\) -]*)<br>@si";

[reddymade]

Hello Pat,
I used the above pattern but i am only getting this: when i scrape the info:
Array
(
[0] => Array
(
)

[1] => Array
    (
    )

[2] => Array
    (
    )

[3] => Array
    (
    )

)

<?php
global $url, $text;
$url = "http://www.411.com/10668/search/FindPerson?firstname_begins_with=1&firstname=rajender&name=bekkary&city_zip=33027";
$text = @file_get_contents($url);  // The code you posted above 

$pattern = "@<div id=\"subtext\".*>.*<strong>([a-zA-Z ]*)</strong>.*</span>([a-zA-Z0-9 ]*)<br>([a-zA-Z0-9, -]*)<br>([a-zA-Z0-9\(\) -]*)<br>@si";
preg_match_all($pattern, $text, $matches); 
array_shift($matches); 
echo '<pre>'; 
print_r($matches); 

?>

<div id="subtext" style="padding-top: 8px;">
<span style="line-height:18pt;">
	<strong>Frank Sanchez</strong><br>
</span>
12345 SW 21st St<br>
Miromar, FL  33321-4286<br>
[phone]<br>
<a href="/10668/track/10126/search/FindNeighbors?search_id=10311351887581679174&amp;element_id=0">Find Neighbors</a>
<br>
</div>

Can you please tell me am i doing something wrong.

Thank you.

[bpat1434]

You'd have to play around with the RegEx.

Google for regex tutorials, plenty of sites that go over this. I've given you a starting point, now it's time for you to play and see what works.