from method to code

wsh

maybe someone would be kind enough to tell me how to make a parser... this is a dmoz dump file parser.. i know what must be done... but... can someone help me figure out how to do it???

I just want the Link, url and description.. not the categories and structure... now this is how the links to external sites (not category links) look like in the rdf file:

<ExternalPage about="http://www.moonshiney.com/">
  <d:Title>Moonshiney.com</d:Title>
  <d:Description>Live conversation with several women.  Photos and stories.</d:Description>
</ExternalPage>

Now... obviously what needs to be done is:

1) Open the file
2) Look first for a <External Page about=" tag
3) Grab the information from between the two quotes (the url)
4) Store that into an array ($insertrow[url]=[THE INFORMATION EXTRACTED])
5) Look for the <d:Title> tag right below the <externalpage about= tag
6) Grab the information between <d:Title> and </d:Title>
7) Store that into our array ($insertrow[title]=[THE INFORMATION EXTRACTED])
8) Lool for a <d😃escription> tag right below the <d:Title> tag
9) Grab the information between <d😃escription> and </d😃escription>
10) Store that information into our array ($insertrow[description]=[THE INFORMATION EXTRACTED])
11) insert that row into a mysql table (plz make it mysql.. its the only thing i have )
The table structure should be:
id -auto_increment unique
title -text or varchar [for fulltext indexing which will be added AFTER the table is built??? unless its better otherwise... you be the judge I have no experience]
url -text or varchar
description -text or varchar [fulltext like title]

$inserttitle=addslashes($insertrow[title]);
$insertdesc=addslashes($insertrow[description]);
INSERT INTO links VALUES("","$inserttitle","$insertrow[url]","$insertdesc") -maybe use faster syntax [if existing]

12) Looks for next <externalpage tag loops through until the file is finished...

--ALSO-- Maybe it should echo starting and then echo out how many urls it has done every say... 50,000 links? maybe 100,000?

--ALSO-- Should be made to ignore the whole category tags and crap (just search for the externalpage tags)

Im sure this whole thing is possible ... but i dont have any idea how to make it... I have been trying 4ever to find a parser.. failed.. so now its time to make one!

Thanks.. and plz SOMEONE help! I have done my research.. but i dont have the experience to make this! Maybe I can offer the final database when done?

JDcrack

try something like this:

$start= strpos($fd, "<ExternalPage"); 
$finish= strpos($fd, "</ExternalPage "); 
$length= $finish-$start; 
$code=Substr($fd, $start, $length);
$code=ereg_replace('texttoreplace', "replacewiththistext=", $code);
echo $code;

It wont do everything you exactly what you are asking it to do, but it will help you to get in the right direction.
Also see:
http://www.php.net/manual/en/function.ereg-replace.php
http://www.php.net/manual/en/function.fopen.php

*cough... rtfm....ahem.

shastry

Hi,

This kind of stuffs(Parsing) can be done very easily using Perl Script.

Weedpacket

You could consider using one of the ranges of XML-parser functions PHP supplies.

wsh

Thx.. I still cant figure this sh*t out.. that code kinda sorta gives me a clue.. but still need some more help

*cough... rtfm... ahem

Its alot easier for me if I can get nice people to at least tell me what sort of functions and crap to look at, I mean I have a little experience with this, but not alot.

Perl script? umm... if you wanna help me figure that out, sure.. but php is easy and i have most of the basics already... i know perl and php are like the same thing.. sooo.... i just couldnt be bothered..

XML parser functions... ill look into that..

thanks all... helps sorta... if anyone can supply a little more code and know-how it would be great...

wsh

$code=ereg_replace('texttoreplace', "replacewiththistext=", $code);

Wtf is that for?

how would i apply that code to my needs (moving it from the big ass 700+mb file to my mysql database?

Weedpacket

See this article

wsh

lol... omg i have something to read now... I have been trying to peice together what JDcrack told me... I was wondering if this code would sorta work.. like from this could someone fix it to actually work and give me the code??? if this code doesnt work... just need an expert to look this over and give me a working version...

$fd = fopen ("content.rdf.u8", "r");

foreach($start= strpos($fd, "<ExternalPage"; $finish= strpos($fd, "</ExternalPage>"); $length= $finish-$start; $buffer=Substr($fd, $start, $length);) {

$start= strpos($buffer, "<ExternalPage about=\""); 
$finish= strpos($buffer, "\">"); 
$length= $finish-$start; 
	$url=Substr($buffer, $start, $length);

$start= strpos($buffer, "</d:Title>"); 
$finish= strpos($buffer, "\">"); 
$length= $finish-$start; 
	$title=Substr($buffer, $start, $length);
$tid++
$start= strpos($buffer, "</d:Description>"); 
$finish= strpos($buffer, "\">"); 
$length= $finish-$start; 
	$description=Substr($buffer, $start, $length);

mysql_query("INSERT INTO links VALUES("","$url","$title","$description")");
}

At least hopefully you can understand how I was thinking of accomplishing it and maybe give me another method?! if necessary. I cannot open and read the whole darn file.. my computer would die... so i attempted to use foreach (that probably doesent work how i have it) to read one one tag at a time.. I know I cant use fgets because I might get part of a tag like the url, title and not the description or something.

wsh

lol... went retarded...

	$start= strpos($buffer, "<ExternalPage about=\""); 
	$finish= strpos($buffer, "\">"); 
	$length= $finish-$start; 
		$url=Substr($buffer, $start, $length);

$start= strpos($buffer, "<d:Title>"); 
$finish= strpos($buffer, "</d:Title>"); 
$length= $finish-$start; 
	$title=Substr($buffer, $start, $length);

$start= strpos($buffer, "<d:Description>"); 
$finish= strpos($buffer, "</d:Description>"); 
$length= $finish-$start; 
	$description=Substr($buffer, $start, $length);

maybe strpos not right? when i run just this with $buffer being the fopen statemet, the variables are empty! PLZ HELP!

wsh

lol... i am a fing dumb ass!!!!!!!!!!!! holy shit!

ok... i tried to just opent he first bunch of the file and get the vars for one set... this is what I ran:

$fp = fopen ("content.rdf.u8.ascii", "r");
$buffer=fread($fp, 102400);

$start= strpos($buffer, "<ExternalPage about=\""); 
$finish= strpos($buffer, "\">"); 
$length= $finish-$start; 
	$url=Substr($buffer, $start, $length);
echo "URL=$url<BR>";

$start= strpos($buffer, "<d:Title>"); 
$finish= strpos($buffer, "</d:Title>"); 
$length= $finish-$start; 
	$title=Substr($buffer, $start, $length);
echo "Title=$title<BR>";
	$start= strpos($buffer, "<d:Description>"); 
	$finish= strpos($buffer, "</d:Description>"); 
	$length= $finish-$start; 
		$description=Substr($buffer, $start, $length);
echo "Description=$description<BR>";
echo "$title, $url, $description<BR><BR>\n\n";

fclose($fp);

I got a HUGE bunch of SHIT because the problem is the URL expression... what I need is someone to tell me how to get this idea to work:


$fp = fopen ("content.rdf.u8.ascii", "r");
$buffer=fread($fp, 102400);

-NEED HELP HERE

How do I get it to select the information between the quotes of the <ExternalPage about="[url]"> tag... for example it would extract [url] from that like and save that as the variable url ( $url="[url]";

-SEE WHAT I MEAN???


$length= $finish-$start; 
	$url=Substr($buffer, $start, $length);


$start= strpos($buffer, "<d:Title>"); 
$finish= strpos($buffer, "</d:Title>"); 
$length= $finish-$start; 
	$title=Substr($buffer, $start, $length);

$start= strpos($buffer, "<d:Description>"); 
$finish= strpos($buffer, "</d:Description>"); 
$length= $finish-$start; 
	$description=Substr($buffer, $start, $length);

Then... once that is figured out... how would i make it loop this through the file and extract only when there is full tags ... such as this reasoning (not this syntax):


foreach($start= strpos($fd, "<ExternalPage"; $finish= strpos($fd, "</ExternalPage>"); $length= $finish-$start; $buffer=Substr($fd, $start, $length);) {


-NEED HELP HERE

How do I get it to select the information between the quotes of the <ExternalPage about="[url]"> tag... for example it would extract [url] from that like and save that as the variable url ( $url="[url]";

-SEE WHAT I MEAN???


$length= $finish-$start; 
	$url=Substr($buffer, $start, $length);


$start= strpos($buffer, "<d:Title>"); 
$finish= strpos($buffer, "</d:Title>"); 
$length= $finish-$start; 
	$title=Substr($buffer, $start, $length);

$start= strpos($buffer, "<d:Description>"); 
$finish= strpos($buffer, "</d:Description>"); 
$length= $finish-$start; 
	$description=Substr($buffer, $start, $length);

mysql_query("INSERT INTO links VALUES("","$url","$title","$description")");
}

fclose($fp);

The problem is... how do i get php to read a chunk of the file and parse up to the last full <externalpage></externalpage> set and then mark its position at the end of that </externalpage>, then continue reading, starting at that set mark, -you know.. make it read it with a buffer, but not get partial tags, and continue where left off... PLZ!! I AM VERY HANDYCAPPED AND MY CODE IS F*ED TO HELL... PZL PLZ READ ... TRY TO UNDERSTAND IT, AND RE-WRITE THE CODE TO DO THIS!!! PPPPPLLLLLZZZZZ... I WILL DO ANYTHING... GIVE YOU THE DATABASE OR ANYTHING!!!