[RESOLVED] Assistance Parsing Multiple links using DOM

[BASiQ]

I have this code which parses out a div from within the page gotten. Works great.

$html = file_get_dom('http://steamcommunity.com/profiles/76561198010758659/games?tab=all');
echo $html('div#game_440 h5', 0)->getPlainText();

but I need the code to parse from multiple links, instead of using one link like it does now. I would like these multiple links to be called from a text file.

How would I go about this? I know about the file() function. but I have no idea how to use it. The examples are all wrong

[bradgrafelman]

BASiQ;10995543 wrote:

I need the code to parse from multiple links, instead of using one link like it does now.

Well assuming that the syntax used is the standard CSS selector type syntax, then I would say you would need to remove the ID value 'game_440' (since ID values must be unique, thus you're only going to get at most one element to match that selector).

However, whether the pattern you end up with still selects only the desired elements and nothing else is something to be considered.

BASiQ;10995543 wrote:

I would like these multiple links to be called from a text file.

What does it mean for links to be "called from a text file" ?

BASiQ;10995543 wrote:

How would I go about this?

No idea - you haven't given us enough information. For example, what does this file_get_dom() function return?

BASiQ;10995543 wrote:

I know about the file() function. but I have no idea how to use it. The examples are all wrong

Well it's good that you know about it, but there are nothing wrong with the examples - they show how to use [man]file/man perfectly well. Since you haven't given us much information about what it is you're trying to do, though, I can't suggest why you might think they're "all wrong."

[BASiQ]

Sorry for the lack of info

quick explanation.
This code

$html = file_get_dom('http://steamcommunity.com/profiles/76561198010758659/games?tab=all');
echo $html('div#game_440 h5', 0)->getPlainText();

outputs this
1708.9 hrs on record

I can put this code on the page unlimited amount of times and as long as I change the "file_get_dom" link to something unique it does exactly what it's supposed to do. Output a list like this

1708.9 hrs on record
1230.4 hrs on record
854.7 hrs on record

My issue is a page filled with 300 lines of that code above is messy and a pain in the ass to code manually. So I would like to make the link in question [MAKETHISLINKVARIABLE] a variable.

$html = file_get_dom('MAKETHISLINKVARIABLE');
echo $html('div#game_440 h5', 0)->getPlainText();

With the variable looping through a text file full of links one link at a time in the end making an output like this recursively without needing hundred lines of unnecessary code lol

1708.9 hrs on record
1230.4 hrs on record
854.7 hrs on record

[bradgrafelman]

Sounds like you simply need to use [man]file/man to load the lines of a file into an array and use a [man]foreach/man loop on that array to execute the above code for each line.

[BASiQ]

I agree it sounds like that, but no matter what I do, I can not get the dang thing to work. I am constantly getting Fatal error's left and right. It is seriously annoying. I am super new to php coding, I am good at CSS and HTML but this is a whole new world for me so I am really struggling.

[bradgrafelman]

Show us what you've tried. You could literally add just two lines to your previous code (and the second is nothing but a '}' closing brace) to do what I said above.

[BASiQ]

lol, this is my mess right now

<?php

include('ganon.php');

 // Parse the google code website into a DOM
  // 		$html = file_get_dom('http://steamcommunity.com/id/ticky1994/games?tab=all');
  //    	echo $html('div#game_440 h5', 0)->getPlainText();

 // 		echo "<br />";

	$html = file_get_dom('http://steamcommunity.com/profiles/76561198010758659/games?tab=all');
  	echo $html('div#game_440 h5', 0)->getPlainText();


	  //  $source = 'array.htm';
	////	$dbg = 1;
	//    $html = file_get_html($source);
	//	echo $html('div#game_440 h5', 0)->getPlainText();


    # memory leak clean up
    $html->clear();
    unset($html);


$lines = file('array.htm');
// Loop through our array, show HTML source as HTML source; and line numbers too.
foreach ($lines as $line_num => $line) {
    echo "" . htmlspecialchars($line) . "<br />\n";

$html = file('array.htm');
echo $html('div#game_440 h5', 0)->getPlainText();
}		


?>

It's a mish mash of code I was able to scrounge and try to make sense of what it did. I am super new so forgive the massive mess. I have spent all day trial-and-error to get almost nothing lol. Also, instead of deleting text at this moment, I am just commenting it out so I can refer back to it incase I forget something

Another reason the code is such a mess is in addition to using file() I also wanted to output the link which the parser "parsed" after the parsed result [if that makes any sense] if not . this.

This is the result now.

1708.9 hrs on record

This is what I am hoping for as an end result.

1708.9 hrs on record - http://steamcommunity.com/profiles/76561197968052100/games?tab=all 

This is ALSO proving to be extremely hard.

[bradgrafelman]

Okay, so let's first ignore all of the irrelevant code...

$lines = file('array.htm');
// Loop through our array, show HTML source as HTML source; and line numbers too.
foreach ($lines as $line_num => $line) {
    echo "" . htmlspecialchars($line) . "<br />\n";

$html = file('array.htm');
echo $html('div#game_440 h5', 0)->getPlainText();
}

Next, look at what you do with $line inside the loop.

First you output it... okay, that doesn't follow what you said you were "hoping for as an end result," so I'm not sure why that's in there.

Next, you... well, there is no next. You never use it after the (out-of-place) echo.

Furthermore, where did the call to file_get_dom() go? It's nowhere to be found.

The only thing you should've changed was to pass this file_get_dom() function the current element from the foreach() loop iteration (e.g. $line) rather than the hard-coded URL.

[BASiQ]

Well, essentially that code was a failed attempt like 8 hours ago. But it did nothing so I ignored it lol.
Problem is, when I see that code, I have no idea what to do with it. I am not sure which is a variable and which is a variable. I can practically speak HTML, but this is a whole new language [duh lol]

All I have been working on was this one. This is as far as I have gotten [that I understand]

$html = file_get_dom('http://steamcommunity.com/profiles/76561198010758659/games?tab=all');
echo $html('div#game_440 h5', 0)->getPlainText();

[BASiQ]

	$lines = file('array2.htm');
	foreach ($lines as $line) {
		$htmline = ('http://steamcommunity.com/profiles/'.$line.'/games?tab=all');
		$html = file_get_dom($htmline);
		echo $html;
	//	echo $html('div#game_440 h5', 0)->getPlainText();
	}

    # memory leak clean up
    $html->clear();
    unset($html);

Got this so far. It's "ALMOST" working. See anything wrong? lol
Also, it is outputting the whole page, not just the defined text atm. Because when i tell if to use the other line, I get a fatal error.

[BASiQ]

Final edit

		$lines = file('array.htm');
		foreach ($lines as $line) {
		$htmline = ("$line");
		$html = file_get_dom($htmline);
		echo $html('div#game_440 h5', 0)->getPlainText();
		echo " ";
		echo "$line";
		echo "<br/>";
		}

This is working, but ONLY if I include ganon.php instead of simple_html_dom.php

The line

echo $html('div#game_440 h5', 0)->getPlainText();

errors if I use simple_html_dom.php but goes through and does exactly what I want [just slow as hell] using the ganon.php

So, either what is wrong with that line and DOM does not like it
or WTF si wrong with ganon that makes it sooooo slow?

[johanafm]

1. Use php tags, not code tags.
2. Indent your code

What's this supposed to do?

BASiQ;10995565 wrote:

		$htmline = ("$line");

BASiQ;10995565 wrote:

This is working, but ONLY if I include ganon.php instead of simple_html_dom.php

Ok. By this comment I assume that something in particular is not working when using simple_html_dom.php. If you fix that, things will work.

BASiQ;10995565 wrote:

[just slow as hell] using the ganon.php

This makes me feel something is taking too much time. Make that part work quicker and you're done.

Easy.

[BASiQ]

I am not sure if you are being sarcastic, or assume I know more then I do.

You essentially said "fix what's broken" and "make it go faster" and you're done.
This is kind of obvious and if I knew how to do both, I would be indeed done. but I don't. Or at least I don't "think" I know. :queasy:

        $htmline = ("$line"); 

Also, this line is to re-define what $line is as for some reason the $line command was being a real PITA so we tried this instead. It's probably not necessary.

[BASiQ]

you were right, did not need that line

		$lines = file('array.htm');
		foreach ($lines as $line) {
		$html = file_get_dom($line);
		echo $html('div#game_440 h5', 0)->getPlainText();
		echo " ";
		echo "$line";
		echo "<br/>";
		}

so now, only this line is giving this error when using simple_html_dom.php
"Fatal error: Function name must be a string in /home/MySite/public_html/parse/index.php on line 47"

		echo $html('div#game_440 h5', 0)->getPlainText();

it still works with ganon but is still quite slow.

[BASiQ]

Finished product for reference

<?php
	include('simple_html_dom.php');
	$lines = file('array.htm');
	foreach ($lines as $line) {
	$html = file_get_dom($line);
	$divline = $html->find('div[id=game_440] h5', 0);
	$divline = preg_replace('/(<h5>)/i', '', $divline); 
	$divline = preg_replace('/(<\/h5>)/i', '', $divline); 
	echo $divline;
	echo " - ";
	echo "$line";
	echo "<br>";
	$html->clear();
	unset($html);
}

array.htm is merely a text file with lines of steam profile links.

[BASiQ]

Final revision to clean up code

<?php
	include('simple_html_dom.php');
	$lines = file('array.htm');
	foreach ($lines as $line) {
	$html = file_get_dom($line);
	$divline = $html->find('div[id=game_440] h5', 0);
	$divline = preg_replace('/(<h5>)/i', '', $divline); 
	$divline = preg_replace('/(<\/h5>)/i', '', $divline); 
	echo $divline." ".$line."<br>";
	$html->clear();
	unset($html);
}
?>

[Derokorian]

You should properly indent your code, As it looks right now its not easy to tell where that closing brace pairs up. Also you can simplify the preg_replace to be one line. Have a look at this simple change:

include('simple_html_dom.php'); 
$lines = file('array.htm');
foreach ($lines as $line) { 
    $html = file_get_dom($line); 
    $divline = $html->find('div[id=game_440] h5', 0); 
    $divline = preg_replace('/(<(\/)?h5>)/i', '', $divline); 
    echo $divline." ".$line."<br>"; 
    $html->clear(); 
    unset($html);  // I don't think this is necessary as you would overwrite the variable on the next iteration anyway.
}

[johanafm]

My only point was that if you need help ask for it. And should you get no useful answers or no answers at all, perhaps also include information you have but we lack. Such as error messages, what something in particular is, where it's not working, what you expected, what you got, any things you've tried so far and what you expected them to do but didn't, information about what is taking "a long time" and the code involved in that part or those parts...

BASiQ;10995586 wrote:

you were right, did not need that line

Since it does nothing. Well it does create a string literal containing your variable and then use string interpolation to find the variable in that string and replace it with its value before assigning it to a new variable, thus being equal to $var2 = $var1.

BASiQ;10995585 wrote:

I am not sure if you are being sarcastic, or assume I know more then I do.

I made no assumption about how much you know, but I most definitely assumed you knew more than I did, which apparently was a correct assumption, since you a liitle while later posted this

BASiQ;10995586 wrote:

so now, only this line is giving this error when using simple_html_dom.php
"Fatal error: Function name must be a string in /home/MySite/public_html/parse/index.php on line 47"

echo $html('div#game_440 h5', 0)->getPlainText();

which would have prompted me to ask for a vardump on $html for starters should you have asked for help about this.

BASiQ;10995586 wrote:

echo $html('div#game_440 h5', 0)->getPlainText();

it still works with ganon but is still quite slow.

And here I'd still make no assumptions about how much you do or do not know, but I still assume you know more about ganon than I do, since I know nothing about it. Since you give no further information than "it's quite slow", I'm among other things assuming that you know where it's too slow, why it's too slow and how to fix it, since certainly you would have otherwise profiled it and told us. Oh, and possibly also tell us what it is and post the code which is taking "too long".

[BASiQ]

New issue.

<?php
	ini_set('memory_limit', '-1');
 	include('simple_html_dom.php');
	$lines = file('array2.htm');                                                   //reads in the file with the list of user numbers
	foreach ($lines as $usernumber) {                                              //loops line by line
    $link = "http://steamcommunity.com/profiles/<sitenumber>/games?tab=all";   //link format
    $link = preg_replace('/(<sitenumber>)/i',$usernumber,$link);           //puts usernumber into link format
	$link = preg_replace('/\s*/m','',$link);                               //remove the stupid space
    $html = file_get_dom($link);                                               //calls the dom function
    $divline = $html->find('div[id=game_440] h5', 0);                          //finds the div block and then the h5 line
    $divline = preg_replace('/(<(\/)?h5>)/i', '', $divline);                    //replaces the <h5> tag with nothing
//
	list($hrs,$u1,$u2,$u4) = explode(" ",$divline);
	if ($hrs < 150) {
	    echo $hrs." - ".$link."<br>";                                          //prints results
	}
//
    $html->__destruct();
    unset($html); 
}

?>

This is my new version with a few upgrades. Main issue. Memory leaks
I run this and get this error. Any ideas what I am doing wrong?

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 2 bytes)

The only way to get this to work is to add a ignore memory line and run it locally using EasyPHP [version of WAMP]. It seems to run the server out of memory eventually otherwise

	ini_set('memory_limit', '-1');

[johanafm]

I don't really see any issues with your code that should cause you to run out of memory, at least not unless you are dealing with huge files. And since http://steamcommunity.com/profiles/76561198010758659/games?tab=all is only 74KB, your 128MB limit should be enough.

As such, I'd guess that the issue is with simple_html_dom. If you need to see what happens, use [man]memory_get_usage[/man] and echo memory usage to see where and when it's used up.

But if the issue is indeed with simple_html_dom, why not use something that works, at least as far as I know, such as PHP's built in [man]DOM[/man] functionality.

How to do the same thing using DOM. Maybe worth seeing if this works.

# The html file is full of markup errors
# This prevents those errors from being output to screen / sent to errorlog
# If you wish to inspect them, use libxml_get_errors()
libxml_use_internal_errors(true);

$lines = array('76561198010758659');
foreach ($lines as $usernumber)
{
	$link = "http://steamcommunity.com/profiles/$usernumber/games?tab=all";
	$d = new DOMDocument();
	$d->loadHTMLFile($link);

$x = new DOMXPath($d);
# This XPath expression matches your CSS selector
$l = $x->query("//div[@id='game_440']//h5");

# Unless there can be zero or several h5 per id="game_440", this is always 1
if(count($l))
{
	$hrs = (int) $l->item(0)->nodeValue;
	if ($hrs < 150)
	{
		echo $hrs." - ".$link."<br>";
	}
}
}

A few suggestions on improving the initial script

ini_set('memory_limit', '-1');
include('simple_html_dom.php');
$lines = file('array2.htm');												   

foreach ($lines as $usernumber)
{
	# You claim this should remove "the stupid space", but is there really a space in
	# $usernumber? If there is, do it directly to $usernumber
	# If there isn't, then there isn't in the link below either and you can remove it
	# completely. Moreover, the /m modifier should not be needed
	# $link = preg_replace('/\s*','',$link);							   

/** Unnecessary work
$link = "http://steamcommunity.com/profiles/<sitenumber>/games?tab=all";
$link = preg_replace('/(<sitenumber>)/i',$usernumber,$link);
* since you can simply...
*/
$link = "http://steamcommunity.com/profiles/$usernumber/games?tab=all";

$html = file_get_dom($link);
//finds the div block and then the h5 line
$divline = $html->find('div[id=game_440] h5', 0);

# I'd get rid of $html as soon as possible, but unset($html) should be enough
# since that should automatically call the destructor. Still, I don't really see
# why this would be enough to cause you to run out of memory.
unset($html);

# maybe replace the pattern delimiter / with something else
# and remove ( and ) to make it more readable
# $divline = preg_replace('#</?h5>/i', '', $divline);

# However, since you know exactly what you want to remove, <h5> and </h5> respectively,
# and they never are changing or depend on anything else, it's more efficient to
$divline = substr($divline, 4, -5);

/* Since the line starts with a number and that's all you want to keep,
 * it's simpler to just
 * list($hrs,$u1,$u2,$u4) = explode(" ",$divline);
 */
$hrs = (int) $divline;
if ($hrs < 150)
{
	echo $hrs." - ".$link."<br>";										  //prints results
}
}