Remove duplicates from $dom->getElementsByTagName

[AndyGB]

My script below is design to extract links from a webpage, however what I am finding is some links are duplicated which is resulting in double the number of SQL queries the code is making. Any ideas how to remove duplicated links?

    $url = 'http://www.blablabla.com/';
    $dom = new DOMDocument;
    @$dom->loadHTMLFile($url);    

    $links = $dom->getElementsByTagName('a');

foreach ($links as $link) {
    // do stuff
}

[Derokorian]

Maybe this is as simple as calling [man]array_unique/man

[dalecosp]

I don't think array_unique() would work as $links is an object??

If not, I do something like this...

$seen_links = array();

foreach ($links as $link) {

   if ( ! in_array($link, $seen_links)) {

   // do my stuff

  // at the end
   $seen_links[] = $link;

   } // if
} // foreach 

[Weedpacket]

Might be expensive even if it does (since it's a lot harder to check if two objects are equal); but if only the URL is needed then those can be collected into an array and array_unique run on that.

[dalecosp]

Righty-oh, Weedpacket; would depend on structure and the expected size of things. It would really depend on whether you wanted to do the processing for all the links up front or "on the fly" as you were doing_something(), and that decision would probably depend on the expected number of links to be returned. Definitely something to be aware of if you're writing a system that will be expanded ... (I currently have one that started doing like, 50-200 links per run on average that is now being run, at times, against a million-line-item file ... learned that lessons quickly, I did 😉 )

[ammejohn10]

Hi all, Is this right to use Dom in Html because it's included with more whites spaces and represent an XML file. So sometimes it's generate errors.