fetchin urls with curl and listing them

[dilbert2010]

good evening dear php-experts,

I'm pretty new to php programming especially.

I would like to scrape the web finding a certain string using curl. I've been trying setting different user agents, and setting other options but I just can't seem to get the urls of that pages, I believe it has something to do with the fact that the query string gets encoded somewhere but I'm really not sure how to get around that.

Nonetheless, I'm trying to get done a very simple script for retriving data web and parsing.i think that this is a a great strarting point to for beginners on weekend codes. but at the moment i got troubles.

i need to fetch all the urls that contain this term;

this term: /participants-database/

//$url is the same as the link above
$ch = curl_init();
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0'
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch,CURLOPT_CONNECTTIMEOUT,120);
curl_setopt ($ch,CURLOPT_TIMEOUT,120);
curl_setopt ($ch,CURLOPT_MAXREDIRS,10);
curl_setopt ($ch,CURLOPT_COOKIEFILE,"cookie.txt");
curl_setopt ($ch,CURLOPT_COOKIEJAR,"cookie.txt");
echo curl_exec ($ch);

i need to fetch all the urls that contain this term;

/participants-database/

Well - Probably this kind of stuff is trivial in PHP. Here PHP is a great language - a good language.

love to hear from you

dil-bert

[Weedpacket]

Fetch the URLs from where? I've got an advertising flyer under my coffee cup. It has some URLs on it. None of them contain the string "/participants-database/" though.

[dilbert2010]

hello dear weedpacket

many thanks for the reply. you see that this is just anohter approach - to the one that we have discussed one day ago. I am trying to dive into programming in Perl / PHP with some real world approaches.

These tasks are great starting points.

Weedpacket;11064737 wrote:

Fetch the URLs from where? I've got an advertising flyer under my coffee cup. It has some URLs on it. None of them contain the string "/participants-database/" though.

weedpacket: Youre right: well i think that i fetch alot of URLs, - afterwards i have to try to iterate through the result to find ` elements: I've tried my parser code on a single cURL and it works (returns an array with the URLs ).

Well - first of all i have to develope a cURL code that fetches all the urls -

well - the task in general is something like the following

$urls = Array(

 'http://www.example1.com/foo_bar/1.htm',
 'http://www.example2.com/foo_bar/2.htm',
 'http://www.example3.com/foo_bar/3.htm',
 'http://www.example4.com/foo_bar/4.htm'
 'http://www.example5.com/foo_bar/1.htm',
 'http://www.example6.com/foo_bar/2.htm',
 'http://www.example7.com/foo_bar/3.htm',
 'http://www.example8.com/foo_bar/4.htm'

 );

weedpacket - well youre right:

now i have to find a regular expression that filters out the foo_bar - in other
words - helps to find each URL that contains the _foo_bar

And - we can express that foo-bar terms in other words: we can express this as the important thing...the string "/participants-database/"

love to hear from you