In over my head

Arivendel

Hi I'm a newbie in php, in fact im so new i ain't sure how im supposed to do this, most of my studies have focused on javascript and c++ and i didn't really touched HTML interactions.

Now I'm on my OJT (On the Job Training) and have been given a project I'm not really sure how to code.

I've been tasked with the creation of a web spider, this spider should be able to open a specific website and grab all the hyperlinks found there, then it should add said links to a database, i have some ideas regarding the database part so im not really concerned there, the problem is im not sure how to script the crawler itself, could anybody please give me a sample code that would allow the script to access a site and read its contents?

I know for senior programmers this job is a 30 minute code but for me I dont really know where to start, and help would be appreciated.

Thanks.

1042

Arivendel wrote:
Hi I'm a newbie in php, in fact im so new i ain't sure how im supposed to do this, most of my studies have focused on javascript and c++ and i didn't really touched HTML interactions.

Now I'm on my OJT (On the Job Training) and have been given a project I'm not really sure how to code.

I've been tasked with the creation of a web spider, this spider should be able to open a specific website and grab all the hyperlinks found there, then it should add said links to a database, i have some ideas regarding the database part so im not really concerned there, the problem is im not sure how to script the crawler itself, could anybody please give me a sample code that would allow the script to access a site and read its contents?

I know for senior programmers this job is a 30 minute code but for me I dont really know where to start, and help would be appreciated.

Thanks.

This could be a nice start http://php-crawler.sourceforge.net/installation.html

Piranha

Arivendel wrote:
Now I'm on my OJT (On the Job Training) and have been given a project I'm not really sure how to code.

Arivendel wrote:
I know for senior programmers this job is a 30 minute code but for me I dont really know where to start, and help would be appreciated.

This is exactly why you should try to do it yourself. Since it is OJT I assume that they don't pay you to do it, but instead they let you do it as a part of your school work. To make sure that you learn it is much better for you to do it yourself, and ask if you have some specific question.

etully

The basic theory is that you write a script that goes out and obtains the contents of a web site and the contents get stored in a string. Examine the contents looking for anything that looks like a URL and repeat the process with this new URL.

There are lots of PHP commands that will obtain the contents of a web site. Look into fopen+fread, file_get_contents, or even fsockopen if you are feeling really adventurous.

Use regular expression functions like ereg, preg_match, preg_match_all, eregi, and string functions like explode to examine words on the page to see if they are URL's.

This is an excellent project to learn from.

Arivendel

@: Ill be sure to try that, thanks.

@: Im sorry if my request seemed wrongly explained, i do not want people to make the project for me, like you said i wouldnt learn from that, but since im not sure how to handle PHP samples that would give me an idea on how to do this would be great help (i am sorry if my post made it sound otherwise).

@: thanks so much for the list of possible commands to use, i really hadnt give the functions and expresions chapters a deep read yet and this would help me know what im suppose to look for.

Thanks so much for the feedback, i really appreciate getting the help to get me started in my project.