Services Web Site & Spiders?

[dgp0961]

I am in the process of building a new web site that provides a listing of services (salons, dry cleaners, etc...) for a given City/State. The goal is to provide a site where people can find services in their area without having to use Google and scroll through hundreds of invalid links.

I would like to populate a database with a listing of all these services that were found doing searches and have the new web site query this database and display the results for a given City / State.

Here is my question - how can I automatically populate this database? Do I need to write a spider to do this? If so, should I use PHP for the spider? If not a spider, what other methods can I use?

Thanks for your help....

[aaron_karp]

It would pretty easy to spider citysearch, based on their query string and HTML structure...

But you didn't hear it from me.

You might also be able to buy a database of yellow pages style data somewhere. I mean, Google had to get it somewhere, right? I doubt they stole it.

[MarkR]

Sounds like you are trying to rewrite Google to me. It's unlikely that you'll do such a good job as them though (They are clever people and have a huge amount of funding).

My experience trying to write a spider suggests that it's far from easy - look at some of my previous posts for details.

Mark

[dgp0961]

aaron_karp wrote:

It would pretty easy to spider citysearch, based on their query string and HTML structure...

But you didn't hear it from me.

You might also be able to buy a database of yellow pages style data somewhere. I mean, Google had to get it somewhere, right? I doubt they stole it.

I assume that using a spider on citysearch is not legal based on your comment?

[aaron_karp]

I'm no lawyer, but check out the second paragraph (Permitted Use): http://seattle.citysearch.com/saved/html/my_citysearch/terms_of_use.html?

On the other hand, it IS public data. But citysearch has done to work of compiling it, and in my eyes, at least, that entitles them to some legal protection.

[dgp0961]

aaron_karp wrote:

I'm no lawyer, but check out the second paragraph (Permitted Use): http://seattle.citysearch.com/saved/html/my_citysearch/terms_of_use.html?

On the other hand, it IS public data. But citysearch has done to work of compiling it, and in my eyes, at least, that entitles them to some legal protection.

Thanks for the follow-up. I will leave citysearch alone.

[aaron_karp]

My recommendation would still be to try and purchase a list of data somewhere. I looked quickly on google and couldn't find anything, but if you dig for a while you might find something.

Scraping other people's might get you into trouble. Although it probably wouldn't.

[dgp0961]

Thanks - I'll start looking for some data to purchase...