How do sites like dogpile and metacrawler work?

teguh123

I know they use search engine results from yahoo and google.

However, if I create a program to search Yahoo or Google, for example, sooner or latter, I'll get an anti automation message.

The thing is both Yahoo and Google has mechanism to prevent automated query from the same IP.

So how does dogpile and metacrawler did it?

So how do they avoid that? Is there a special API for programmers from yahoo.com and Google.com?

MarkR

Both of them do provide special APIs, yes. But the bottom line is, they presumably have an agreement with Yahoo, Google etc, which allows them to do this (thus they are spared the IP block).

Maybe they use screen scraping (or they originally did) but in any case, they have to have an agreement.

Mark

teguh123

Special API?

How to access that? I think I remember reading somewhere that Google hate automated search but they don't mind if people use the special API?

Weedpacket

Has it occurred to you to ask Google?
http://www.google.com/intl/en/about.html

teguh123

You mean, an actual person work in Google? I was doing adsense. Got banned, tried to appeal with absolutely no result or feedback on what's wrong.