PHP and Webcrawlers

Anon

Does anyone whether webcrawlers such as Google index .php pages? Or do they just index .htm files? Thanks.

Anon

As a matter of fact, I was curious about that myself, so I wrote AltaVista. One of there techs wrote back and explained that prettu much all search engines do.

There is a catch, if your refering to databases within side your url such as

http://www.bigdaddymama.com/index.php?rank=1

This will not be indexed, the engine stops here and moves on to the next.

Their are some great articles here in the website which cover this subject greatly. Enjoy!

php4hosting

If its just .php with no variables i.e. php?something=somthingelse then no "most" wont

Google does as do a few others

We have been running extensive tests recently and found out a lot about the various search engines.

We have some spider detection code that we are working on to aid the monitoring of spider activity.

Some will ignore .php or .php3 extensions so better to be safe then sorry use .htm

Regards
Darren
http://www.php4hosting.com

plutarck

While it's true you should be better safe than sorry, don't worry too much about using .php instead of .htm.

Chances are if the search engine sucks enough not to include anything but .htm, then it's not worth your time to try to get high in it's rankings.

Remember than being in the top ten of a few good search engines is better than being in the 100s of 1000 mediocre search engines.

The best search engines and directories list site pages of any extension, but they do ignore anything after "?", or at least the majority of them do. Evidenced by the fact that I've yet to see any site in the search engines I use have a "?" or "&" anywhere in them.

Of course one way to really boost your search engine ranking with webcrawlers is to use Cloaking software. As long as you don't try and fool the search engine into listing you for keywords that aren't relevant to your site, the search engine's probably won't even know you're using a good Cloaker, much less ban you.

Just while we are on the subject, don't try and use the little slick trick where you put keyword text with the same color as your page's background. It use to work, but many of the top engine's have installed filters to ban any offending sites. Most people don't think it's worth the risk.

Anyway, enjoy 😉

Anon

To everyone: thank you for the responses.
What's Cloaking software?

plutarck

"cloaking" software is a special program that checks all inbound requests for pages and compares them to it's list. They are designed to find search engine crawlers, webbots, etc, and then generate a special page just for them.

It's a page that isn't very pretty or effective for normal people but it's been highly optimized for indexing. All the text are keyword laced, the title, the meta content, everything.

Incredibly good for boosting the heck out of your ranking, and chances are most highly competitive sites are starting to use them. As long as you don't try to use something like "xxx britney spears" as keywords on your PHP Tutorial website, you won't be banned.

So let that be a lesson to yah. If you say there are illegal naked pictures of celebrities on your site, there had darn well better be illegal naked pictures of celebrities on your site or you'll get in trouble.

Ironic, isn't it? 😉

Here's the only commercial cloaker I know of, but there are more:

http://www.ip-delivery.com/

It sounds incredibly cool, and it protects from things like Web Position Gold and other page-grabbers and it keeps your competition from studying your technique.

In the market I'll be entering (indie music, mp3, and CDs) high-ranking is hard to come by, especially considering the competitors are MP3, CDNow, and Amazon, so I expect to be using such software when the time comes.