Identifying search engines

mike_cal

my website is pretty much database driven and i've typically parsed the data in PHP and just output flat HTML. however, i'm trying to save some badwidth by spitting out just the critical information, and have javascript parse out the data.

however, i'm kinda worried how search engines would react to this. (especially google) would search engines ignore the keywords if they are inside javascript tags?

how do i detect search engines? maybe i can just setup a special script that output's "search-engine-friendly" output.

matt72

I'm pretty sure search engines still generally ignore javascript. If bandwidth is the only reason for this, maybe you should check out php's zlib or apache's mod_gzip and use compression instead.

etully

Just out of curiosity, how much bandwidth per month are you saving? Matt is right, the content is mostly invisible to the Search Engines.

bogu

If u dont like spiders: robots.txt let only thoes that u like ..

mobtex

it's not great to output search engine specific content as they might think that you're trying to dupe them with alternative content.

nor is it great to do this in javascript as most spiders will not be able to read or understand what it's saying.

really if bandwidth is such an issue the compression is a good idea. i ran some tests some time ago and using zlib compression saves stacks and stacks of data transit.

cut and pasted from my dodgy blog

To start i load our index page and check the page size which without compression comes in at 20.97 kb / 21,743 bytes

Now I add the following to our .htaccess:

enable zlib compression

php_value zlib.output_compression_level 1

Level 1 is the lowest compression, 9 is the most. The higher the compression the more the server will have to work, the harder the server works the slower it becomes so we're playing a balancing game here.

The Scores:

None - 20.97 kb / 21,743 bytes

Level 1 - 5.83 kb / 5,969 bytes
Level 2 - 5.71 kb / 5,843 bytes
Level 3 - 5.61 kb / 4,747 bytes
Level 4 - 5.34 kb / 5,465 bytes
Level 5 - 5.24 kb / 5,367 bytes
Level 6 - 5.20 kb / 5,326 bytes
Level 7 - 5.20 kb / 5,325 bytes
Level 8 - 5.19 kb / 5,316 bytes
Level 9 - 5.19 kb / 5,316 bytes

WOW

Level 1 is by far the most efficient in this test.

With the tiny differences in file sizes between level 1 and level 9 it's not worth the server load to go any higher than 1 but the difference between nothing at all and level 1 is stunning.

mike_cal

well, i already have my site compressed. i guess internet explorer is showing uncompressed size so i got a bit worried. i just checked firefox, and it seems i don't have to go javascript

my original was (average) 66Kb. and on javascript, it's 10Kb
while compressed, it's 7kb vs 3 kb...

still a 50% difference, but not that big in the grand scheme of things.