SEO, converting dyanamic URL's to static

boombanguk

Hi,

Bit confused by all this but here goes. I'm currently in the process of updating my website (www.flashculture.com). I want to bring in most of the content from a mysql database, via php. What i'm concerned about though is google indexing of that content. Im confused as to how all this works, as I keep reading that spiders don't index dynamic too well, and to make things better you need to either convert your dynamic URL's to static ones, and/or use a sitemap? (not sure how to convert dynamic URL's to static btw)

What I ideally want on the home page, is some tabs, which will feature all the latest news/articles URLS store in the database. People can click on various tabs and that will pull down another lot of URLS (all in the same home page). How would indexing work with this approach? Do I just create a huge sitemap with links to all possible other links i'm storing in the database?

Any advice is appreciated!

boombanguk

etully

So that you understand SEO better, let's take this to non computer terms.

Say you own a restaurant and you want the local newspaper to do a review of the food you sell. But you don't print a menu. In your restaurant, people ask for the food they want and you either tell them you don't have it or you serve them their meal. This is dynamic. They give you data. You give them an answer back (food or no food). But the problem for the newspaper is that they want to review your restaurant without saying the names of 500 million possible foods to you.

So let's compare that to a DB driven web site. You have pages that are seen like this:

http://www.flashculture.com/display_content.php?x=7162534

What is Google supposed to do? Pass in millions of possible values for X and see what comes back? That's a waste of their time and yours. This is why you have a menu - a site map.

Sometimes Google will follow a URL like the one above. sometimes they don't. If you make an <a href> link with the URL, they will often follow it. If it's in a <form>, they will often not. To increase the chances of getting them to follow it, you need to remove the question mark. That's the part that scares them.

Let's take this again to non computer terms: You know sometimes you go to a restaurant and you order a hot dog but the waitress looks at the kitchen and calls it something else? This is a little conversion thing that the waiter does so that you can think of it as a hot dog but the kitchen can know that they are grabbing product 432 off the shelf. You can do the same with Google.

You can make your link look like this:

http://www.flashculture.com/convert-chicago-cubs_x-12345-y-87654.html

So Google thinks it's going to a normal HTML file that must have something to do with the chicago-cubs... But when your server hears that request, mod_rewrite converts that request to:

http://www.flashculture.com/display.php?x=12345&y=87654

Apache on your server doesn't know or care that the outside party was asking for some other URL... it just displays this page. The outside party doesn't know or care that this is a dynamically built PHP page... it's just happy to get content back. Just like at the restaurant where the behind the scenes details are hidden from the customer who just wants a hot dog.

So learn mod_rewrite and you will be able to make your dynamic URL's look like static URL's for Google's sake. Then make a site map full of these static URL's and Google will likely follow them.

NogDog

Google has no problem following a link such as www.yoursite.com/page.php?id=12345. It may, however, have a problem following (or ignore) highly complex query strings like www.yoursite.com/page.php?cat=foo&subcat=bar&loc=12345&part=abc123&search=this+that+and+the+other+thing.

If your page URL's fall into the first (simple) category, I would not worry too much about it. If they fall into the second (complex) category, then it is probably worth looking at things like using Apache's mod_rewrite to simplify them.

For lots more info, see http://www.google.com/support/webmasters/bin/answer.py?answer=35769 (and note that it uses a URL query string 🙂 ).

Roger_Ramjet

One point that google also make is that a uri that relates directly to the content will get a better page ranking. So if the content is about the chicago cubs it is better to use
http://www.flashculture.com/convert-chicago-cub.html
rather than
http://www.flashculture.com/display.php?x=12345&y=87654

Meaningful plain-language urls are also easier to remember and type.

boombanguk

etully wrote:
So that you understand SEO better, let's take this to non computer terms.

Say you own a restaurant and you want the local newspaper to do a review of the food you sell. But you don't print a menu. In your restaurant, people ask for the food they want and you either tell them you don't have it or you serve them their meal. This is dynamic. They give you data. You give them an answer back (food or no food). But the problem for the newspaper is that they want to review your restaurant without saying the names of 500 million possible foods to you.

So let's compare that to a DB driven web site. You have pages that are seen like this:
http://www.flashculture.com/display_content.php?x=7162534
What is Google supposed to do? Pass in millions of possible values for X and see what comes back? That's a waste of their time and yours. This is why you have a menu - a site map.

Sometimes Google will follow a URL like the one above. sometimes they don't. If you make an <a href> link with the URL, they will often follow it. If it's in a <form>, they will often not. To increase the chances of getting them to follow it, you need to remove the question mark. That's the part that scares them.

Let's take this again to non computer terms: You know sometimes you go to a restaurant and you order a hot dog but the waitress looks at the kitchen and calls it something else? This is a little conversion thing that the waiter does so that you can think of it as a hot dog but the kitchen can know that they are grabbing product 432 off the shelf. You can do the same with Google.

You can make your link look like this:
http://www.flashculture.com/convert-chicago-cubs_x-12345-y-87654.html
So Google thinks it's going to a normal HTML file that must have something to do with the chicago-cubs... But when your server hears that request, mod_rewrite converts that request to:
http://www.flashculture.com/display.php?x=12345&y=87654
Apache on your server doesn't know or care that the outside party was asking for some other URL... it just displays this page. The outside party doesn't know or care that this is a dynamically built PHP page... it's just happy to get content back. Just like at the restaurant where the behind the scenes details are hidden from the customer who just wants a hot dog.

So learn mod_rewrite and you will be able to make your dynamic URL's look like static URL's for Google's sake. Then make a site map full of these static URL's and Google will likely follow them.

Thanks the reply! (and everyone elses!).

ok I get that, but what I want to do, is to have some tabs on my home page, and when you click on a tab, it refreshes the page and displays a different set of data from the mysql database. But the page URL still stays www.flashculture.com/home.htm or whatever, its just content inside changes. So how would a spider gain access to that data? because its not on seperate pages, its just pulled from the database, and displayed on the same page depending on what tab is used?

Weedpacket

Generally speaking, spiders do not execute Javascript (off-hand I don't know of any that do). If what happens when you click on a tab is that some javascript executes to alter the content of the current page then in that case spiders won't try following those links (have you ever seen a "javascript:" page come up in a Google search?).
If you're talking about (i)frames and the links reference http: URLs, then the spiders would follow them, indexing the content as being at that URL.

MarkR

URLs with query strings in are possible for spiders to process in general.

But there are some rules which can help:

Give each page one and EXACTLY ONE URL. This means:
Don't point more than one DNS entry at the web site (this includes www.example.com/example.com) - if you have additional domains, get them to do 301 redirects (see below)
Make sure that any aliases for the page do a redirect
Always link to the canonical name for the page, not a redirect, if possible
Do not rely on Javascript navigation techniques (as pointed out by Weedpacket). You can still USE Javascript, but don't require it to find your pages.
Don't make your site excessively deep.
Do create a site map if you feel it's necessary.

A site map should not be necessary, but on some sites it is very helpful.

Another tip is to send 301 (permanent redirect) redirects from any old or incorrect URLs for your pages. But you MUST ensure that nonexistent pages are served with a 404 (Not found) error. Otherwise as far as a spider is concerned, any broken links just link to lots of similar but valid pages - it will rapidly get bored of looking at lots of identical pages and give up.

Mark

boombanguk

sorry if this sounds a dumb question, but......so theres no way other than using JS to alter the content (say by hitting a tab button) within a php page? wouldn't it be possible to make the tab button link to the same page again, but with a different search criteria? (so all the same content is displayed apart from a different tab configuration containing different data) ?

NogDog

boombanguk wrote:
sorry if this sounds a dumb question, but......so theres no way other than using JS to alter the content (say by hitting a tab button) within a php page? wouldn't it be possible to make the tab button link to the same page again, but with a different search criteria? (so all the same content is displayed apart from a different tab configuration containing different data) ?

Yes, it's entirely possible. Make the tab element a link, and have the link use a URL query string like:

<a href="page.php?search=<?php echo urlencode($search_string) ?>">Link Text</a>

When the link is clicked, the page.php page will see the search value in $_GET['search'] which you can then use to populate the page contents.

Weedpacket

Another possibility is to write the link so that it works with and without JS.

For it to work without Javascript it obviously has to have the desired page in the href= attribute. You won't get it loading within the page, but as a page of its own, of course, but if you could have everything you wanted without Javascript you wouldn't need Javascript. This means the response to a non-JS request would have to be an entire page, not just the new content. In short, you'll need some way of recognising at the server which sort of request was made.

Meanwhile, to get it to do something different when Javascript is available, add an onclick= attribute and put a call to the script there. Tack "return false;" on to the end so that the client doesn't then go on to follow the href-specified non-js link as well.

boombanguk

but using JS means no spider tracking of the new links that are displayed correct?
I think just displaying the same page with different content will work fine, I'll have a Flash movie playing, which will obviously reset, but not much I can do about that.

one thing though, regarding sitemaps. Lets say I had my tab's all done in Flash (a Flash movie, which pulls the content from the mysql database). Obviously the google spiders couldn't get to those displayed URLs wihch have come from the database, but! if I had a sitemap that contained all those URL's (which are external URL's that Im keeping in the database), would the spiders be ok with that?

or am I losing the plot completely when it comes to what needs to be spidered and what doesn't? my site (www.flashcultulre.com) contains lots of external URL's, I thought it was good to have all those out going URL's spiderable (is that a word?) by the google bots, but is it not that important? is the only thing that matters links to my own internal pages? (for ranking)

bradgrafelman

boombanguk wrote:
but using JS means no spider tracking of the new links that are displayed correct?

Pure JS? Correct. Weedpacket, however, was talking about a way to use JS + HTML to allow spiders to correctly crawl the linked pages though maintaining the JS funtionality.

boombanguk

one thing I was curious about is sitemaps? would it be possible for me to create the site in Flash(or certain parts) but build the sitemap dynamically from the data thats in the mysql database? so the user sees the Flash, and I create a sitemap of all the links for the spiders to follow? would that work well?

The only thing that bothers me about that approach is that google says they still try to index your website the "normal" way even if you do have a sitemap, so i'm not sure how useful a sitemap really is!

thanks for any advice.