also, keep in mind sometimes all it takes is for a website w/ high traffic or high pagerank to post a link to your website.
all of a sudden you will have tons of visitors, either directly from people following that link, or possibly due to googles opinion of how important your site is, leading to your website being a high ranker in search results.
an extreme example of this is commonly referred to as the "slashdot effect" or being "slashdotted".
slashdot.org is a website w/ extreme amounts of visitors, and when someone posts a link to your website, the amount of traffic resulting from it can bring your server to a crawl.
google for "slashdot effect" to read more
but really, the first thing you need to do is look at the log file, or if you have some visitor statistics to look at. take a look at where these visitors are coming from.