Why is my fast new server slower?

sneakyimp

I have a live site, let's call it mydomain.com. The site has been up for 46 days and served up about 1.5 million page views last month which means about 8 million hits.

I'm setting up a new server with a faster CPU (which happens to be dual-core), more RAM, 3 hard drives instead of one, and a 100 Mb/s network interface instead of 10Mb/s.

I've copied my site code over to the new machine which I can access by visiting server2.mydomain.com.

For some reason, the pages seem to render more slowly on the new machine even though it should be faster. In particular, the new machine appears to load images more slowly. Why might this be?? I can clear the cache in my browser, visit the same page on both sites and the live machine (a 2ghz celeron) appears to serve the images more quickly than the hot new box (2.2ghz dual-core AMD chip). Might this be due to some DNS weirdness as in the the old machine is a more widely visited IP address than the new machine? Is there some way to benchmark this sort of thing?

NogDog

Not my area of expertise at all, but I might start by trying some ping tests to see if same route is being taken to each machine, and what the lag time is for each test. This might at least indicate or eliminate networking issues as part of the problem.

sneakyimp

How does one do that?

NogDog

To get the basic response time, open up a cmd window and just type "ping hostname.com", then look at the timing information it returns. If you want to see what route was taken to get there, type "tracert hostname.com".

PS: This is assuming testing from a Windows PC, not sure how it might differ on Linux or Mac.

cahva

You didnt mention what webserver you are using. If using apache, check the value "HostnameLookups" on both servers. If its off on the live server and on in the new server that might explain the difference.

In general, you can turn it off. It can do great speedup for server. In the logs you will have only ip's for the visitors but if you want to have them in statistics(for example Awstats) you can set the hostnamelookup when updating statistics.

mpb001

Images indicate a possible bandwidth/network issue. From a windows box you can also use pathping which is a more intensive version of tracert. Takes a while to run but more informative. Use traceroute from linux.

The other thing to do if you are using MySQL is look in the my.cnf on the old server and the new and compare settings to see what they do.

DNS is cached by the operating system in Windows and also Firefox has a built in DNS cache as well. Once the domain name has been looked up it isn't again for a while.

Apache has hostnamelookup as mentioned before but also keep alive etc. Have a look in both httpd.conf files and do a quick scan for anything juicy.

sneakyimp

Thanks!

I'm using Apache. I checked the conf and hostnamelookups is off.

The two servers are both hosted by the same hosting company (OLM.net) so I would imagine that the network conditions are similar if not identical.

I'm pretty sure the speed issue is not related to mysql config but I will be checking that in some detail. The issue appears to relate solely to the speed of image downloads.

I compared the httpd.conf for the two servers and found a difference in the KeepAlive values but I'm not sure what it means. The old server:

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to " On" to deactivate.
#
KeepAlive  On

Note that it says 'set to "on" to deactivate which really doesn't make any sense.

And the new server had it off but also had a different directions...now we set it to 'off' to deactivate. Does that mean someone at apache got hip to the typo and fixed it? Or is the meaning of the var reversed?

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

Thanks for the tip...this may be exactly what the problem is! As for 'juicy' stuff...any more advice about what constitutes 'juicy' ?

bpat1434

could be different apache versions (2.2.6 vs 2.0.x vs 1.1.3)

Now, just to explain this, I'll just quote the manual:

Apache 2.2.6 Manual wrote:
The Keep-Alive extension to HTTP/1.0 and the persistent connection feature of HTTP/1.1 provide long-lived HTTP sessions which allow multiple requests to be sent over the same TCP connection. In some cases this has been shown to result in an almost 50% speedup in latency times for HTML documents with many images. To enable Keep-Alive connections, set KeepAlive On.

So in essence, your site will allow multiple connections from one IP so that images can be downloaded simultaneously instead of in-order. You really probably should turn this on unless you're expecting a huge load (like being Digged) and think the server might crash from millions of hits in a day with multiple connections per hit.

sneakyimp

You guys are awesome thanks so much. I should have checked the apache docs myself. I'm guessing the old server has a mistake in the conf file.

My tracert results indicate 27 hops to the new server and only 18 to the old. Can I expect that to improve over time?

bpat1434

It can change depending upon the dns servers used. So each hit may show a difference of like 5 hops or so. Not really sure if there's a way to get the fastest.

sneakyimp

Thanks again bpat. As I understand it, DNS is a pretty fluid process and the path to a particular server can evolve over time as more/fewer people access it. Is that right or just wishful thinking?

bpat1434

Umm... I'm not too sure. I'm not really hip to all that jazz that is DNS, but I know that your path from you to server can very from time to time depending upon which nameserver you get to respond and where that points to... but as for making it shorter, I think the shortest it could be depends on how many hubs between you and the server there are. But I can't be 100% positive.

MarkR

When you say "seems slower"

What tools did you use to measure it quantitatively?
Please post the results
I assume all other factors were the same
Did you actually performance-test this new server (or an identical one in your lab, or the physical one before it was shipped to the datacentre)
I am assuming ALL other factors are the same. Are all of the following the same?
OS (including minor version)
Web server
PHP version
Database version
Operating system and application settings, including Apache's config and PHP's config

If the answer to any of those is either "no" or "Don't know", then you have problems.

Mark

sneakyimp

So far, I have used no tools to measure it quantitatively. I asked about that in my original post: Is there some way to benchmark this sort of thing. I'd be really excited to know if there's some kind of client-side technique to test load performance.

And none of the other factors are the same.
- OS: Old machine is fedora core 1. New machine is CentOS 5
- Web Server: Old is 2.0, new is 2.2.3.
- PHP Version on old is 4.something. new is 5.1.6
- Database version: Old is 3.23, new is 5.0.22

It seemed like a really bad idea to try and stick with the old mysql and php and OS. I was under the (perhaps mistaken) impression that new software means better security. I have learned better than to expect better performance but still thought it might actually perform better with newer software.

The hardware is the big difference. The new machine is a) not a Celeron, b) is dual core, c) has twice the RAM, d) has a 10% higher clock speed, e) has 3 hard drives instead of one, f) has a network card with 10 times the thruput.

bpat1434

I don't know of any client-side scripts that will measure server-side load .... but perhaps you can just put at the top of some of the scripts a $start=microtime(); and at the end do $finish=microtime(); and then either log that, or display it at the bottom as "Page too XX.XX seconds to generate". That can give you an idea as to how fast it's going.

Your best bet is to SSH into the server and watch the physical load as you browse your pages. You'll be able to see what services are being bottlenecked. You should NOT be running Fedora Core 1 since, well, it's been out of commission for some years now (they're on 7 you know) and CentOS I find to be a great stable non-paid RHEL distro 🙂

sneakyimp

I wasn't looking for client-side scripts for server-side load, I was looking for client-side scripts to quantify total client-side loading time. Or a plugin or something for mozilla. I fully expect to do some performance profiling (perhaps with the auto-prepend and auto-append settings in PHP.ini). I'm nearly certain that the new machine parses the page faster, it's the actual download of the page that appears slower and I need some way to quantify this independently of server-side page parsing time.

I am fully aware that Fedora Core 1 is old! That's the reason for the new server upgrade. What I'm battling at the moment is that the folks I'm working with think the new machine is going to be slower so I have to be able to counter their skepticism with hard numbers detailing where the slowness is.

bpat1434

In Firefox you can use the Firebug extension to view the Net requests and see exactly what image or what file is taking a long time to download. But if you're having slow download times, it's probably in how your server is serving the page, or the physical internet connection not the browser or your scripting (unless it's your script takes an exorbitant amount of time to parse (hence the "Page took XX seconds to load" tag))...