poor performance and bottleneck detection?

sneakyimp

So I have a site that's been up for a year now and we had a crazy burst of page requests about 2 weeks ago. 3 days in a row passed that had 5 times the previous record for page requests.

So i started wondering about whether my site is ok...whether it was slow...whether maybe that burst of page accesses was maybe rather my site barfing and angry visitors clicking their refresh buttons over and over when the site didn't respond.

== ITEM 1 ==
So I'm thinking at the very least I need a 'slow page detector' script. Every page on my site includes a single init file to set up all the application environment like database access and stuff. i figured maybe i would add something like this to that file:

$page_start = time(); // ADD THIS AT THE VERY TOP OF THE INIT FILE

// ALL THE APP SETUP CODE WOULD GO HERE

function slow_page_detector() {
  global $page_start;
  global $db;

  $page_total = time() - $page_start;
  if ($page_total > 5) {
    // create a database record for future analysis
    // that stores $page_start, the page name, the query string, and the execution time
  }
}
register_shutdown_function('slow_page_detector');

Is that useless or totally stupid? Anybody got a better idea.

== ITEM 2 ==
I would like to better understand what might cause the bottlenecks. Is it CPU? RAM? Hard drive? Surely there is a science to figuring this out...or diagnostics. Can someone suggest an approach or software that doesn't cost $1,500 that can help me locate my bottleneck?

MarkR

You should enable time logging in Apache. This logs the time taken for a request to be served (for large downloads, this may be affected by the user's connection speed however). You can then process this through some script to find slow requests.

Typically I'd say

Find the slow page(s)
Reproduce the slowness on your dev server
Install APD, the advanced PHP debugger, which contains a profiler
Look at the profiling data

If you're accessing a database, that could be the cause of slowness. Check the number of queries (by enabling query logging, ON YOUR DEV SERVER ONLY). If there are too many queries (100 is usually too many on a commonly used page) , consider a way of reducing the number. This could be by moving some functionality into a stored procedure, or reorganisation your DB. This may require a database design change which will affect your entire application.

If the problem is not due to too many queries, but a small number of very slow queries, consider optimising those queries- you may use a different query, add indexes or do a DB design change (again, this will affect your entire application).

If however, the problem lies inside PHP (the APD profiling info will tell you this), you need to redesign your page to optimise it.

If you can't reproduce the problem on your dev server, it's probably a load / RAM issue. Consult your OS documentation for how to measure memory, CPU load etc.

If you're using Apache on Unix, consider
- Not setting MaxClients too high - this is VERY important, as the amount of memory used depends on MaxClients
- Using a threaded (worker) MPM instead of prefork - be sure to test this well on your dev box, as there are some problems with this (but it works well for most apps)

There are loads of other things you could do too, but try those first.

Mark

sneakyimp

These are the most detailed tips I've gotten on this yet. Thanks, MarkR.

I'm going to try this, however, I am concerned about the performance of my server under actual user load. I don't know how to reproduce the situation where many users are accessing the server at once. Any tips about that?

Weedpacket

Apache ships with the ab utility; ab -n1000 -c10 http://localhost/foo.php will request foo.php from localhost a thousand times, with up to ten requests being made concurrently.

But have you checked your logs to see if it was caused by people doing a lot of refreshing, or was it a rather rude spider downloading the entire site?

MarkR

Apachebench is good but unfortunately it won't mimic real user activity- it never POSTs stuff, and it won't retain sessions, cookies etc, instead just hitting a given page a lot.

At some point you have to rely on data collected from your production box- but some types of profiling can't be enabled on production as they will slow it down too much (e.g. query logging, APD profiler).

Mark

sneakyimp

MarkR wrote:
You should enable time logging in Apache. This logs the time taken for a request to be served (for large downloads, this may be affected by the user's connection speed however). You can then process this through some script to find slow requests.

I was reading the apache 2.0 configuration directives and didn't find anything specifically about 'time logging'. I googled around and just found a bunch of software downloads. Are you suggesting I just alter the configuration to write a new access log?

logformat "%r %T" myformat
CustomLog logs/speed_log myformat

Obviously writing another log introduces some drag and would probably not be advisable on a production machine. What do you think of my original suggestion...adding that bit of PHP to my application which writes a DB table on really slow pages? I'm nowhere near 100 queries per page (which sounds like a lot! 100??!).

MarkR wrote:
If you can't reproduce the problem on your dev server, it's probably a load / RAM issue. Consult your OS documentation for how to measure memory, CPU load etc.

I have tried using the linux commands uptime, top, etc. to look at my load averages, cpu usage, etc. and this has been somewhat helpful but nothing seems to be redlining. it does seem to be digging into the disk drive a bit for extra memory. does this look bad?

             total       used       free     shared    buffers     cached
Mem:        765520     716512      49008          0      96160     315376
-/+ buffers/cache:     304976     460544
Swap:      1052248     101744     950504

i have one thorny problem i've been working on (with help from Roger Ramjet) which has a really slow query and doesn't seem to require much at all in terms of the resources I've been able to monitor.

In that case, MySQL seems like a black box. I have my suspicions but no definite idea why the query is so slow.

I'm looking into your MaxClients and MPM suggestions but:
does limiting MaxClients mean people will get turned away? That would be bad.
there's a dire warning HERE about PHP and threaded MPM. The whole concept of 'thread-safe' is new territory for me. I suppose I'll have to start another thread for that 🙁

Thanks for your tip about APD.

Weedpacket, thanks for the AB tip. I'm guessing if I ran that on the box that it would be talking to itself so I need to run it on another machine to get a true idea of how performance might be.

Also, any tips on sifting my log files? Scouring thru 1.2 million access requests seems like a tough chore. I found this:
http://www.analog.cx/

THANKS guys.

sneakyimp

and for anyone who cares to comment, i've attached some graphs of traffic...

MarkR

sneakyimp wrote:
Are you suggesting I just alter the configuration to write a new access log?

Yes, that's exactly what I'm suggesting- writing another log in parallel with your existing access logs /error logs.

I've done it loads of times, I normally use e.g.:

LogFormat "%h %t \"%r\" %>s %D" time 
CustomLog logs/time_log time

Here %D = microseconds - this is the important bit.

Obviously writing another log introduces some drag and would probably not be advisable on a production machine.

Writing one line on one file which is already open will not introduce a major load on a server running PHP. Even running "Hello, World" in PHP is quite intensive and involves opening several files for each request.

What do you think of my original suggestion...adding that bit of PHP to my application which writes a DB table on really slow pages?

That is going to involve more code AND add more load than using Apache.

I'm nowhere near 100 queries per page (which sounds like a lot! 100??!).

That's good to hear. I don't recommend that you enable all query logging on a production box - that's a disc killer.

does this look bad?

             total       used       free     shared    buffers     cached
Mem:        765520     716512      49008          0      96160     315376
-/+ buffers/cache:     304976     460544
Swap:      1052248     101744     950504

Not really. You've got 768M of ram, I assume, and about 100M of swap used, which is nothing really to write home about. The amount of "free" RAM is irrelevant seeing as Linux usually uses most of the free ram for disc caching.

I'm looking into your MaxClients and MPM suggestions but:
* does limiting MaxClients mean people will get turned away? That would be bad.

In principle, yes. But in practice, no. Remember that a web browser is only a client while it's loading the page or during keepalive (which is determined by your maximum keepalive setting in "KeepAliveTimeout"). Otherwise, just looking at an already loaded page doesn't count.

Once the MaxClients limit is reached, the OS will queue further connections and send them to Apache as slots become available.

Once the OS queue reaches its limit, further connections will lag, being retried on the client OS, until it eventually times out.

Believe it or not, this is a far, far better scenario than the server running out of virtual memory 🙂

* there's a dire warning HERE about PHP and threaded MPM. The whole concept of 'thread-safe' is new territory for me. I suppose I'll have to start another thread for that 🙁

I'm well aware of this dire warning, and am extremely pissed off that win32 users can have an allegedly stable threaded version of PHP, but Unix users can't (Remember that win32 doesn't have prefork mode). In fact I started a thread about it.

I ran PHP 5.1.2 with threaded (worker) Apache for many months with no problems, however I've now upgraded to 5.2.0 and am hitting thread-related issued, so I've switched back to prefork.

In any case, prefork is not a problem it just uses (much) more memory.

Don't set MaxClients too high- it is counter-productive and may cause your machine to run out of memory. It's better to have clients queuing for a free worker than your kernel swapping like mad to give the workers enough RAM.

If you have a look at the "server status" screen (Provided by Apache's mod_status) during a period of heavy(ish) traffic, you'll see how many workers are used at a given time. You'll probably be surprised at how low it is.

You may have 300k hits per day on your site, but you'll proably find that there are moderately long periods where none happen for many seconds or minutes (i.e. they're all bunched up)

Mark

sneakyimp

you think it's safe to write those time logs on the production server or is that a performance drag?

I've been reading a lot of docs on how to make my code thread-safe and I don't get the feeling it's going to help my current problem too much because my PHP is really light stuff. the real heavy lifting is done by the queries run through MySQL. They consume the vast majority of the execution time so it's obvious to me that's where my bottleneck is.

I have tried a bunch of different variants of my queries in order to find out what is causing the slowness (see here ) and I am on the verge of re-structuring how my application handles the data. It appears that it will be necesary to maintain a large number of redundant, 'distilled' data tables in order to get reasonable performance from my site.

Sadly, MySQL is in many respects a black box. What's killing me now is that the server doesn't appear to be working hard at all or using much RAM. I'll run TOP from an ssh window and order the running processes by cpu consumption with an update frequency of 0.1 seconds. then i'll call up a certain search page that typically takes 10 seconds to run, apache barely even registers as a CPU consumer and only appears when the page is actually loading. mysqld, although consuming 11% of memory, doesn't even register on the CPU rankings.

wtf? THERE IS WORK TO BE DONE, SERVER! I'm wondering if there's some kind of disk drive bottleneck or slow bus issue here. Also, I'm wondering of moving to MPM will affect whether mysql runs as multi-thread or single thread?

MarkR

sneakyimp wrote:
you think it's safe to write those time logs on the production server or is that a performance drag?

As I suggested before, writing the time logs on production is pretty safe. It only adds slightly to the work that needs to be done.

If you're paranoid, trying using some combination of SetEnvIf and only doing time logs on .php requests (non-php requests usually complete pretty quickly).

I've been reading a lot of docs on how to make my code thread-safe and I don't get the feeling it's going to help my current problem too much because my PHP is really light stuff.

You don't need to do anything, unfortunately the PHP authors DO. In 5.2.0 I've seen a major regression in thread-safety (as vs. 5.1.2).

Sadly because it doesn't seem to be a priority, their official position is still "PHP is not thread safe and never will be, you can run it threaded anyway but we don't care if it breaks"

wtf? THERE IS WORK TO BE DONE, SERVER! I'm wondering if there's some kind of disk drive bottleneck or slow bus issue here. Also, I'm wondering of moving to MPM will affect whether mysql runs as multi-thread or single thread?

MySQL runs in a separate process from PHP- it's multithreaded on all platforms in all cases anyway.

Can you reproduce the slowness of these queries on your dev server? This sounds like an important first step.

You can log which ones they are by using the MySQL slow query log.

Mark

sneakyimp

Is it possible to do a SetEnvIf based on the time of a page access - say if it exceeds 2 seconds or 5 seconds?

As for PHP's thread safety, what is so unsafe about it? Are there gobs of non-reentrant functions in the php collection? I've been having a difficult time imagining anything but search, compression, or video crunching that might need multi-threaded execution. Are you trying to use it on something in particular?

As for my troublesome queries, the slow queries are definitely reproducible. Fortunately they are in the beta section and not widely accessed. There is an extensive thread on my efforts to speed them up here:
http://phpbuilder.com/board/showthread.php?t=10331619

Roger has helped a lot but even he has apparently given up now. I think I should investigate faster hardware but don't really know where to start.