[RESOLVED] Pages partially loading

rogair

I've got a server running a bunch of LAMP architecture websites. These sites have been running smoothly for years. Just recently the server started truncating it's output. The page output just stops. No error messages, no warnings, just stopped HTML output. The longer the page is the more likely this occurs. In Firefox it results in a partial page. In IE it results in a page loading error (which of course is a very bad thing for my visitors). I've not changed configurations recently, and this is occurring on all pages. Has anyone encountered this before? Any ideas on how to troubleshoot?

Thanks,
Roger

Shrike

Have you looked in the server logs?

rogair

There is nothing in the apache error logs. What other server logs should I check?

MarkR

I recommend that you install two network monitors, one near the client, another near the server (by "near", I may mean, on the same physical box, or the same ethernet segment, or on the router or something).

Then if you can trigger the error, you can look at the packet dump and compare the two.

I take it there are no transparent proxies (or reverse proxies), or any other kind of proxies involved?

Can you reproduce this on your dev box?

Mark

rogair

Recommendations for network monitors and use? I have no clue in that arena.

As far as reproducing this error elsewhere, no. It's only occurring on this one production box, and I've never seen it anywhere else. Ever.

MarkR

rogair wrote:
Recommendations for network monitors and use? I have no clue in that arena.

Ethereal. tcpdump if you want to capture packets somewhere and look at them with Ethereal later.

As far as reproducing this error elsewhere, no. It's only occurring on this one production box, and I've never seen it anywhere else. Ever.

Have you considered the possbiility that it's a hardware fault?

Are there any logs on any of the operating system logs; are there any unexpected reboots?

Have you considered simply replacing the server?

Mark

rogair

I've considered the possiblity it's hardware. Unfortunately I have no box to move this stuff to right now. I'd need to purchase a new surver and configure it, so I'm hoping I can avoid that. It seems that PHP is mangling the headers. I have not been able to reproduce the error with a plain html page that is not parsed.

Error:
500 Bad header: '</li>'
500 Not empty: 'ills/rings/C/ric001"><b><font size=+1>Front giant</font></b></a>'

No unexpected reboots.

rogair

Well I think I found the problem....
It appears I was hacked.
Found:
.pwdhack
.sniffer
in /var/log
Mucked with stuff a bit a rebooted, problem gone. Now I have to track down how I was hacked to prevent future reoccurance... Ugh.

MarkR

rogair wrote:
Well I think I found the problem....
Mucked with stuff a bit a rebooted, problem gone. Now I have to track down how I was hacked to prevent future reoccurance... Ugh.

No, the problem is not gone. You must take the machine offline immediately and replace it.

If a machine has been compromised, you MUST NOT continue to run it. It could have a zillion backdoors and be running a botnet and all sorts.

Please discontinue its use, take it offline, and restore from backups ultra-carefully.

Ideally restore all executable files (including PHP) from their original sources (e.g. your source code control), NOT from backups, as they could have been compromised and had backdoors installed.

Mark

rogair

I'm aware and am working on it.