Hi there everyone!
I've written a search system on my site that utilizes keywords and weights to determine best matches. Unfortunately, I didn't write this until I had 1,000 items in the database already so I had to write a function to (re)build the searchpool of keywords and weights. This script goes through each ID in the item database and creates strings and weights for things like title, description, keywords, etc.
the script uses and updating progressbar that I'm using to monitor it's progress. When the script starts running, it's blazing fast. By about 100 into it, I notice that the performance has been probably cut in half. By 500 in, it's slower still and it just continues worsening as it continues. By record 1000, it's probably doing a record every three seconds.
I monitored the server load as the script ran and it correlates to the performance degradation of the script itself. The first few hundred didn't seem to impact the server at all and by the end, the numbers were quite high.
Since the script starts out running great and only slows as it progresses in a very uniform escalation, I'm wondering if it's the dynamic page / progress bar itself that's causing my issue. It utilizes php's flush to perform it's magic and I'm wondering if this might be causing the problem.
The page generation stuff:
echo'
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
<style>
body {
background: #f06d06;
font-size: 200%;
padding: 20px;
}
.className{
width:500px;
height:150px;
position:absolute;
left:50%;
top:50%;
margin:-75px 0 0 -250px;
}
</style>
</head>
<body>
<div class="className">
<!-- Progress bar holder -->
<div id="progress" style="width:500px;border:1px solid #ccc;"></div>
<!-- Progress information -->
<div id="information" style="width:100%; padding-top:20px; text-align:center"></div>
</div>
';
/* Time to do some actual work. */
/* Wipe out current searchpool data for this $ciid. */
$q_dsp = "DELETE FROM searchpool";
$r_dsp = mysqli_query ($link, $q_dsp) or die('Catastrophic failure [Super secret code 7219229]');
$i = 1;
$q_prim = "SELECT * FROM cart_items ORDER BY id ASC";
$r_prim = mysqli_query ($link, $q_prim) or die('Catastrophic failure [Super secret code 48155]');
$n_prim = mysqli_num_rows($r_prim);
$total = $n_prim;
while ($row_prim = mysqli_fetch_assoc ($r_prim)) {
$ciid = $row_prim['id'];
$title = $row_prim['title'];
$mpn = $row_prim['mpn'];
$description = $row_prim['description'];
/* Lots of stuff happening here. See below for actual searchpool stuff */
$percent = intval($i/$total * 100)."%";
$i = ++$i;
// Javascript for updating the progress bar and information
echo '<script language="javascript">
document.getElementById("progress").innerHTML="<div style=\"width:'.$percent.';background-color:#ddd;\"> </div>";
document.getElementById("information").innerHTML="'.$i.' records processed.";
</script>';
// This is for the buffer achieve the minimum size in order to flush data
echo str_repeat(' ',1024*64);
// Send output to browser immediately
flush();
// Sleep one second so we can see the delay
//sleep(1);
}
// Tell user that the process is completed
echo '
<script language="javascript">document.getElementById("information").innerHTML="Process completed"</script>
';
sleep(3);
echo'
<script>setTimeout("top.location = \'index.php\'", 2000);</script>
<meta http-equiv="refresh" content="0; url=/'.$vu_overpath.'?do=search-rebuild&alldone">
';
exit();And this is what the script is doing for each item ID: http://pastebin.com/AebZZEKq
Is this a terrible way to handle long running scripts that have a bunch to do? My next thought is to run the script with simple echoing of an asterisk to show it's progress, stripping all the progress bar/ php flush stuff out of the page to see if it performs better but I wanted to ask if anyone spotted a bad problem area before tearing it apart.
Any suggestions on resolving the issue would be greatly appreciated. Thanks for your time!
#no_one_is_perfect_especially_me).
