save_array too large?

[boujii]

Hello and greetings. My first post of probably many i am sure 🙂

What I have here is an array that gets hung up when doing very large amounts of data.

if (isset($_GET['add_sec'])) {
  $sections[] = array(
    'top_x' => @$_GET['top_x'],
    'top_y' => @$_GET['top_y'],
    'bot_x' => @$_GET['bot_x'],
    'bot_y' => @$_GET['bot_y'],
    'pat'  => @$_GET['pat'],
    'seed'  => @$_GET['seed'],
    'seed2' => @$_GET['seed2'],
    'type'  => @$_GET['type'],
    'anim'  => @implode('|',$vAnimArray),
    'tree'  => @implode('|',$vTreeArray),
    'deco'  => @implode('|',$vDecoArray),
    'building'  => @implode('|',$vBuildingArray),
    'buyanim'  => @$_GET['buyanim'],
    'buytree'  => @$_GET['buytree'],
    'buydeco'  => @$_GET['buydeco'],
    'buybuilding'  => @$_GET['buybuilding'],
    'active' => @$_GET['active'],
    'place' => @$_GET['place'],
    'rotate' => @$_GET['rotate'],
    'walk' => @$_GET['walk']
    );
  save_array($sections, 'sections.txt');
}

I have quite a few of these arrays (different plugins) that do the exact same thing when it comes to writing a very large amount of data to the text file. They work fine when it's small to medium amount of data. Any ideas on this? Is there a cap size for the text file? Looking for any suggestions at all 🙂

[traq]

what do you mean by "large"?

what does save_array() do?

It would depend heavily on your server and what resources your hosting plan allows you, as well.

I can't help but wonder why you wouldn't use a database for this...?

[bradgrafelman]

I would echo everything in traq's post (especially the question about why you aren't using a database for this), but I would also ask what you mean by "gets hung up" ?

[boujii]

Sorry I am not the author of the plugin, and the author is MIA.
The plugin is used by a program. When the program is open, I can access it also via http://127.0.0.1:3001/plugins/Sections/main.php

When selecting just a few animals and save it writes the text fine. But when I check all animals and save it refreshes and comes back as a white screen (aka white screen of death).

[boujii]

By the way, it appears to be "flooding" out per say. When all animals are selected, and save, white screen shows instead of the normal load_array. It doesn't not even make the text file.

[bradgrafelman]

Enable error logging if it isn't already, make sure error_reporting is set to E_ALL, and check the error log for any errors. It's possible you're getting a fatal error due to being out of memory or something like that.

[traq]

boujii;11002297 wrote:

The plugin is used by a program. When the program is open, I can access it also via http://127.0.0.1:3001/plugins/Sections/main.php
[ ... ]
When selecting just a few animals and save it writes the text fine. But when I check all animals and save it refreshes and comes back as a white screen (aka white screen of death).

SO,

this is on your local machine, correct (not online)?

Are you using the plugin via the program you refer to, or in your web browser?

Is this "white screen of death" actually freezing your browser (or your computer)?

Or is it simply a blank page (no response from the plugin script)?

As brad says, start by enabling error reporting. From there, you need to try to narrow down what might be causing this --

how large are the arrays you're passing to 'anim', 'tree', 'deco', and 'building'?

How many elements are there in the $sections array?

Can you find the definition for your save_array() function, so we can see what it actually does?

[boujii]

bradgrafelman;11002300 wrote:

Enable error logging if it isn't already, make sure error_reporting is set to E_ALL, and check the error log for any errors. It's possible you're getting a fatal error due to being out of memory or something like that.

error_reporting = E_ALL & ~E_NOTICE
display_errors = On
display_startup_errors = On
log_errors = On
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = On

But having an issue locating the actual log. The php.ini itself is in the program root directory.

[bradgrafelman]

boujii;11002302 wrote:

But having an issue locating the actual log.

You'll want to examine the error_log PHP directive to determine that.

boujii;11002302 wrote:

The php.ini itself is in the program root directory.

And you're sure that's the one being parsed? Do a [man]phpinfo/man and verify 1) the settings you listed above coincide with the values printed in the tables, and b) the 'Loaded Configuration File' points to the same php.ini file you're talking about.

[boujii]

That helped TY. The php.ini was missing the error_log. And yes settings for phpinfo() are pointing to correct .ini
Now it is also making and writing the error.log

This is what I have as far as error reporting now.

error_reporting = E_ALL | E_STRICT
display_errors = On
display_startup_errors = On
log_errors = On
html_errors = On
output_buffering = 4096
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = On
error_log = error.log

Now I do have an error in the log showing
[19-Apr-2012 13:57:41] PHP Notice: Undefined offset: 1 in C:\2.1\parser.php on line 2862

But that is a completely different php and has no bearing on the current php I am working with. After clicking SAVE no error is logged by still returns white screen. Any more suggestions?

[boujii]

The thing I don't understand is I would think if there was an error in the code, it would error all the time, not just when doing massive amounts of data in the array. I mean, for instance, I could select 300 animals and work fine, but if more then 500 I get the white screen. So I would think it be a performance issue. Since no error is showing, perhaps the php is crashing? And like I said, this pertains to multiple plugins whereas selecting massive amounts of things for the array it happens. I tried changing a few things in the php.ini but no luck as of yet.

[boujii]

I seen this somewhere.. Think this could be what's happening?

"You're hitting the max number of array elements.
This is 'fixed' in PHP5 where there can be more elements in an array. It has nothing to do with the file size."

[traq]

No: there is no "max number of array elements."

PHP does have a maximum amount of memory it's allowed to allocate while executing. If your arrays are very large, you might be overrunning this limit while trying to write the values to file.

Likewise, if your computer is taking too long to complete the writing, you might be overrunning your max execution time.

[boujii]

traq;11002301 wrote:

SO,

this is on your local machine, correct (not online)?

Are you using the plugin via the program you refer to, or in your web browser?

Is this "white screen of death" actually freezing your browser (or your computer)?

Or is it simply a blank page (no response from the plugin script)?

As brad says, start by enabling error reporting. From there, you need to try to narrow down what might be causing this --

how large are the arrays you're passing to 'anim', 'tree', 'deco', and 'building'?

How many elements are there in the $sections array?

Can you find the definition for your save_array() function, so we can see what it actually does?

The program uses IE as it's browser. But once program is open, I can open it manually in any browser. When saving the array that is extremely large, result is same on all browsers (IE, Chrome, Opera).

It does not freeze me at all. Rather the white screen is instant. When it is saving a normal amount of data, it takes a few seconds, probably because it then load_array again as a refresh per say. The second I hit SAVE for the

    if (isset($_GET['add_sec'])) {
    $array_size = count($sections);
    for($i = 0; $i < $array_size; $i++)
    {
      $sections[] = array(
        'top_x' => @$_GET['top_x'],
        'top_y' => @$_GET['top_y'],
        'bot_x' => @$_GET['bot_x'],
        'bot_y' => @$_GET['bot_y'],
        'pat'  => @$_GET['pat'],
        'seed'  => @$_GET['seed'],
        'seed2' => @$_GET['seed2'],
        'type'  => @$_GET['type'],
        'anim'  => @implode('|',$vAnimArray),
        'tree'  => @implode('|',$vTreeArray),
        'deco'  => @implode('|',$vDecoArray),
        'building'  => @implode('|',$vBuildingArray),
        'buyanim'  => @$_GET['buyanim'],
        'buytree'  => @$_GET['buytree'],
        'buydeco'  => @$_GET['buydeco'],
        'buybuilding'  => @$_GET['buybuilding'],
        'active' => @$_GET['active'],
        'place' => @$_GET['place'],
        'rotate' => @$_GET['rotate'],
        'walk' => @$_GET['walk']
        );
	echo $array_size;
      save_array($sections, 'sections.txt');
      }
    }

It's white screen immediately. The sections.txt is also never even created.

I am trying to get a way to "count" by throwing an
$array_size = count($sections);
for($i = 0; $i < $array_size; $i++)
but have not worked out the echo yet for it. Sorry novice php lol.

Error reporting shows no error as it seems to crash immediately.

[boujii]

traq;11002326 wrote:

No: there is no "max number of array elements."

PHP does have a maximum amount of memory it's allowed to allocate while executing. If your arrays are very large, you might be overrunning this limit while trying to write the values to file.

Likewise, if your computer is taking too long to complete the writing, you might be overrunning your max execution time.

This I thought as well. So my current php.ini is for testing purposes.
max_execution_time = 3600
max_input_time = 3600
;max_input_nesting_level = 64 ; Maximum input variable nesting level
memory_limit = 2048M

This is run on a I7 with 9GB of memory on a windows 7 ultimate platform. Result is same on Windows XP dual core with 2gb memory.

OK the heart of the matter is:
This is a program for Farmville on Facebook. The program can contain several plugins. A lot of the plugins result in the same white screen when doing a very large array. Probably because they are written similarly I'm sure. So there are hundreds of people using this program (aka bot). So the white screen issue is very common. The actually program itself uses sqlite. But as you see the plugins are all coded to write to a text file rather then their own sqlite database.

[boujii]

Having a bit of an issue returning the count for the array. Always show 1

    if (isset($_GET['add_sec'])) {
      $sections[] = array(
        'top_x' => @$_GET['top_x'],
        'top_y' => @$_GET['top_y'],
        'bot_x' => @$_GET['bot_x'],
        'bot_y' => @$_GET['bot_y'],
        'pat'  => @$_GET['pat'],
        'seed'  => @$_GET['seed'],
        'seed2' => @$_GET['seed2'],
        'type'  => @$_GET['type'],
        'anim'  => @implode('|',$vAnimArray),
        'tree'  => @implode('|',$vTreeArray),
        'deco'  => @implode('|',$vDecoArray),
        'building'  => @implode('|',$vBuildingArray),
        'buyanim'  => @$_GET['buyanim'],
        'buytree'  => @$_GET['buytree'],
        'buydeco'  => @$_GET['buydeco'],
        'buybuilding'  => @$_GET['buybuilding'],
        'active' => @$_GET['active'],
        'place' => @$_GET['place'],
        'rotate' => @$_GET['rotate'],
        'walk' => @$_GET['walk']
	);
	$result = count($sections);
	print_r($result);
	//save_array($sections, 'sections.txt');
    }

echo count($sections, COUNT_RECURSIVE);
gives me 21 no matter how many animals I select

[traq]

...correct me if I'm wrong:

if (isset($_GET['add_sec'])) 
{
    $array_size = count($sections);

// we loop once for each of the existing $sections elements
for($i = 0; $i < $array_size; $i++)
{

    // we _add_ a new element with (primarily) the _GET values
    //    in fact, we do this _once_each_loop_ --
    //    we end up doubling the size of the array 
    //        (with half the array -all the new elements- being identical)!
    //    not to mention what weight those indices that contain imploded arrays might add
    $sections[] = array(
        /*  . . .  */
    );

    // we do this once per loop, too
    save_array($sections, 'sections.txt');
}
}

I can see this getting out of hand really quickly. On the other hand, I've been up a long time, and I need sleep.

I'll do some testing tomorrow to try to figure all this out.

[Weedpacket]

Incidentally, the size of a GET request is limited to what can fit in a URL.

[traq]

traq;11002335 wrote:

I'll do some testing tomorrow to try to figure all this out.

Confirmed

<?php
//  test case  //

session_start();

$sections = empty( $_SESSION['sections'] )? array( 'start' ): $_SESSION['sections'];

$array_size = count($sections);

for($i = 0; $i < $array_size; $i++){
    $sections[] = time();
}

$_SESSION['sections'] = $sections;

print '<pre>';
var_dump( $sections );
print '</pre>';

first run:

array(2) {
  [0]=>
  string(5) "start"
  [1]=>
  int(1334940119)
}

second run:

array(4) {
  [0]=>
  string(5) "start"
  [1]=>
  int(1334940119)
  [2]=>
  int(1334940146)
  [3]=>
  int(1334940146)
}

next:

array(8) {
  [0]=>
  string(5) "start"
  [1]=>
  int(1334940119)
  [2]=>
  int(1334940146)
  [3]=>
  int(1334940146)
  [4]=>
  int(1334940164)
  [5]=>
  int(1334940164)
  [6]=>
  int(1334940164)
  [7]=>
  int(1334940164)
}

As you can see, the result is doubling in size each run, and all of the new entries are identical.

[boujii]

It may be easier if you had the whole thing to look at.

http://75.147.75.181/farmville/Sections_v2.4.5.zip

The majority of it is in forms.php

I could easily send the whole program with plugin if you thought it be easier. But would require facebook/farmville account (i use a public dummy for testing if you needed it). Kind of hard explaining when it is sometimes easier doing it yourself 🙂

This is the normal process by selecting a few animals:

This is the normal process after clicking save:

This is the select ALL animals that causes the white screen when clicking save:

White screen:

I had once thought it had to do with writing the text file for
// save_array($sections, 'sections.txt');
But commented out everything functions the same. I believe it saving it to a text file is only for when shutting the bot down and reopening it. Since it seems to use the load_array. Just a hunch, but what do i know. Edit: That would seem correct, with save_array commented out, closing and reopening the bot starts fresh instead of loading the previous changes. So it's nothing to do with the writing of text file.

The animal array list in total is way to many for my to count offhand. I've done them in sets of 100 before adding about 4 sections (100 x 4) and looked like I could still go another few hundred. So the total array of animals when selecting ALL could be possible 800 to 1000. In the pic above you can see I have like 3 animals selected.