[RESOLVED] Help With Generators

Bonesnap

Hey everyone!

So I've been experimenting with generators for the last couple days but I am not sure if I am doing it correctly or if my situation is one that would yield (heh...) the best results. Unfortunately I cannot post source code since it would probably result in me being fired or worse, but I think I can accurately describe my situation.

I am calling a web service provided by the client that returns a very large JSON object. I am applying json_decode() on it and then performing a foreach() and executing commands on each item in it (creating an object, checking some values, then inserting into a database with prepared statements). The issue is the memory footprint is quite large because I am assuming the JSON object returned by the web service has to be loaded into memory. I thought maybe I could utilize generators since the most obvious appeal of them is the very low memory footprint they require.

However, while I can get the generator to work and the script runs as it's supposed to (all expected entries are inserted into the database correctly), the memory footprint remains identical to what it was before. So I am wondering if I am doing something wrong or if simply my situation doesn't apply. I'll try to describe it better below:

Query web service and retrieve JSON response
Apply json_decode() on the response to create a standard class object.
Perform foreach() on said standard class object and yield each result
Call the function doing the yield and do the necessary actions on each yielded item

In fact, let me see if I can accurately describe this in some dummy code:

function retrieveData()
{
    // bunch of curl() stuff
    $response = curl_exec($ch);
    // check response, etc.
    foreach (json_decode($response)->d->results as $value) {
        yield $value;
    }
}

foreach (retrieveData() as $value) {
    importData($value);
}

function importData($value)
{
    // do stuff to the $value
}

I found an example similar to what I am doing (kinda) but the author is seeing the obvious memory footprint benefits, whereas I am not. Here's a link to the example: http://syrwebdev.com/php/2015/01/22/experimenting-with-php5.5-generators.html. Is it because my example is so much larger than the author's?

The memory before the script performs the import: ~367KB
The memory immediately after the script completes the import: ~200MB

For what it's worth this is being performed on PHP 5.6. We would love to do this on 7 but the client is running the SqlSrv extension which only has a release candidate out (just a couple days ago), so not ready for production yet.

Would appreciate any insight! Thank you!

Derokorian

Bonesnap;11056123 wrote:

function retrieveData()
{
    // bunch of curl() stuff
    $response = curl_exec($ch);
    // check response, etc.
    foreach (json_decode($response)->d->results as $value) {
        yield $value;
    }
}

This still has to decode and load the whole object into memory (actually twice, once for the variable $response, and once for the internal representation of the decoded object). I'm not 100% certain, but this may reduce the footprint a little bit:

function retrieveData()
{
    // bunch of curl() stuff
    $response = curl_exec($ch);
    $response = json_decode($response)->d->results;
    // check response, etc.
    foreach ($response as $value) {
        yield $value;
    }
}

No promises though, but you're replacing the entire string of json and the entirely decoded object with a partial set, though during the decode process both the entire object and string will be in memory, but it may not hold on to as much at least?

Bonesnap

Thanks for the reply, Dero!

I modified the script to mimic your example. Unfortunately it had no impact on the memory footprint. When I modify my query to the web service to pull an inordinate amount of records (enough to intentionally induce a fatal memory error), the line is always on the json_decode(). So I suspect that my situation isn't applicable, and that the enormous memory footprint is solely due to the sheer size of the response from the web service. Unfortunately not much I can do about that**.

Comparing my situation with the example I linked, the one difference I can see is I am not returning an array from the initial foreach() - I'm simply looping through it and doing what I want. I think the generator approach is only applicable if I had planned on creating derived array from the initial "set", which I am not.

** So the web service returns a lot of junk along with the actual data I need. I have been trying to use a regex to isolate/replace certain junk values before json_encode()ing it, but unfortunately I am not having any luck. I very rudimentary example would be the following:

{
    "d" : {
        "results" : [
            {
                "__metadata" : {
                    // contains three entries (nodes?) I don't care about
                },
                // the actual data I care about
            },
            // the rest of the entries I want - each includes a useless "__metadata" entry/node
        ]
    }
}

I would love to perform some kind string replace on that pattern to remove the "__metadata" entry entirely, or even remove the contents inside it - anything to reduce its size. I have already confirmed with the client that the web service cannot avoid sending it with the response :glare:.

I tried this regex and some variations of it to try to isolate the contents inside the metadata entry: /{"metadata":{(.*)},/ Unfortunately my regex strength isn't up to par. I figured I'd get an array of matches of whatever is inside the brackets, but instead I just get a huge dump of stuff, some what I need and some I don't (some of it is the actual data I need).

Weedpacket

Well you can search for the "__metadata" keys and collect their locations, find the following '{' and scan forward through the text for the matching '}' (counting '{' and '}' as you go) to find where the node ends. including the following ',' if it's not the last node, then substr_replace().

If you wanted to use regexps you'd need to use the \R pattern to make the match recursive (by themselves, regexps can't count so you can't use them to match things that might be arbitrarily nested).

If you wrote your own JSON parser you could hook into it how you want: skim through the "boring" parts to maintain the parser's state but not bothering to record the actual content, and only recording the "interesting" content. You could do this in a generator so that each time you capture a complete "interesting" node you can yield it right then and there.

Bonesnap

Thanks for the reply, Weedpacket.

I realized the values within the metadata key was pretty constant, so I used a series of preg_replace() and str_replace() functions on it enough to get it down to simply "metadata":{},, which I was then able to easily apply str_replace() on. It added about 20 seconds to the script but the memory usage dropped to ~142MB, so that's kinda neat.

A custom parser would be awesome, but unfortunately there's no time to write one; I had to upload the scripts to the client's QA server last night so it's working as-is.