HELP with file(url) and 4.2.2 -> 4.3.0 migration issue.

[Damodred]

Hello,

I am interested in snagging a webpage, parsing it, and displaying the results. I
have written the code for this, and it worked on two servers, one running PHP
4.2.2 and one running 4.2.x (not sure which subversion).

Recently, one of the hosts has upgraded to 4.3.0 - most likely on the same day
(I don't get noticed about the upgrade, more's the pity), the script stopped
working.

It still works on the other one.

Now, this is where it gets weird. I use a file("url"), then iterate through the
result array to change it into a long string for parsing. I made it print
sizeof() the array just out of curiosity.

I find that the file() function gets only the first 51 or so lines from this
site. On the site that works, the whole thing is around 500 lines - obviously I
need all of the info to be able to parse it.

Any other page is obtained fine, the full length. I would think the host has
blocked me, but this behavior occurs on YET another site, completely unrelated.

So, are there any limitations in file() that I may not be aware of, particularly
when going from 4.2.x to 4.3.0?

Long winded I know, but the problem's kinda ... gnarly. 🙂

Thanks,

Marcin

[X_Gryphon_X]

I am looking for a solution to this same problem 🙂

[stolzyboy]

more than likely you are running into a register_globals problem, if your new version has it OFF, you will have to access your variables with

$POST['var'] //for posted forms
$GET['var'] //for forms with get method
$SESSION['var'] //for session variables
$COOKIE['var'] //for cookies
$SERVER['var'] //for server variables
$FILES['var'] //for uploaded files

[X_Gryphon_X]

As a test I am running something quite simple to try find out what is going on, I dont think register_globals has an effect on this :/

At test2.php I am running the following.

<?php
$lines = file ('http://myplanetside.station.sony.com/character.jsp?charId=435258&worldId=3');
foreach ($lines as $line) {
    echo "" . htmlspecialchars($line) . "<br>\n";
}
?>

Which should spit out all the source from myps.com/character.jsp but it is stopping after about 40 lines.

[stolzyboy]

well then it may be a fopen problem

http://www.php.net/manual/en/ref.filesystem.php#ini.allow-url-fopen

or a safemode problem

http://www.php.net/features.safe-mode

the code you supplied worked just fine, and my safemode settings are set to OFF, and my allow_url_fopen is set to 1

run a phpinfo() page to see what your settings are set at, this is likely your problem

[X_Gryphon_X]

I have the same settings :/

http://www.alliedtribalforces.com/myps/phpinfo.php

[Damodred]

Nope, no problems there either, at least I don't think so. Please consider:

http://www.1st-cavalry.com/test.php

this very page, 2000 some odd lines, and it gets sucked in fine.

http://www.1st-cavalry.com/test2.php
Sony's page, with the damned 41 lines. No matter whose character I try to get, it's 41 lines.

As you see, the problem is not with opening, or permissions. It's something perfidious, something else.

[stolzyboy]

post the full code for your test2.php, or is it what is up there already?

[stolzyboy]

nevermind...

you can view it here

www.cloverfish.net/file.php

[Damodred]

It's identical, except I echo a sizeof of the file()'d array in the first line.

[stolzyboy]

yeah, i noticed, look up a post :rolleyes:

[X_Gryphon_X]

There has always been a bump in that part of loading the page, even just browsing the site. 🙁 Driving me crazy. On my last server I had it parsing about 120 characters every 30 minutes no hiccups.

*edit
Haha Damodred, your original post was regarding MyPS? What a coincidence.

[Damodred]

Yep! I made a script to read my Outfit's stats daily, and parse it into sig images and a roster list. Worked fine, then quit after the upgrade to 4.3.0. Bleh.

I run it on my site instead:
http://www.damodred.net/allstats.php

I only ran it twice a day on the other server, I don't think they're blocking me?

[X_Gryphon_X]

No, I doubt that they are blocking you. Mine ran every 30 minutes for a couple months.

I have no other place to run mine as all my sites are on the same dedicated server :/

It is like at that bump in loading the page it thinks it is done loading. All my timeouts are set normal though.

Wish Sony would get off their arse and add xml heh 🙂

[Damodred]

Maybe the timeout for urls in 4.3.0 is broken? Gotta be a reason that 4.2.x works, but not 4.3 with the same setup...especially if you have access to your setup. I don't (shared hosting), so I thought maybe they changed something in the config, but if you do, than the issue is shadier still.

Yeah, xml would solve our problem rather instantly, although not the underlying weirdness.

[X_Gryphon_X]

I doubt that it is a timeout problem. Bleh, this is quite annoying. I think we tapped out stolzyboy's knowledge on this problem hehe.

[stolzyboy]

Originally posted by X Gryphon X
I doubt that it is a timeout problem. Bleh, this is quite annoying. I think we tapped out stolzyboy's knowledge on this problem hehe.

you may think so, but, it has to be them blocking you, the one on my server seems to work fine, but i haven't accessed it as many times as you, i'm sure sony has people checking your access logs and are noticing you are using their pages and probably don't care for you using it too much

[X_Gryphon_X]

Im sure that is not it, most sites wont block partial page blocks; if my server was blocked there would be zero results. There are lots of other scrtipts that scan the whole myps site and all the characters and outfit data daily with can be a huge strain, yet they are operating fine. Plus, this is a move to a new server and it has never been able to be ran fully so there are no accesses by it so far for them to block.

I have a feeling it is some php setting, or coding that will somehow make sure it pulls the whole page. aRggg 🙁

[stolzyboy]

i have the same phpinfo as you guys and i used your code, so yes it is really weird why you can't get it to work

[X_Gryphon_X]

Ok got it solved. The guys at TribalWar helped out, here is the thread http://www.tribalwar.com/forums/showthread.php?threadid=237593 (requires 6 months membership to view forum, but may possibly help someone)

Here is code that works, it needed the User-Agent specified .... interesting.

<?php
    function incomplete($sp)
    {
        fwrite($sp, "GET /character.jsp?charId=435258&worldId=3 HTTP/1.0\n");
        fwrite($sp, "Host: myplanetside.station.sony.com\n");
    }

function complete($sp)
{
    fwrite($sp, "User-Agent: MSIE/6.0\n");
}

$sp = fsockopen("tcp://myplanetside.station.sony.com", "80", $errno, $errstr);

if ($sp)
{
    incomplete($sp);
    //complete($sp); //uncomment this to get full results
    fwrite($sp, "\n");


    while ( ($data = fread($sp, 8192)) != "" )
        echo "$data";

    fclose($sp);
}
?>