fsock - Getting file from POSt

lolao

Hi.

I've been puzzling around with a method, and can't seem to get it working.

What I want to do, is to connect to a page (secure / SSL), login to an account and then get the content of a remote file, which is automatically generated by posting a form.

Most of it works already, but when my script has posted the login (which is successfully carried out) I can't seem to get/download this file I want, from the account front page.
Problem is that the file is generated (XML) when you click on a <button>, which means I don't really know the exact location of the file.
So I've tried submitting/posting the form <button> the same way I've done it with the login, and that seems to be working, but how do I get the content of the generated XML file? Normally, I mean, a download popup would appear if you submitted/posted the <button> through a web browser, but how do I receive the content with my script and maybe place it in a variable ($XML_fileContent).

I've been intensively trying to solve this problem and get it to work for three days now, but really can't seem to figure out a solution. I thought of this forum and I hope that someone can help me out.

[Using fsockopen, no CURL]

Any suggestion? Anyone tried this before who actually made it work?

BlueKai

I like the term 'puzzling around'. If you don't know where a file is, it's going to be difficult to get the contents of it. If the file is generated by a submit button, you'll have to look at the code that handles that form submission to find the path to the file it generates.

lolao

Thanks for the answer.

There's no value in the action attribute of the form, so either it's all being carried out on the same page as the form appears, or they simply use redirecting methods to handle the submission somewhere else. Nor is there any javascript which indicates a path I could use.

So in short; I don't have any information about where file is, but I know the technique exists, because there are people who've made a fully functional browser emulator, also accepting download dialogs and so on.

It can be done, I just need to know how to make a script automatically accept a download dialog OR simply paste the content of the requested file in my document somehow. I believe it has something to do with "chunked data", if that will lighten up any solution?

Thanks for the answer though, I thought this forum was dead. 🙂

Can anyone help?

big_nerd

Could you not just post the information directly (using header) and use fgets to recieve the page output?

I know the paypal IPN Verification works in this manner (however, not SSL)

<?php
$req = 'cmd=_notify-validate';

foreach ($_POST as $key => $value) {
$value = urlencode(stripslashes($value));
$req .= "&$key=$value";
}

$header .= "POST /cgi-bin/webscr HTTP/1.0\r\n";
$header .= "Content-Type: application/x-www-form-urlencoded\r\n";
$header .= "Content-Length: " . strlen($req) . "\r\n\r\n";
$fp = fsockopen ('www.paypal.com', 80, $errno, $errstr, 30);

$item_name = $_POST['item_name'];
$item_number = $_POST['item_number'];
$payment_status = $_POST['payment_status'];
$payment_amount = $_POST['mc_gross'];
$payment_currency = $_POST['mc_currency'];
$txn_id = $_POST['txn_id'];
$receiver_email = $_POST['receiver_email'];
$payer_email = $_POST['payer_email'];

if (!$fp) {
//
} else {
fputs ($fp, $header . $req);
while (!feof($fp)) {
$res = fgets ($fp, 1024);
if (strcmp ($res, "VERIFIED") == 0) {
//
}
else if (strcmp ($res, "INVALID") == 0) {
//
}
}
fclose ($fp);
}
?>

I am unsure of what you are doing to connect to the SSL connection; but maybe you can take something from this code (fgets).

If there is nothing in the action field of the form, that means generally that the form is posting to itself (if I am not mistaken).

I hope this is of some help...

lolao

It is looking great, but the file I need to get is being generated temporarily to a specific location. Returning the output of the page only shows the same page as where the form appears.

I believe it's about not being able to handle the chunked data (which is the encoding type of data which the XML file outputs).

Maybe this can only be done in CURL, since it's probably just doing things from the shell.. I don't know. You think CURL would be able to do this better?

big_nerd

Curl should work, however one more idea...

Maybe you can use FF w/ Live HTTP Headers.

You can see what method they are using and what request string is being sent to the browser when you manually complete the process.

Perhaps this can shed a bit more light on it?

lolao

Good suggestions, but that doesn't seem to be working either. Thanks anyway.

I've also tried a few things with cURL now, but how do I do this? My script is logging in and navigating to the page I want, perfectly.. But then again; It posts the form, and ... what to do now? How do I get the file's content, when I don't know it's name or location? Or how do I get to know any of those... Perhaps I already have stored the file? How do I know that?

he.. Well. Anyone up to the challenge? I really appreciate some help here.

BlueKai

What kind of information is in this file that you don't already have? If you're submitting a form through this script, you have the form, all the fields in it, and what you're submitting.

Getting any more information would be guessing at how that information is being generated. For example, if I post a blog and that creates a link to wherever.com/myblog/blog_name.html - I know that the url will be 'wherever.com/myblog/' + blog name (replace spaces with _) +.html...

MarkR

Don't write your own http implementation (like big.nerd did).

Instead use fopen with a context. You can set the relevant POST headers and content into the context object (stream_context_create etc) and then have those sent, and read the response.

You don't need curl, AND you don't need your own http implementation.

Remember to enable all warnings and notices and have them logged to somewhere that makes sense.

Mark

lolao

Yeah well... I have enabled all error reportings.

I don't think fopen can handle chunked data can it? I thought about cURL because cURL seems to handle it by default.

So here goes my scripts current situation:

It navigates to url-login < Works
It logs itself into my account < Works
It then navigates to url-formpage where the form is located < Works
Now it submits the form with the correct information (looked it through way too many times, I am sure of it) < Works
After submitting the form, it seems that I am being redirected to the url-formpage, even though I have set both max redirects and follow redirects to "0".

So actually it seems that my script "oversees" and ignores the file, that I should have gotten right after the submission.

MarkR

lolao wrote:
I don't think fopen can handle chunked data can it?

Chunked responses are only used in HTTP/1.1. fopen uses 1.0.

A server which sends a HTTP/1.1 chunked response to a 1.0 client is BROKEN!

After submitting the form, it seems that I am being redirected to the url-formpage, even though I have set both max redirects and follow redirects to "0".

So actually it seems that my script "oversees" and ignores the file, that I should have gotten right after the submission.

What do the developers of the other web app think about this? Can they provide you with a more machine-friendly interface, or explain why their app is giving this behaviour?

I assume you are testing this against a non-production instance of their application- are they giving you the ability to see what's happening inside their code?

Mark