Writing interactive spider

cysten

Hi all, I have a problem.

I have a database listing a number of cities in the US. I need a value (kWhr/m2/day)
for each city. The information can be found here: http://www.wunderground.com/calculators/solar.html

I need to enter the City, State, and then click "Find Location." The "Give raw insolation data:" checkbox needs to be checked. Then I click "submit."

It then returns the (kWhr/m2/day) I need.

At first I thought I could do this with cURL, but it doesn't seem at first glance at the php manual that cURL supports this kind of interactivity. Also the site's internal file structure is hidden so I can't just post variables in the URL.

Is there a way to do this with cURL or is there some other kind of program I can use?

bradgrafelman

cysten wrote:
At first I thought I could do this with cURL, but it doesn't seem at first glance at the php manual that cURL supports this kind of interactivity.

What "interactivity" ? Sounds like you're just POST'ing simple values.

Unless you think the cities you're querying for are going to get up and move at any point, there's no need to go through the whole "Find Location" process... just do it once, grab the latitude/longitude values, and then hardcode those values into your script.

cysten wrote:
Is there a way to do this with cURL

Of course - that's one of its main uses. Simply examine the HTML form elements that are submitted and use cURL to POST those same values.

I notice that Wunderground has an API/XML data feed for part of their data... you might try contacting them to see if such an API exists for this solar calculator you're using. If there is one, it'd be a lot easier to use (since it'll be made for computer-to-computer interaction, not human-to-computer interaction such as a pretty HTML form).

cysten

Um I could get the loc for each but the box still needs to be checked. I've never used cURL before. Guess I am a newb. When you submit, the url does not display the values selected. I could figure out them from looking at the source code, but would the server accept my attempt to post via the url? I couldn't find any other way to select values on a page other then by including them in url.

cysten

The submit button actually isn't part of a form. Its a button linked to some Javascript. How would I use cURL to check the box, enter the latitude and longitude, hit submit, and grab the contents of the result?

bradgrafelman

cysten wrote:
When you submit, the url does not display the values selected.

Of course it doesn't - the form uses the POST method, not GET.

cysten wrote:
would the server accept my attempt to post via the url?

I don't know what you mean by "post via the url." If you mean insert the form variables in the URL rather than POST'ing them like the form does not, no, that most likely won't work (unless the site programmers have designed the processing application to look for the values in either location).

cysten wrote:
I couldn't find any other way to select values on a page other then by including them in url.

Including them in the query string is analogous to submitting a form with the GET method... what you need to do is use POST instead.

cysten wrote:
How would I use cURL to check the box, enter the latitude and longitude, hit submit, and grab the contents of the result?

I don't think you understand what cURL does. There is no boxes to check or buttons to press... that stuff is just the graphical interface your web browser presents you. In the end, the data still gets POST'ed in a standard format.

All you're doing with cURL is sending the same variables that get sent as if you had done all of the above in your browser.