scrap web page data, best tool

[joejoejoe]

I have been reading about the CURL extensions in php. I was wondering is this still the best way to do web scraping? Specifically, what I would like is to go to a site like the National Weather Service and then store the observations(the text 3 day summary) in a MySql database I have hosted. What do the experts think?

Robin

[bradgrafelman]

joejoejoe wrote:

I was wondering is this still the best way to do web scraping

There is no good way to do web scraping, because web scraping itself is to be avoided at all costs (it's the worst solution to your problem).

Instead, look for public API's that you can use which return the data you want in a sensible, computer-friendly format (not a human-friendly one).

EDIT: Just as a sidenote, if you've exhausted all attempts at requesting access to a public API of some sort (and as such are forced to do web scraping) and web scraping (and/or otherwise reproducing, reusing, or retrieving) data from the 3rd-party isn't against their TOS/AUP/etc., using [man]DOM[/man] is preferred over something like regular expression matching.

[joejoejoe]

Ok, that was a bit dumb but couldn't help myself. Thank you for not just saying, don't do it and end it there. I agree an API is the best solution, and you bring up a good point about the TOS but baring that accessing the page using the Document Object Model sounds like the right approach... now back to learning for me.

Thanks,
RS