polyhedron wrote:this is not easy because of so many evolving methodologies.
... and because of the AUP/TOS/copyright/etc. that many websites have in place (which often specifically talk about "automated" or "computerized" requests being prohibited).
This is also why many sites are providing some sort of external API (often involving XML).