scraping data

qadeer_ahmad

Hi all developers,
Most of our projects have scrap data related we use different ways commonly Curl / preg_match but all is depends on html source so if some one change the html structure change then code stop working.
Is there any perfect solution of this issue?

Thanks

bradgrafelman

qadeer_ahmad wrote:
Is there any perfect solution of this issue?

Yep: don't do it. :p

To elaborate, "scraping" data is a very crude way of getting the information you want from a remote source and, as you've pointed out, can easily be broken by even the smallest of design changes. This is why you should make every feasible effort into using some sort of XML feed to query the remote server for the information you want.

If that fails, the first thing you should realize is that many websites explicitly state that any automated access/retrieval/storage/use of the information on their site is prohibited (e.g. by an AUP or something similar).

If that isn't true, and you're still left with no other option that retrieving the HTML source code displayed and scanning it for data, you might look into using [man]DOM[/man] to navigate to a specific point in the HTML document and retrieve the data. In theory, this kind of solution would be more resilient to slight design changes since it doesn't rely on preg_match() to search for specific HTML markup.

dagon

Copyright law applies no matter what the site rules say. Any site that wants you to take their content will provide an api, rss feed or some mechanism to do so. Any business model that relies on theft is not sustainable.

I'm blunter than bradgrafelman

qadeer_ahmad

bradgrafelman;10943046 wrote:
If that isn't true, and you're still left with no other option that retrieving the HTML source code displayed and scanning it for data, you might look into using [man]DOM[/man] to navigate to a specific point in the HTML document and retrieve the data. In theory, this kind of solution would be more resilient to slight design changes since it doesn't rely on preg_match() to search for specific HTML markup.

Yes this can be idea we follow the [man]DOM[/man] to navigate to a specific point.
We can create function for all tags separately and these function handle all possible condition. That can be a library to handle such things.