web mining

[j_miguel_y]

I am looking for a way or open source that can help me mine information/links from other web sites.

For example, I would like to mine car listings from autobytel.com, carprices.com, etc. Obviously, there would be constraints like pricing, year, model, etc.

How would I go fetch data from those sites?

An example site that does this function is www.carminer.com, but it uses ASP.

Any ideas?

[mobtex]

regardless of php, jsp or asp each site is outputting data in html form, find the common ground within the documents and build up from there but wouldn't the sites mind? rss would be a better, approved way of using someone else's content.

[MarkR]

You'll need the cooperation of the companies involved (and SLAs, etc). Even then, it's going to be very difficult.

Most of the companies will probably only give you a custom feed of their data if you pay them to do so.

Each company will have their data in a different format - you could try to impose your own format on them, but their web developers will make a lot of errors which you'll have to work around, because no matter how hard you try, they won't fix them.

Each web developer will be sending you the data in a format that they think is right, but is actually wrong in some way (and they won't fix it unless you pay them more money).

SO you'll have rubbish data you'll have to unpick out of multiple broken formats, then any price comparison will quickly become very difficult.

I speak from experience

Mark