"reading" from a website like www.amazon.com

tefnut

So I'm looking around the web, searching for books on www.amazon.com on a subject that interests me. I see a cool book, I note its ISBN, such as 0156027755, and I want to add this to a listing on my own website.

However, I don't want just the ISBN. I want my website to automatically, given the ISBN, go the respective amazon.com URL:
http://www.amazon.com/o/ASIN/0156027755

And then - and the is the part I'm not sure about - "read" the page and pick out key things. Notice that the amazon.com listing page for a book has things like:

List Price: $18.00

Price: $12.60 & eligible for FREE Super Saver Shipping on orders over $25. See details.

You Save: $5.40 (30%)

Availability: Usually ships within 24 hours

and...

Product Details

Paperback: 416 pages ; Dimensions (in inches): 0.98 x 8.00 x 5.32
Publisher: Harvest Books; (December 16, 2002)
ISBN: 0156027755

Other Editions: Hardcover | All Editions

Average Customer Review: Based on 8 reviews. Write a review.

Amazon.com Sales Rank: 27,512
(Publishers and authors: improve your sales)

last but not least...

All Customer Reviews
Average Customer Review:

Write an online review and share your thoughts with other customers.

9 of 10 people found the following review helpful:

you guys are reviewing the wrong book, November 30, 2002
Reviewer: Alex Korsunskaya from Boston, MA USA

just a correction: Nabokov wrote two different books "lectures on literature" and "lectures on Russian literature." most reviewers here are talking about the wrong book. --This text refers to an out of print or unavailable edition of this title

I need my website to "read" this, that is, get a full text of the site into a variable somehow, and then using string routines I can extract the bits that interest me. Either the website's HTML code or the formatted content as it shows up in the browser would do.

How would I do this fetching of website content into a variable?
(I'm thinking search engine / spidering here, but I have no clue what their actual code is like :o )

piersk

Take a look at this page...

tefnut

Oooh, shiny. Thanks, piersk.

Yet, supposing that I was interested in doing this on another website that's not as considerate towards developers, how would I go about it?

piersk

Originally posted by tefnut
Yet, supposing that I was interested in doing this on another website that's not as considerate towards developers, how would I go about it?

What do you mean by this?

bubblenut

Well, first thing's first, you ask them, if you don't you're stealing and that's not nice. Also, if they're a business ask them for a datafeed, almost all online shops now have some kind of datafeed system and even if it's just a CSV file it's still going to be a hell of a lot easier than picking through their html.

tefnut

Originally posted by piersk
What do you mean by this?

I want to be able to do this website "read" without a website's specialized services that provides some sort of feed.

Originally posted by bubblenut
Well, first thing's first, you ask them, if you don't you're stealing and that's not nice. Also, if they're a business ask them for a datafeed, almost all online shops now have some kind of datafeed system and even if it's just a CSV file it's still going to be a hell of a lot easier than picking through their html.

Well, they're making disturbing amounts of money off of their affiliate programs and generally people linking to them, so I don't think they'll mind 😉

piersk

Originally posted by tefnut
Well, they're making disturbing amounts of money off of their affiliate programs and generally people linking to them, so I don't think they'll mind 😉

Wanna put money on that??

Anyway, why not just use the services I linked to?

tefnut

Originally posted by piersk
Wanna put money on that??

Anyway, why not just use the services I linked to?

I'm guessing you want to hear that I wish to commit horrible felonious acts that will commit me to 25-50 years in pound-me-in-the-ass prison while also bringing about the downfall of www.phpbuilder.com 🙂

It's just curiosity at this point, I am going to end up using the services that you linked to, but I know that some websites (I ran across this internet archive that took screenshots of every major page on the internet, every few weeks) do this information retrieval and I'm just curious how it works.

But speaking of copyrights, how do search engines, which obviously scan and reproduce contents on every page on the internet, not get sued? And what would be the legal difference between obtaining amazon.com content by using their feed vs. devising my own script? I'm not trying to start anything, but my current understanding is that as long as you're loosely abiding by the rules, you're fine. Using common sense, such as providing a credit, using the information to give the source site more customers or more hits, is fine. If anything is not, the owners contact you, tell you to stop, and then you do. Here's an example: car manufacturers do huge photography shoots and copyright the photos. Yet these photos are freely used by millions of websites out there, and the manufacturers do nothing about it. Why? The content itself was created in order to publicize their product or service. It's about common sense, and a harmless slap on the hand when you guessed wrong. But, of course, this is just my opinion/experience and should not be taken as authoritative in any way.

Anyway, this is slightly offtopic now.

michaelkenneth

If you were gonna do it without a datafeed, you'd have to use reqular expressions, or something of the sort. It's really not as fun as it looks. I did it once to jack events off of a website one time, took probably....hmmm, 5 hours total. This was a pretty HTML friendly site also.

BuzzLY

There are three different issues here:

Legality: In most cases, it's not legal, because you need the site's permission. In the case of search engines, I am not sure how they get around this, but I'm sure they have crack legal departments to deal with it.
Morality: I believe it is not morally right to grab info from a website unless you have their permission. They took the time to compile and (hopefully) research the information, why should you be allowed to grab it for free?
Practicality: Is it possible? Of course. People do it all the time. How to do it depends on what you want, and from where. For example, if you want specific information from a specific site (like scores from a sports site), then you can write some simple proprietary function to do the job. If you want the ability to grab general info from multiple sites, you better dust off your regexp skills.

I seriously advise you consider 1 and 2 before you attempt to do anything. If a site, like amazon, offers feeds, then by all means, use them. If a site has a policy document forbidding screen scrapes, don't do it. If you aren't sure, either don't do it, or ask permission.

You would most likely get slapped on the wrist if you were caught taking a free newspaper from a neighbor, and most likely wouldn't get caught. That doesn't make it right.