Geting A Description About A Page

[Anon]

Geting A Description About A Page , might sound easy if its got META tags i can do that no problem ......but what if it hasnt?? ......i have removed all the HTML and done a substr but depending on the web page is returns loads of crap like the JS script that my have been in the body of the document.

Really what i need is to get a brief useful description about a page that has no meta tags , can anyone help ......im having problems with this one.

<Pulling Hair Out>

Simon !!

p.s Ive designed a spider , that basically goes to the sites sent to my search engine over night and appends them 2 my database ...just like yahoo does. 🙂

[Anon]

Well, this is not easy, because creating descriptions
of texts means understanding the text so you can decide what parts are important and which parts arent.

This is something that the searchengine builders have nightmares about.

Some solve it by just taking the first paragraph of text that is "normal" text (ie, not javascript or some other sh*t)

If you want true descriptions of a site you'll either have to do it yourself, or build one heck of a langua-intepreting routine that can describe what a text is about.

I'd opt for the "first paragraph" idea.