inverted index

[mcalpine]

im creating an inverted index , with millions of records...

i need to find fast way to search the index for two or more keywords...

please help...

or point me to some articles that i can read to find more...

thanks
-m

[steadyguy]

Databases are usually built with reliable internal logic to figure out if an index is required, and then use it if it can - because they also know that indexes are MUCH faster than full table scans. Depending on your database, you might be able to set what kind of index you want it to be (as far as I am aware, MySQL doesn't yet allow this - but PostgreSQL, for example, does), so you can choose an appropriate index type to optimize for speed. I don't really think that you need to search the index directly - unless you want to use and index on an index (though I think that'd be making your life harder than it has to be.) Or am I missing the point of your question?

[mcalpine]

yip, youo are missing the point...

an inverted index is a backward table ... instead of recording the words in a document, it records the documents for a word...

so i need to be able to search that table for two or more keywords...

-m

[steadyguy]

In that case, you still probably don't need to do all the work yourself ... in MySQL, for example, you can make the column in question a FULLTEXT column, index it, and you now have a column that is fully searchable for any word. Doing this on a table with a million records will take some time, but that's a given, I think. The search will also take a bit of time, too ... but if the column is indexed, it will help.

If you want to search through the table and build up an an index manually, then Perl is probably a much better choice than PHP, as it was pretty much invented just for quickly manipulating large amounts of text.