[Resolved] indexing files into database

cbrknight

I know this has probably been covered. I have looked for a premade solution but none of them really seem to be what I am looking for. I figure if I post someone will be able to point me to a script that will do this.

I have many files (over 40 Gb). Mostly doc and tif files. Right now I have a search page that users can search for filename. The search through all those files in many subfolders is just taking too long.

I thought if I could find a way to index those files into a database and just run a cron job every few hours that I could just have the users search the database for names and then get the file that way.

Anyone know of a good solution for this. The user does not need any rights to add or delete files. Just to read them.

Thanks in advance.

devinemke

i assume you already have some code going to scan the dir using [man]opendir[/man] or [man]scandir[/man] for your seach script. that code could be easily tweaked into an indexing script. the basic idea is to initially scan the dir and on each iteration add a record to the db with that filename. then later to update the index simply rescan the dir into one array, pull the records for the db into another array, and then use [man]array_diff[/man] to see what has changed. use the resulting "diff" array to add/delete records to the db.

make sense?

ps. be sure to index the db table to speed up the db accesses!

cbrknight

i assume you already have some code going to scan the dir using opendir or scandir for your seach script. that code could be easily tweaked into an indexing script. the basic idea is to initially scan the dir and on each iteration add a record to the db with that filename. then later to update the index simply rescan the dir into one array, pull the records for the db into another array, and then use array_diff to see what has changed. use the resulting "diff" array to add/delete records to the db.

the code that I have was written by someone else. I have never used the opendir() and scandir() functions before. From what I can tell from the php.net manual... scandir() only works on php5 which puts me in a bind since my boss probably isnt going to want to upgrade just yet.

as for arrays... I am still trying to get a handle on them... that is why I was looking through hotscripts.com trying to find something that I could use while I figure all this out.

Thanks anyway for your help

dewen

Are you using linux? Can you try to use "tree" command and put dir and files into a file:

tree /work_dir/ > dir_file

then parse the file to retrieve the path and file name.

The tree command is fast, I have 7714 directories, 131931 files on my server, it take only 20 seconds

Cron + database can be faster, but could miss newly added files

cbrknight

I am using windows but have apache2 server.

This is a stupid question I know but what exactly does it mean to parse a file?

dewen

You should try something with linux, it's lots of fun.

To parse a file, you open and read a file, try to get information you need from it using logics.

If you are using windows, then job + database should be your choice. Then you should have an interface when people adding or removing files, so the database log the change and update index. It's not ez to get a model to use. What database you are using?

cbrknight

mysql

I have a linux laptop but dont use it that often. I couldnt get my boss to use linux. I dont think he wants to learn how to use it.

so you are saying that

Fopen() and fclose() are some functions used to parse a file?