Need Suggestion loading file data

Anon

Hi all. My system setup is PHP/MYSQL/Apache.

Here is the situation:
I have continuously hourly new data stored in
files that needs to be (1) loaded into the database
(2) once loaded into the database, some queries
are make to construct more summarized data
(stored in another table).

Currently, I use php to read the file and insert
them into the database. However, this takes up
database connection, and depends on Apache
running continuously.

I am looking at other possible languages that will
(1) connect to the database directly(by pass Apache)
(2) has some SQL query capability(the summarized
data depend on the group by clause)

I would appreciate any suggestion. Thank you.

-Will

Anon

Depending on the form of your data file you should be able to use the mysql "LOAD DATA INFILE" option to let mysql load the datafile by itself.

basically any query you run through php/apache can be run from the commandline using the mysql binary.
This casn even be done in batch mode by the crontab.

Anon

Vincent, thanks for replying.

I am currently using the mysqlimport utility with --force to load the raw data into the database.

My biggest problem is, currently I have several php pages that runs every hour to generate summarized report on these rawdata.

What I mean is, these php pages do the following:

(1) open up database connection
(2) query the raw data which we have just imported, using GROUP BY, TABLE JOINING operations.
(3) does some calculation with the fetched data and insert the summarized data into another table in the database.

The problem is, this obviously takes up database connection(the summarized data are
what users are seeing when they come to this site).

Also if for any reason Apache Server
goes down, not only can't the user see the summarized data, but neither can we load any raw data or summarized data into the database. This is what I am trying to avoid
(I want the data loading to bypass Apache, so if i take Apache down, the data loading will continue to operate).

Do you have any language in mind which can do this (as has the capacity to query the database and does not use a web server)?

-Will

Anon

You can run any php script from the commandline, without apache, if you compile php to be a cgi binary.

You can use both the apache module and the cgi version at the same time.

Check out the manual for more info.