Performance and fgets()

[Anon]

Hello,
I am reading distance-data from an 160 MB ASCII-file (it is matrix-like). Unfortunately I need nearly 7.000 distances every time I read from the file. At the moment I use fseek() and fgets() to get every single distance.

As this takes much time (ca. 10 seconds) for all 7.000 distances I measured performance with the microtime()-function.

I was surprised that the fgets()-function takes about 2000 time units if there are only few bytes between the positions I read the distance-date from. But if there is a certain amount of bytes between the last position I read from an the new position I am beginning to read from the fgets()-funktion takes about 60.000 time-units.

Is there some optimization that does only work if both positions are not to far away? Is there any way to speed these fgets()-functions up? I tried to use fread() and fgetc() instead but this did not increase performance.

Ufortunately it is not possible to put the data into a database.

Thank You,

Stefan

[[deleted]]

"Ufortunately it is not possible to put the data into a database"

There's no data format that cannot be put in a database. The only excuse for not using a database is that a) your host does not support it, or b) you don't want to/know how to put it in a database.

Anyway, reading from a 160MB file will never be fast (that's why they invented databases)

Are you using fseek() to go to the new position in the file before reading the rest?

[Anon]

Hello vincent,

I am using fseek() to go to the new position in the file. Every fseek() takes about 1500 time-units no matter how great the amount of bytes between the two positions is.

The data format can be put in a database, of course. But there is not enough space on the server to do this. (and my hoster only supports mysql).To test this, I put the data into a mysql-table. The table had 170 MB. But i had to create 450 MB of additional indexes to speed the query up.

Thanks,
Stefan