What to do when processing large databases?

[Joe826]

Hi,

I'm working on a program that requires me to process serveral thousand mysql database entries and I had a preformance question. I could bring the entire database into memory each time the program runs (with select *) or I could query the database each time. I thought bringing the whole db into memory would be fastest but when it gets very large (40,000+ entries) it seems slow down my system almost a crawl. Is there a better way to do this?

Thanks,
Joe

[dopey]

Hi,

Your best bet is definately to run a query for the records you require for processing at that time and not to load the entire dataset into memory. Get your keys right on the table and you should get good performance.

[Joe826]

That's much better, thanks! 🙂