Very large MySQL files

BMW · 08-21-2007, 12:05 AM

I was trying to build a very advanced database for my league. As expected, the MySQL export files quickly ballooned to massive size.

First, the issue was 16MB. The players_at_bat_batting_stats.mysql.sql file grew past 16MB, and my hosting service couldn't import it any longer. There was a file size cap. Also, trying to import too much too fast would often throw random errors because MySQL server was too busy.

So I spent a week or so to write a script to import the files in bite-size segments. Finally, I was successful, and I got past that 16MB limit.

Then comes the 70MB limit!

PHP also bugs out when it runs out of memory. It seems that the amount of data that it can load into memory at once is about 70MB before it throws a fatal error.

I realize that not may people are looking to import files this size. But the few who want to do so may not have the Enterprise-level web hosting that is needed to actually import all this stuff without capping out the server.

Can a feature be added that would simply chop up the files to a certain amount of bytes as they come out of OOTP?

Nutlaw · 09-26-2007, 12:08 PM

Please post all suggestions in the Suggestions forum. Thanks.

08-21-2007, 12:05 AM	#1
BMW Hall Of Famer Join Date: May 2003 Location: New Jersey Posts: 2,030	Very large MySQL files I was trying to build a very advanced database for my league. As expected, the MySQL export files quickly ballooned to massive size. First, the issue was 16MB. The players_at_bat_batting_stats.mysql.sql file grew past 16MB, and my hosting service couldn't import it any longer. There was a file size cap. Also, trying to import too much too fast would often throw random errors because MySQL server was too busy. So I spent a week or so to write a script to import the files in bite-size segments. Finally, I was successful, and I got past that 16MB limit. Then comes the 70MB limit! PHP also bugs out when it runs out of memory. It seems that the amount of data that it can load into memory at once is about 70MB before it throws a fatal error. I realize that not may people are looking to import files this size. But the few who want to do so may not have the Enterprise-level web hosting that is needed to actually import all this stuff without capping out the server. Can a feature be added that would simply chop up the files to a certain amount of bytes as they come out of OOTP?

09-26-2007, 12:08 PM	#2
Nutlaw Hall Of Famer Join Date: Apr 2003 Posts: 9,538	Please post all suggestions in the Suggestions forum. Thanks.