Speed up Catobase?

MogulChamp · 01-19-2005, 05:46 PM

Ok my computer isn't a slouch by any means. Athlon XP 2400+, 1 gig PC2700 RAM. It should create the HTML pages relatively fast...well it doesn't. Are there any tricks to speeding up this part of the process? The imports are fast, it's the creation of the webpages which takes a bit too long for my tastes.

I currently play a season, import, then another. I will play about 10 seasons before I create the web pages. I don't use Lefty/Righty splits, however I do have it recreate team pages.

Suppose I created the pages after every season, would it still have to go back 10 years and recreate all the old team pages? I would think it wouldn't but it appears that it does.

Help me out here, how do you speed up this process?

SandMan · 01-19-2005, 06:14 PM

The process for creating the web pages will be slow and it gets slower the more seasons you have. You have probably maxed out the benefit of have a good amount of ram, hard drive space and processor. What is causing the program to appear slow is that ACCESS MDB using DAO, ADO has a lot of overhead and is slow. CATOBASE is a DATA intensive program with a lot of calculations being created. Also I suspect the program is created with Visual Basic which doesn't create very fast executables compared to other programming languages.

Simply put it is a software issue and not a hardware issue after you reach a certain point. I am using a 1.7Ghz system, 1Gig 2700sdram and think that there is a way to make the program run faster, but there just isn't at this time.

Ideally if the program created more tables, was written in C++ or C#, used VMA (Virtual Arrays for holding vast amounts of data), created pages only as needed instead of at one time then the program would be fast. Re-creating all the pages at one time is very time consuming but was in the initial design of the program. To make the program faster would mean a completely rewritten program....

With that said, I simply think CATOBASE is very good and I probably would not play as many seasons if the utility were not available. I used to use the Exporter, create my own tables and reports but that was time consuming, CATOBASE does everything I wanted and more....

DaveHorn · 01-19-2005, 06:34 PM

Agreed. The largest bottleneck is the database (Access) mostly and slightly the program technology (VB). Access is garbage and painfully slow. But it acheived the likely desired goal; letting folks use CATO without going through the overhead of installing a real database and getting it configured on their system. I would be willing to bet that generating the exact same database on a PC with half your resources would come relatively close to the same amount of time.

rem · 01-19-2005, 06:44 PM

Turning off OOTP and any other programs helps.

Ktulu · 01-19-2005, 06:53 PM

Quote:

Turning off OOTP and any other programs helps.

Having a nap or a few beer also helps.... the only way to speed up the process is to not sit and wait for it.

DaveHorn · 01-19-2005, 07:26 PM

What am idiot I am. Access sucks and is slow, but I would put big time bucks the bottleneck is all in the file creation. Creating a few thousand files is definitely is a time kill. In addition since CATO creates two HTML pages for every player, if each level has 25 players in a 30 team league, you have 3000 files right there.

The biggest problem with CATO in terms of performance when generating HTML is that it isn't smart enough to recognize that a player has retired so it still creates the player HTML page. As the seasons progress and more and more players retires, pretty soon CATO is spending FAR more time generating player pages that haven't even changed than it is spending creating the player pages for the actual players currently in your league.

So after many seasons, those 3000 player files, become MANY MANY more after players retire out of the league each season.

disposableheros · 01-20-2005, 02:13 AM

nobody mentioned it yet, so i will....

turn off any anti-virus programs you have running. that will make it run almost twice as fast.

Treches · 01-20-2005, 05:21 AM

Quote:

Originally Posted by DaveHorn

The biggest problem with CATO in terms of performance when generating HTML is that it isn't smart enough to recognize that a player has retired so it still creates the player HTML page. As the seasons progress and more and more players retires, pretty soon CATO is spending FAR more time generating player pages that haven't even changed than it is spending creating the player pages for the actual players currently in your league.

Right on, that's the main problem CatoBase has, making both HTML generation and later upload to the FTP site a painful experience when the league has some history behind it. I just let it run or upload overnight.

jmm8356 · 01-20-2005, 10:28 AM

Quote:

Originally Posted by DaveHorn

What am idiot I am. Access sucks and is slow, but I would put big time bucks the bottleneck is all in the file creation. Creating a few thousand files is definitely is a time kill. In addition since CATO creates two HTML pages for every player, if each level has 25 players in a 30 team league, you have 3000 files right there.

The biggest problem with CATO in terms of performance when generating HTML is that it isn't smart enough to recognize that a player has retired so it still creates the player HTML page. As the seasons progress and more and more players retires, pretty soon CATO is spending FAR more time generating player pages that haven't even changed than it is spending creating the player pages for the actual players currently in your league.

So after many seasons, those 3000 player files, become MANY MANY more after players retire out of the league each season.

Some parts of retired players' pages change year to year, both their career leaderboard placements and similiar batters/hitters... plus the average HoF ink test score must get recalc'ed every year too.

But if there were an option to "only create active players' pages", that would be great.

SandMan · 01-20-2005, 11:02 AM

Quote:

Originally Posted by disposableheros

nobody mentioned it yet, so i will....

turn off any anti-virus programs you have running. that will make it run almost twice as fast.

This only works when you don't have much data in the database, after about 10-20 seasons this will not improve the running of the program. After this time is when it is a database engine problem. I still think it would be nice to have the program written in C with a small footprint database engine like codebase. I did a test and took the data from the MDB file and put the data into COBOL dat files and the savings was - get this - The MDB file was 36MB, the dat file was 3.4MB. Of course the dat files are optimized for size and ACCESS does not allow this.

Ideas anyone?

DaveHorn · 01-20-2005, 01:48 PM

Quote:

Originally Posted by jmm8356

Some parts of retired players' pages change year to year, both their career leaderboard placements and similiar batters/hitters... plus the average HoF ink test score must get recalc'ed every year too.

But if there were an option to "only create active players' pages", that would be great.

Good point, didn't think about the ink tests. But one of the two pages (the sp page) never changes. Also, there are no career leaderboards on a player page are there? Those are just in season leaderboards, so they wouldn't change.

Ink tests and similar batters though obviously can change, and I agree that would be a nice option.

The virus scanner option also only helps depending upon your virus scanner and/or the options you are using with it. For some folks, turning off the virus scanner will not make a difference, but it is always worth a shot.

MogulChamp · 01-20-2005, 04:13 PM

Yeah I always shutdown any applications when dealing with anything OOTP related, virus software as well. Unfortunately it doesn't help at all. It would definately save a lot of time if the retired player pages weren't recreated. I will continue to wait it out, but it sure would be nice if someone could create a newer, faster version..

jmm8356 · 01-21-2005, 09:48 AM

Quote:

Originally Posted by DaveHorn

Also, there are no career leaderboards on a player page are there? Those are just in season leaderboards, so they wouldn't change.

Check one of your Hall of Famer's pages and after all the season leader lines ("1966-1-109)", you'll see a line that starts with CAR ("CAR-1-1839")... that's his career leaderboard placement.

DaveHorn · 01-21-2005, 12:48 PM

Quote:

Originally Posted by jmm8356

Check one of your Hall of Famer's pages and after all the season leader lines ("1966-1-109)", you'll see a line that starts with CAR ("CAR-1-1839")... that's his career leaderboard placement.

Ah.... thanks. I don't think any of our HOFers have that yet. Since the league is in its 8th season and all of our HOFers played most of their career in the MLB and not HFTC. But I did look a good player who has played for our full 8 season in HFTC and see exactly what you mean.

01-19-2005, 05:46 PM	#1
MogulChamp All Star Reserve Join Date: Aug 2004 Posts: 656	Speed up Catobase? Ok my computer isn't a slouch by any means. Athlon XP 2400+, 1 gig PC2700 RAM. It should create the HTML pages relatively fast...well it doesn't. Are there any tricks to speeding up this part of the process? The imports are fast, it's the creation of the webpages which takes a bit too long for my tastes. I currently play a season, import, then another. I will play about 10 seasons before I create the web pages. I don't use Lefty/Righty splits, however I do have it recreate team pages. Suppose I created the pages after every season, would it still have to go back 10 years and recreate all the old team pages? I would think it wouldn't but it appears that it does. Help me out here, how do you speed up this process?

01-19-2005, 06:34 PM	#3
DaveHorn All Star Starter Join Date: Aug 2003 Posts: 1,146	Agreed. The largest bottleneck is the database (Access) mostly and slightly the program technology (VB). Access is garbage and painfully slow. But it acheived the likely desired goal; letting folks use CATO without going through the overhead of installing a real database and getting it configured on their system. I would be willing to bet that generating the exact same database on a PC with half your resources would come relatively close to the same amount of time. __________________ Dave HFTC Commish

01-19-2005, 06:44 PM	#4
rem Hall Of Famer Join Date: Jul 2003 Location: College Park, Md. Posts: 5,024	Turning off OOTP and any other programs helps. __________________ OTBA: Washington Filibusters CLBR: Minnesota Twins 2005 Winner of LL's Pick the Loser Contest! - 15-0

01-19-2005, 07:26 PM	#6
DaveHorn All Star Starter Join Date: Aug 2003 Posts: 1,146	What am idiot I am. Access sucks and is slow, but I would put big time bucks the bottleneck is all in the file creation. Creating a few thousand files is definitely is a time kill. In addition since CATO creates two HTML pages for every player, if each level has 25 players in a 30 team league, you have 3000 files right there. The biggest problem with CATO in terms of performance when generating HTML is that it isn't smart enough to recognize that a player has retired so it still creates the player HTML page. As the seasons progress and more and more players retires, pretty soon CATO is spending FAR more time generating player pages that haven't even changed than it is spending creating the player pages for the actual players currently in your league. So after many seasons, those 3000 player files, become MANY MANY more after players retire out of the league each season. __________________ Dave HFTC Commish

01-20-2005, 02:13 AM	#7
disposableheros Hall Of Famer Join Date: Jun 2004 Posts: 4,332	nobody mentioned it yet, so i will.... turn off any anti-virus programs you have running. that will make it run almost twice as fast. __________________ 2 Wild Cards, 11 Division Champs, 4 League Champs, 3 World Champs, and 3 Best GM awards Baseball Maelstrom - New York Mets - 180-149 .547 Corporate League Baseball - Coke Buzz - 889-649 .578 Western Hemisphere Baseball League - Santiago Saints - 672-793 .459 Record - 2428-2271 .517

01-19-2005, 06:14 PM	#2
SandMan Hall Of Famer Join Date: Aug 2003 Posts: 2,946	The process for creating the web pages will be slow and it gets slower the more seasons you have. You have probably maxed out the benefit of have a good amount of ram, hard drive space and processor. What is causing the program to appear slow is that ACCESS MDB using DAO, ADO has a lot of overhead and is slow. CATOBASE is a DATA intensive program with a lot of calculations being created. Also I suspect the program is created with Visual Basic which doesn't create very fast executables compared to other programming languages. Simply put it is a software issue and not a hardware issue after you reach a certain point. I am using a 1.7Ghz system, 1Gig 2700sdram and think that there is a way to make the program run faster, but there just isn't at this time. Ideally if the program created more tables, was written in C++ or C#, used VMA (Virtual Arrays for holding vast amounts of data), created pages only as needed instead of at one time then the program would be fast. Re-creating all the pages at one time is very time consuming but was in the initial design of the program. To make the program faster would mean a completely rewritten program.... With that said, I simply think CATOBASE is very good and I probably would not play as many seasons if the utility were not available. I used to use the Exporter, create my own tables and reports but that was time consuming, CATOBASE does everything I wanted and more....

01-20-2005, 04:13 PM	#12
MogulChamp All Star Reserve Join Date: Aug 2004 Posts: 656	Yeah I always shutdown any applications when dealing with anything OOTP related, virus software as well. Unfortunately it doesn't help at all. It would definately save a lot of time if the retired player pages weren't recreated. I will continue to wait it out, but it sure would be nice if someone could create a newer, faster version..