Auxiliary Historical Database project

GaryStephen · 06-04-2006, 05:50 PM

I'm planning to begin work on a player database that will supplement the Lahman database.

By converting real-life AAA and AA stats to their major league equivalents, we can create real players that can be used in conjunction with major league players, thus giving you a larger pool of players to choose from. These players can serve the following purposes:

-- Use real players instead of fictional ones to fill in gaps in historical seasons
-- Give meaningful stat lines to "cup of coffee" players
-- Bring up future stars a year or two sooner, or watch them develop in the minors on their way up
-- staff minor league teams for a historical league
-- create larger-than-life MLB leagues

These supplemental players were used in a long-running APBA/Baseball For Windows league, and they worked pretty well.

What I'd like to do is build a database in the same format as the Lahman database, so they can be merged and imported all at once.

I'm currently collecting Sporting News Guides so I will have all the necessary stats. I plan to start with the 2005 season and work backwards. I feel this database can go as far back as 1946. I've got the TSN guide from that year, and the necessary stats are there. Before that you've got WWII, the segregation problem, and the available minor league stats are vague.

Just wondering if anyone is interested in the project, or had done something similar. I don't want to duplicate effort.

Cowboys3356 · 06-04-2006, 05:59 PM

I believe the Arod23/Garlon database has already addressed these issues but I could be wrong. Might be worth checking out before pushing ahead. In general it is a good idea since it would be a way to get around the need for fictional players.

Le Grande Orange · 06-04-2006, 06:03 PM

Something like this was started by myself last year, and had the help of a number of other interested parties, but it didn't get too far since manually inputting all the information is tedious and time-consuming.

If the information could be scanned directly out of the Guides with OCR that would greatly speed up the process since it would elminate the need for manual data entry.

bp_ · 06-04-2006, 07:07 PM

Quote:

I'm currently collecting Sporting News Guides so I will have all the necessary stats. I plan to start with the 2005 season and work backwards. I feel this database can go as far back as 1946. I've got the TSN guide from that year, and the necessary stats are there. Before that you've got WWII, the segregation problem, and the available minor league stats are vague.

I think you can get the last few years of translated minor league stats from BaseballHQ.com in excel format if anyone has access here.

GaryStephen · 06-04-2006, 08:24 PM

Quote:

Arod23/Garlon database

I saw that, but that one doesn't have historical minor leaguers, and from what I've read they don't intend to make any. What I may do is offer as an additional supplement to theirs. On a long thread about it, several people did ask about the minor leaguers, so apparently there is interest.

Quote:

If the information could be scanned directly out of the Guides with OCR

That's what I'm planning to do.

Quote:

translated minor league stats from BaseballHQ.com

Actually, that's how I got started down this road -- I wanted to do an APBA/BBW league with MLEs, but at the time BBHQ was the only place I could find that did them for pitchers. Now several outfits do them. I'm developing my own formulas for it all.

Le Grande Orange · 06-04-2006, 11:03 PM

Quote:

Originally Posted by GaryStephen

That's what I'm planning to do.

I know a couple of others tried to do it that way, but the type in the Guides was simply too small to OCR properly. Perhaps you'll have better luck, because if the stats can be scanned in via OCR then things should go reasonably quickly.

By the way, do you have the 1961 TSN Guide? I need the Mexican League schedule. It played an interleague schedule with the Texas League schedule that season, and I only have the Texas League half. I'd really like to get the Mexican League half...

DrArbiter · 06-05-2006, 12:20 AM

Quote:

Originally Posted by Le Grande Orange

I know a couple of others tried to do it that way, but the type in the Guides was simply too small to OCR properly. Perhaps you'll have better luck, because if the stats can be scanned in via OCR then things should go reasonably quickly.

I'm convinced OCR simply won't work any time in the foreseeable future, at least at a reasonable quality level. Getting 90% accuracy in OCR is amazing under the best of circumstances, but even if you achieve that, you still have to proof the whole darn thing to get reasonable results for statistical purposes. You're better off just keying in the thing to start with.

But I don't think it's that hard, really. For a separate project, I developed a complete database of all minor leagues from 1982. I find I can input a league-season completely -- batting, pitching, and fielding -- in about 2-3 hours, with fairly good accuracy.

Yeah, there are tons of leagues to be done. But, I suspect many of the concerns of the historically-oriented players would be assuaged simply by having as much AAA data as possible, as far back as possible. That's not a horribly tall order.

Of course, at that, there's still issues to be resolved -- identifying which season lines go with which players, as well as birthdates and origins. But it's a start.

I've been hoping to get around to posting my data (which also so far has some 1981 seasons, as well as some of the open-class years of the PCL). Hasn't happened yet, but when it does, I'll of course post here. Hopefully, at least, it will help avoid reduplication of effort.

Scruff · 06-05-2006, 02:40 PM

Baseball Cube - www.sports-wired.com has a database with minor league numbers for everyone going back a long ways (not sure exactly how far at least into the 70s). I wonder if they'd sell/donate their DB for these purposes.

DrArbiter · 06-05-2006, 03:27 PM

Quote:

Originally Posted by Scruff

Baseball Cube - www.sports-wired.com has a database with minor league numbers for everyone going back a long ways (not sure exactly how far at least into the 70s). I wonder if they'd sell/donate their DB for these purposes.

BaseballCube's data is only complete for the last 4 years. Beyond that, it only contains basically minor league data for players who made it to the majors (or who appeared in a TSN Register in some cases).

From comparisons with some of the data entry I've been doing, it's also of questionable accuracy before that.

Gambo · 06-05-2006, 06:31 PM

I contemplated doing something like this and it would be very cool...

Essentially I decided that the best option was to do a database like mine. I sought to rectify the issues you talked about in terms of stat lines for minor leaguers, cups of coffee etc.

DB work is amazingly time consuming and at least in my case took away from just enjoying the game. (As a shameless plug) Give the newest version of my DB (version 4.0 available soon!!!) a try is see if it is what your looking for.

Spritze · 06-07-2006, 04:02 PM

I have a database I could send you containing the batting, fielding and pitching records of 3,300 players and their minor league stats from 1948 through 1973. It also includes all Japanese players who played 14 or more seasons and all negro league players who played in organized ball. Plus it outputs the data in Lahman format and can calculate MLE for the minor years plus it calculates rating I found that worked better than what 6.5 imported. It is about 400mb so your e-mail won't accept it but maybe it could be posted somewheres or I could mail it to you. Requires Ms-Access.

blackrussian · 06-07-2006, 06:49 PM

Quote:

Originally Posted by Spritze

I have a database I could send you containing the batting, fielding and pitching records of 3,300 players and their minor league stats from 1948 through 1973. It also includes all Japanese players who played 14 or more seasons and all negro league players who played in organized ball. Plus it outputs the data in Lahman format and can calculate MLE for the minor years plus it calculates rating I found that worked better than what 6.5 imported. It is about 400mb so your e-mail won't accept it but maybe it could be posted somewheres or I could mail it to you. Requires Ms-Access.

do you think you can share thru YSI? www.yousendit.com
thanks in advance.

Spritze · 06-07-2006, 06:55 PM

I can send it using this service but I didn't see sharing at first glance. I'll look closer after workee.

DrArbiter · 06-07-2006, 07:45 PM

As promised, I have posted such minor league data as I have in coherent formats:

http://econweb.tamu.edu/turocy/minors

Currently, it's basically all levels of 1982, with AAA for 1981. I've got other leagues in process.

NOTE TO THOSE OF YOU LOOKING FOR A MINOR LEAGUE PLAYERS DATABASE: This isn't one. It's part of a minor league players database, in very raw form. It'll require a lot of care and feeding to become part of a proper database for OOTP. But, it does represent one (small) piece of the puzzle.

There have been a number of attempts at "projects" to try to gather this sort of data. All fizzle mightily, because, as Gambo notes, data entry, validation, and editing into a database is a huge task. Rather, by posting this page, I'm hoping simply to help organize what does exist already for other reasons (the data I'm working on is for another project -- if it becomes part of an expanded OOTP-targeted database or roster set, so much the better!). A more "open source" approach may succeed where more ambitious "projects" have failed.

DrArbiter · 06-07-2006, 07:53 PM

Quote:

Originally Posted by Spritze

It is about 400mb so your e-mail won't accept it but maybe it could be posted somewheres or I could mail it to you. Requires Ms-Access.

May I suggest that if you're willing to distribute this, that you take the time to export it in some non-proprietary format (e.g., CSV)? Aside from the fact that keeping one's data in proprietary formats isn't a good idea, I suspect you'll find that the raw data is far, far less than 400mb in size, which will make it much easier to transmit.

GaryStephen · 06-07-2006, 10:40 PM

Wow -- any existing data in digital format will be a huge help. I'll be sure to give everyone credit.

Spritze, I'll buy some space on XDrive or some such service, and give you instructions on how to upload the file there. That will be tremendously helpful.

Sprtize's database goes from 1948 through 1973. I've got most of the Sporting News Guides from after that.

I'll get a website for the project and post the information here. Anyone who wants to help out may do so.

Incidentally, I intend to publish only converted stats for each players -- not the raw minor league stats. So I wouldn't be republishing anything that is provided to the project.

Gambo · 06-08-2006, 05:54 PM

Go for it! I'm done working on mine so i'd be interested in keeping up with your guys work. If you guys set a limit for how early you want to go then you know the range of what you want to do.

If anybody wants some Sporting News baseball registers (not as complete as the guides but still good.) I had a bunch going back to the 1960s. I've sold a few off but they may help out. Let me know.

DRaysGoRocco · 06-08-2006, 06:17 PM

Quote:

Originally Posted by Gambo

Go for it! I'm done working on mine so i'd be interested in keeping up with your guys work. If you guys set a limit for how early you want to go then you know the range of what you want to do.

If anybody wants some Sporting News baseball registers (not as complete as the guides but still good.) I had a bunch going back to the 1960s. I've sold a few off but they may help out. Let me know.

Is there a link to download version 4.0?

GaryStephen · 06-08-2006, 08:08 PM

Gambo, I'll have to study your DB and see how you handle years for players who had little or no MLB time. I suspect that an MLE conversion would produce similar results, so there's no point in duplicating what you've already done. This would allow me to focus on players who never appeared in the bigs. My concern is that auxiliary players not skew the results. I don't want the 1980 AL batting title to be between George Brett, and some guy who 49-for-136 at AA Montgomery.

What I think I will do first is make a set for 1982 (from Dr Arbiter's data) and post the results and my methodology.

Gambo · 06-08-2006, 11:35 PM

Quote:

Is there a link to download version 4.0?

I sent it to BruceM so it should be up there at least soon...

Quote:

Gambo, I'll have to study your DB and see how you handle years for players who had little or no MLB time. I suspect that an MLE conversion would produce similar results, so there's no point in duplicating what you've already done. This would allow me to focus on players who never appeared in the bigs. My concern is that auxiliary players not skew the results. I don't want the 1980 AL batting title to be between George Brett, and some guy who 49-for-136 at AA Montgomery.

What I think I will do first is make a set for 1982 (from Dr Arbiter's data) and post the results and my methodology.

Basically what I did is took the players actual statistics and filled in the remainder of their games as a below average player. So basically a guy who played 10 games would have 162 games worth of stats. 10 of their actual and 152 of a below average players stats. It was quick and pretty easy. Not overly complicated.

06-04-2006, 05:50 PM	#1
GaryStephen Bat Boy Join Date: Jan 2003 Posts: 5	Auxiliary Historical Database project I'm planning to begin work on a player database that will supplement the Lahman database. By converting real-life AAA and AA stats to their major league equivalents, we can create real players that can be used in conjunction with major league players, thus giving you a larger pool of players to choose from. These players can serve the following purposes: -- Use real players instead of fictional ones to fill in gaps in historical seasons -- Give meaningful stat lines to "cup of coffee" players -- Bring up future stars a year or two sooner, or watch them develop in the minors on their way up -- staff minor league teams for a historical league -- create larger-than-life MLB leagues These supplemental players were used in a long-running APBA/Baseball For Windows league, and they worked pretty well. What I'd like to do is build a database in the same format as the Lahman database, so they can be merged and imported all at once. I'm currently collecting Sporting News Guides so I will have all the necessary stats. I plan to start with the 2005 season and work backwards. I feel this database can go as far back as 1946. I've got the TSN guide from that year, and the necessary stats are there. Before that you've got WWII, the segregation problem, and the available minor league stats are vague. Just wondering if anyone is interested in the project, or had done something similar. I don't want to duplicate effort.

06-04-2006, 05:59 PM	#2
Cowboys3356 All Star Starter Join Date: Sep 2003 Posts: 1,572	I believe the Arod23/Garlon database has already addressed these issues but I could be wrong. Might be worth checking out before pushing ahead. In general it is a good idea since it would be a way to get around the need for fictional players.

06-04-2006, 06:03 PM	#3
Le Grande Orange Hall Of Famer Join Date: Feb 2002 Location: Up There Posts: 15,412	Something like this was started by myself last year, and had the help of a number of other interested parties, but it didn't get too far since manually inputting all the information is tedious and time-consuming. If the information could be scanned directly out of the Guides with OCR that would greatly speed up the process since it would elminate the need for manual data entry.

06-05-2006, 02:40 PM	#8
Scruff Minors (Triple A) Join Date: Sep 2005 Posts: 243	Baseball Cube - www.sports-wired.com has a database with minor league numbers for everyone going back a long ways (not sure exactly how far at least into the 70s). I wonder if they'd sell/donate their DB for these purposes.

06-05-2006, 06:31 PM	#10
Gambo Hall Of Famer Join Date: Mar 2002 Location: Oklahoma Posts: 2,388	I contemplated doing something like this and it would be very cool... Essentially I decided that the best option was to do a database like mine. I sought to rectify the issues you talked about in terms of stat lines for minor leaguers, cups of coffee etc. DB work is amazingly time consuming and at least in my case took away from just enjoying the game. (As a shameless plug) Give the newest version of my DB (version 4.0 available soon!!!) a try is see if it is what your looking for.

06-07-2006, 04:02 PM	#11
Spritze OOTP Historical Czar Join Date: Dec 2001 Location: Bothell Wa Posts: 7,254	I have a database I could send you containing the batting, fielding and pitching records of 3,300 players and their minor league stats from 1948 through 1973. It also includes all Japanese players who played 14 or more seasons and all negro league players who played in organized ball. Plus it outputs the data in Lahman format and can calculate MLE for the minor years plus it calculates rating I found that worked better than what 6.5 imported. It is about 400mb so your e-mail won't accept it but maybe it could be posted somewheres or I could mail it to you. Requires Ms-Access.

06-07-2006, 06:55 PM	#13
Spritze OOTP Historical Czar Join Date: Dec 2001 Location: Bothell Wa Posts: 7,254	I can send it using this service but I didn't see sharing at first glance. I'll look closer after workee.

06-07-2006, 07:45 PM	#14
DrArbiter Minors (Double A) Join Date: Feb 2006 Location: Observing Posts: 177	As promised, I have posted such minor league data as I have in coherent formats: http://econweb.tamu.edu/turocy/minors Currently, it's basically all levels of 1982, with AAA for 1981. I've got other leagues in process. NOTE TO THOSE OF YOU LOOKING FOR A MINOR LEAGUE PLAYERS DATABASE: This isn't one. It's part of a minor league players database, in very raw form. It'll require a lot of care and feeding to become part of a proper database for OOTP. But, it does represent one (small) piece of the puzzle. There have been a number of attempts at "projects" to try to gather this sort of data. All fizzle mightily, because, as Gambo notes, data entry, validation, and editing into a database is a huge task. Rather, by posting this page, I'm hoping simply to help organize what does exist already for other reasons (the data I'm working on is for another project -- if it becomes part of an expanded OOTP-targeted database or roster set, so much the better!). A more "open source" approach may succeed where more ambitious "projects" have failed.

06-07-2006, 10:40 PM	#16
GaryStephen Bat Boy Join Date: Jan 2003 Posts: 5	Wow -- any existing data in digital format will be a huge help. I'll be sure to give everyone credit. Spritze, I'll buy some space on XDrive or some such service, and give you instructions on how to upload the file there. That will be tremendously helpful. Sprtize's database goes from 1948 through 1973. I've got most of the Sporting News Guides from after that. I'll get a website for the project and post the information here. Anyone who wants to help out may do so. Incidentally, I intend to publish only converted stats for each players -- not the raw minor league stats. So I wouldn't be republishing anything that is provided to the project.

06-08-2006, 05:54 PM	#17
Gambo Hall Of Famer Join Date: Mar 2002 Location: Oklahoma Posts: 2,388	Go for it! I'm done working on mine so i'd be interested in keeping up with your guys work. If you guys set a limit for how early you want to go then you know the range of what you want to do. If anybody wants some Sporting News baseball registers (not as complete as the guides but still good.) I had a bunch going back to the 1960s. I've sold a few off but they may help out. Let me know.

06-08-2006, 08:08 PM	#19
GaryStephen Bat Boy Join Date: Jan 2003 Posts: 5	Gambo, I'll have to study your DB and see how you handle years for players who had little or no MLB time. I suspect that an MLE conversion would produce similar results, so there's no point in duplicating what you've already done. This would allow me to focus on players who never appeared in the bigs. My concern is that auxiliary players not skew the results. I don't want the 1980 AL batting title to be between George Brett, and some guy who 49-for-136 at AA Montgomery. What I think I will do first is make a set for 1982 (from Dr Arbiter's data) and post the results and my methodology.