Spritze/Garlon DB

[Garlon]

Hello fellow historical simmers,

Spritze and I started working on a new modified historical DB for OOTP recently. Spritze has done 99% of the work on it and has made (and is still in the process of making) many meticulous edits to provide the best results with OOTP.

The biggest modification we have made with the DB is that it will have park-neutralized player stats. We are using the method described by Bill James in his Historical Abstract to both park-neutralize player stats and convert them to a 750-run environment. If you go to baseball-reference.com you can look up any player and click an option to neutralize stats to a 750-run park-neutral environment. The stats in our DB will essentially be the same as what you would find there since we are using the same conversion technique.

Don't be too concerned with the fact that teams weren't scoring 750-runs per season in the Deadball Era and that this conversion will have a negative impact on simulation results. In fact, just the opposite will happen and we willl see even better results across all eras. OOTP's League Totals Modifiers control the environment and ratios of things like BA and HR per AB, the conversion of the stats in our DB to a higher run-scoring environment just changes player ratings...not the results of the simulation. The important aspect of player ratings in OOTP is how players compare relative to one another, not the absolute value of the ratings. This park-neutral 750-run conversion puts all players on an equal footing for comparison so that players who played for the Dodgers in Dodger Stadium for instance who have poor hitting stats because that is perhaps the worst hitter's park in baseball will benefit from the conversion, while players who played in Coors Field have their statistics reduced. Willie Davis who played fo the Dodgers 1960-1979 had a career .279 BA, after conversion he goes to .301. This is the primary advantage this DB will provide - better representation of a player's abilities.

We are also normalizing player fielding stats to their year and league. Players who played on teams with high strikeout pitching staffs had fewer opportunities to make plays in the field. We adjust player Put Out and Assist totals to adjust for things like this. We also adjusted Double Play totals of all players on teams based on thei teams Expected Double Plays Per Game compared to the League Average Double Plays Per Game of that season. Both of these adjustment formulas come from Bill James' book Win Shares.

Readme/FAQ

1. Fixed missing batter Strikeout data.
American Association 1882-1888 and 1890
National League 1897-1909
American League 1901-1912

2. Fixed missing Stolen Base data 1876-1885.
Stolen Bases were set to the Deadball Era frequency on a per times reaching first base basis.

3. Adjusted Stolen Base data 1871-1875 and 1886-1897.
Due to differences in the definition of a stolen base before 1898, stolen bases during these periods were corrected to the frequency of the Deadball Era.

4. Fixed missing Caught Stealing 1871-1919 NL/AL and 1926-1950 NL.
For the 1871-1919 period, league SB% was set to a 55% success rate.
A formula was used so that players would range above and below 55% based on their frequency of scoring runs.
In general, better players have higher success rates, but the league as a whole remains at 55%.

For the 1926-1950 period in the NL, league SB% was set to match the SB% of the AL in any given season.
The same formula was used to give higher success rates to better players.

5. Deleted any playes who debuted in the following leagues and did not have at least 550AB or 162IP in the real major leagues.
Union Association 1884
Players League 1890
Federal League 1914-1915

This was done to prevent OOTP leagues from being flooded with non-major league talent.

6. Deleted any WWII replacement players who debuted between 1942-1945 who did not have at least 550 AB or 162 IP after the 1945 season.
This was done to prevent OOTP leagues from being flooded with non-major league talent.

7. Pro-rated any missed seasons during WWII for the 400+ players who served in the military.
Their estimated stats are based on their performance before and after WWII.

8. Fixed gaps in the playing records of all players in the DB.
Many players missed seasons for various reasons during their career, being sent to the minor leagues, injuries, military service, etc.
We filled in these gaps of a player's recod by taking an average of the bookend seasons of the gap and applying it to all missing seasons in that gap.
This was done so that no players would be permanently missing when you start a league in OOTP.
It will also help with recalc mode by providing additional statistics.

9. Fixed low AB or IP totals for players who had at least one full season of career stats but may have only had a few AB or IP in a given season.
We pro-rated any missing AB to give a player 251 AB on the season based on his career average.
We pro-rated any missing IP to give a pitcher 40 IP on the season based on his career average.
This was done so that OOTP can provide better ratings in these instances.
All players in the DB who had at least 550 AB career or 162 IP career, will have at least 251 AB and 40 IP in any season in which they played.

10. Adjusted cup-of-coffee players to replacement level based on the seasons and their position.
This was done becaue these players do no have a large enough sample of AB or IP for OOTP to create valid ratings. We would rather prescribe the replacement level stats than have OOTP do it.

11. Limited the # of Starting Pitchers in the DB.
Set the Games Started to 0 for all pitchers with fewer than 125 career Games Started.
This was done to keep pitchers who had very short careers from stealing rotation spots from more deserving ptichers in OOTP.
These pitchers will now import as Relievers with an Endurance of roughly 2 IP.

12. Added extra player ID's for players who had significant careers as both a position player and a hitter, like Babe Ruth.

13. Extended the careers of significant post-integration players.
Due to segregation in baseball until 1947, many players like Jackie Robinson did not have a chance to play in the major leagues at a younger age.
These players had their careers extended backwards so that they now debut at age 21.
Campanella, Doby, Easter, Irvin, Minoso, Robinson.

14. Added 50 Negro League stars to the DB.
These players were given the career average stats based on MLE's of their available Negro league stats when possible. Otherwise we base their stats on comparable major league HOF'ers.

The Negro League Players by position, their career span, and their major league equivalents:

c - Clarence Wlliams (1889-1905) - Ewing
c - Louis Santop (1911-1927) - Bresnahan
c - Biz Mackey (1919-1935) - Cochrane
c - Ted Radcliffe (1924-1940) - R. Ferrell
c - Josh Gibson (1933-1949) - Foxx

1b - John Frye (1883-1899) - Connor
1b - Ben Taylor (1910-1926) - G. Kelly
1b - Buck Leonard (1929-1945) - Mize
1b - Buck O'Neil (1933-1949) - Terry

2b - Frank Grant (1887-1903) - McPhee
2b - Bingo DeMoss (1911-1927) - Evers
2b - Newt Allen (1923-1939) - Frisch
2b - Sammy Hughes (1932-1948) - Bi. Herman

3b - Sol White (1890-1906) - J. Collins
3b - Candy Jim Taylor (1905-1921) - Baker
3b - Jud Wilson (1915-1931) - Lindstrom
3b - Judy Johnson (1921-1937) - Traynor
3b - Ray Dandridge (1935-1951) - Kell

ss - Grant Johnson (1896-1912) - G. Davis
ss - Pop Lloyd (1906-1922) - Wagner
ss - Dick Lundy (1920-1936) - T. Jackson
ss - Willie Wells (1927-1943) - Appling

lf - George Williams (1885-1901) - McCarthy
lf - Pete Hill (1902-1918) - Clarke
lf - Mule Suttles (1922-1938) - Wilson
lf - Turkey Stearnes (1923-1939) - Klein

cf - Bud Fowler (1879-1905) - O’Rourke
cf - Spotswood Poles (1911-1927) - Carey
cf - Oscar Charleston (1918-1934) - Heilmann
cf - Cool Papa Bell (1925-1941) - Cuyler

rf - Benjamin Boyd (1880-1896) - K. Kelly
rf - Cristobal Torriente (1916-1938) - S. Rice
rf - Martin Dihigo (1927-1943) - L. Waner
rf - Willard Brown (1935-1951) - Slaughter

p - William Whyte (1882-1898) - Clarkson
p - George Parago (1883-1899) - Radbourn
p - William Malone (1887-1903) - Rusie
p - George Stovey (1887-1903) - Young
p - Rube Foster (1901-1917) - Plank
p - Smokey Joe Wlliams (1907-1923) - Waddell
p - Jose Mendez (1908-1924) - Marquard
p - Bullet Joe Rogan (1911-1927) - Rixey
p - Andy Cooper (1918-1934) - Grimes
p - William Bell (1919-1935) - Haines
p - Bill Foster (1926-1942) - Lyons
p - Satchel Paige (1928-1944) – Grove
p - Chet Brewer (1928-1944) - C. Hubbell
p - Ray Brown (1929-1945) - Gomez
p - Hilton Smith (1933-1949) - Di. Dean
p - Leon Day (1938-1954) - Feller

15. Fixed miscellaneous problems in the DB files.
Erased the pitching stats of non-pitchers.
Erased the non-pitcher fielding stats of pitchers.
Created career average hitting stats for pitchers.
Fixed missing birthdays, height, weight, bats, throws fields in the master file.

16. Converted Batting and Pitching statistics of all players to a 750-run environment.
This conversion will give the best representation of a player's skills in a park-neutralized environment converted to a league average scoring of 4.63 runs per team game.
This does not mean that players who have their statistics boosted will become sperior players than what they should be.
OOTP's league totals modifier system keeps league totals in check, so you will not be scoring 4.63 runs per game in the Deadball Era even though we adjusted the stats this way.
The translation of the stats gives a better representation of a player's abilities relative to the other players in the league by eliminating park effects.

17. Adjusted the fielding statistics of all players.
We normalized the Put Outs and Assist totals of alll players based on the balls in play per game of their team versus the balls in play per game average of their league.
This was done so that players on teams whose pitchers did not strike out many batters do not get superior defensive ratings in OOTP just because they had more opportunities to make plays.
We adjusted Errors to the new Put Out+ Assist totals so that a player's FLD% remained the same after the conversion.
We normalized Double Play statistics based on a teams Expected Double Plays per Game versus the league's Double Plays per Game.
This was done so that players on teams which allowed alot of base runners from Hits and BB do not receive superior Turn DP ratings in OOTP due to their extra opportunities.

18. Added 14 Japanese greats to the DB.
Except for Sadaharu Oh, we have not selected these yet.

19. Extended backwards the careers of modern Japanese players in MLB.
This way players like Ichiro Suzuki and Hideo Nomo will start their careers at age 21.

[Royals]

Where can we find this db? Thanks for your work.

Royals

[Garlon]

When we finish it, I will post a link where everyone can d-load it.

[greendiver]

I am looking forward to your database. It appears you are tackling quite a number of the things that are a little off with Lahman.

I'm curious how you intend to handle fielders who have only a couple of appearances at a position in a season. OOTP tends to play these players in odd positions through their career if they have a rating based on even limited appearances. Full time SS in my current 2001 league include Tony Womack, Kelly Gruber, and Matt Williams. The best 2B is Nomar Garciaparra, who played one game there in his career. Chris James is the starting 3b for the Marlins and Houston's backup C is Richard Hidalgo. I know that the database will only go part of the way to solving this problem, but if you aren't already planning on doing it, may I sugest eliminating fielding positions where a player played in only a minimal number of games.

Thanks for your work.

[Garlon]

greendiver - We plan to eliminate positions in the fielding record at which a player played less than 20% of his games. This should eliminate most of the problems with the AI playing guys out of position just because they had a handful of games played in their fielding record and thus get rated there.

[greendiver]

Excellent. I am really looking forward to your database.

[fredmanrique]

Garlon,
Just wanted to say that I can't wait for this DB. I found your previous DB's to be the best for what I like to do, and was missing your work until now.
One question- is there any way you can include a small explanation or tutorial about adding a player here and there? This DB you are about to release with the added Negro League and Japanese players is exactly what I was hoping for, but if I feel like there end being a couple of players missing that I had hoped to be in there I don't know how to add them myself.
Also, not to get greedy, but... will you be releasing this DB with random debut years, etc?
Either way, thanks again in advance- can't wait!!!

[No Pepper]

Garlon and Spritze, like the others I eagerly await your release. Apart from other questions I have about your database, how are you differentiating "cup of coffee" players who you adjust to replacement level and players that you delete, who have less than 550 AB etc, played in the FL, WWII-era, etc? Maybe a somewhat naive perspective of mine would say these are similar type players: maybe not quite major-league level, but certainly minor-league, that just had (un)lucky circumstance dictate their careers.

[Garlon]

fredmanrique - to create a custom player in the DB you need to create a player id in the master file, then create his batting/pitching/fielding stats in those csv files as well.

no pepper - the cup of coffee guys are those who had fewer than 200 career AB or 154 IP. Not all of the UA/PL/FL players get dumped, only the ones whose career were basically solely limited to those leagues. A WWII replacement player is defined as a player who debuted between 1942-1945 but did not have a full season's worth of playing time post-1945.

[kagnew35]

Garlon, any news??? any idea when available??

[Garlon]

Still some finishing touches and testing o do. Just finishing up some edits to the fielding files so that players don't get used out-of-position by the AI. Still gotta make some fixes to the master csv file too.

[Chief Wahoo]

Just wanted to say that this sounds like a great set and I can't wait for it to become available.

[No Pepper]

Any update? I hope all is going well.

[Spritze]

Yup this db appears to be done. All that is left is to test the beejebees out of it to catch errors of which I caught one today in the formulas that created the batting CSV. Brain fart is all I can say about that.

[fredmanrique]

That is great news! Can't wait.

[Hurkman]

Awesome! I have a new idea for a dynasty thread, but I wanted a database with at least some historical Negro League players, and this sounds even better than what is out there now. I can't wait to try it out!

[fredmanrique]

How's it going? No pressure, just wondering...

[Garlon]

An update:

I converted all batting lines of all players 1871-2006 to the park neutral 4.63 runs per game environment. If you go to baseball-reference.com you can go to any player's page and click "neutralize stats" then click the 750-runs option. The neutralized stats you will see there are essentially the same as what will be in this DB. I used 10 decimal places in the conversion process before finally rounding the stat totals, so there are very slight discrepancies of a couple Hits or AB here or there over the course of a player's career.

Ty Cobb's career totals
Baseball-reference vs Garlon totals

G: 3216 vs 3216
AB: 12408 vs 12407
R: 2609 vs 2610
H: 4732 vs 4730
2B: 808 vs 809
3B: 337 vs 335
HR: 128 vs 128
RBI: 2253 vs 2252
BB: 1392 vs 1392
SB: 1027 vs 1029

The conversion method used by baseball-reference was published by Bill James in his Historical Abstract pg 740-743. Bill James provides a conversion of Willie Davis's career into this context.

Willie Davis career totals

Baseball-reference vs Bill James vs Garlon

G: 2439 vs 2429 vs 2439
AB: 9496 vs 9473 vs 9494
R: 1457 vs 1462 vs 1456
H: 2858 vs 2860 vs 2858
2B: 441 vs 447 vs 441
3B: 153 vs 154 vs 153
HR: 199 vs 201 vs 199
RBI: 1248 vs 1250 vs 1244
BB: 465 vs 459 vs 465
SO: 981 vs 977 vs 981
SB: 443 vs 457 vs 443


After I finished that I fixed missing player strikeout totals for 1897-1912 by estimating the missing strikeouts for individual players based on their team's K:Out ratio in a given season.

I also adjusted the 1871-1875 and 1886-1897 stolen base totals to the frequency of stolen bases per times reaching base of the period of 1897-1909. The definition of a SB was different or inconsistent before 1897 so this adjustment needed to be made. The period of 1876-1885 had no stolen base totals whatsoever, so I had to estimate those using a formula based on SB per times reaching base from 1886-1897. This was all done after finding the new data after converting to the park and run-neutral environment.

Caught stealing data is essentially not available for 1871-1919. There is some available data from 1914-1915, which gives a recorded success rate of stealing of 55%. The period of 1920-1925 also yields a 55% success rate. I finished estimating the caught stealing of all players 1871-1899 based on a formula which puts the league average at 55%, but allows for individual players to be above or below that mark...generally speaking, great players tend to turn out with an above average SB% using this method.

I still have to fix the missing Caught Stealing data for the NL and AL from 1900-1919, and the NL from 1926-1950.

After I get that finished, we have to go in and do a host of minor fixes:
-Pro-rate the missing WWII seasons for players

-Adjust players who only had a cup-of-coffee and have very low stat totals. We will set these players to the replacement level.

-Adjust low AB seasons of players who had at least 500 career ABs. For example, after neutralizing his stats, Ty Cobb in 1905 only had 163 AB. We will bring him up to 251 AB by crediting him with 88 extra AB pro-rated at his career average. So in 1905 instead of him having 163 AB and 43 hits (.264), he will have 251 Ab and 77 hits (.307). This type of edit will alleviate alot of issues in OOTP involving ratings of players with low AB totals since everyone in the DB will be adjusted to a minimum of 251 AB in any given season.

-Fix gap seasons in the playing records of all players. For example, Willie Mays did not play in 1953 (military service), so if you start a league in 1953 Mays will not be in the game, nor will he import in 1954 because he was not a rookie in 1954 so h will be missing from your universe forever. There are players in any given season in history who missed a year here or there due to injury, military service, or just getting sent back down to the minor leagues for an entire season. We will fill in these "gap seasons" in the records of all players by taking an average of their seasons before and after a given gap in their records to estimate their performance level. Willie Mays in 1953 will have a stat line of 366 AB and 120 hits (.328).

Once we do all of that then the batting files will be finished. Then we need to work with the pitching and fielding files.

[Captain Walrus]

This database sounds amazing. I always screw something up when I edit the LahmanDB myself, and the player editor in game is not easy to use in terms of neutralizing statistics.

This database is something I greatly look forward to for my cross-era leagues.

[Spritze]

Version 1 of this db should be uploaded to one of the hosters by Wednesday.

I have simmed 1901 to 2000 twice and also played out twenty years (1940 to 1959) and all seems well. Nothing horribly out of place that I noticed.

Please post suggestions and comments in this thread as we are already working on version two. I am sure there are improvements that can be made to make gameplay even better.

Currently the main change in version 2 will merely be that the 750 run environment conversion will happen first and then all the changes/additions/deletions will be applied. In this version all the changes/additions/deletions are made and then the conversion is applied. The difference between the two appears to be very minimal and probably has no impact whatsoever on the OOTP ratings produced.