Bias in OOTP Historical data?

[VanillaGorilla]

One of these players is not like the others.....

The included clip is of the career ERA leaders in my current league.

The player is not Alex Serrano (though he is similarly unique).

more in a bit

[ezpkns34]

I see the problem(s). The game is giving out names like Smoky Joe & Vinegar Bend in the 21st century

[David Watts]

never mind.

[VanillaGorilla]

Originally Posted by ezpkns34

I see the problem(s). The game is giving out names like Smoky Joe & Vinegar Bend in the 21st century

LOL

The answer is Bill Hawke.

more to come

ADD: Just saw your edit note. Whatever the reason was behind your post, the post still made me laugh.

[VanillaGorilla]

Yes, this is a random debut league which began in the 2004 season.

I have a 22 year-old Mike Torrez in AAA and I clicked on his RL stats to see how he was developing in comparison to RL (I am using 5 year recalc with player development on).

The fact that Torrez had won 185 ML games had been deleted from my memory bank. I had forgotten how long he had pitched and how well he had pitched throughout his career. It is a shame that when I think I think of Mike Torrez, I think of Dickie Thon, and that is about it.

As I am reviewing his career line I see he has a career ERA+ of 121. That kind of career figure puts him in HOF company. Was he THAT good? Triple shame on me if he was THAT good and all I remembered about him was one tragic pitch.

I go to BBREF and his career ERA+ there is 98. I feel better. Torrez pitched for 18 seasons at the ML average (obviously having seasons much better than average, and some below average, but he was a quality ML pitcher for many years.

A 121 ERA+ is what Don Drysdale carried for his career. Nothing against Torrez, but he was not Drysdale.

Bill Hawke is the answer to the question because he is the only player in the list who has a career ERA+ on BBREF that is better* than the ERA+ on his OOTP RL stat sheet.

In my HOF threads, I take the time to note that sim displays of OPS+ and ERA+ are non-park adjusted (npa) because a pitcher with a 4.00 ERA in Coors the same year as a pitcher with a 4.00 ERA in Safeco gets the same ERA+ in OOTP. Minor detail that I can live with. That isn't a game breaker.

However, the historical data for the players is not park adjusted, either. Comparing npa ERA+ to actual ERA+, there should be an approximately equal number of players with a higher npa ERA+ than park adjusted ERA+ and about half with a lower npa ERA+ than pa ERA+.

That 20 out of the top 22 career ERA pitchers in my simulation just happen to have a better ERA+ listed in their OOTP RL stats (one is exactly equal, Serrano) indicates that the historical data is biased and when neutralized is creating better output for players who pitched in pitcher friendly environments because their adjusted stats for the sim are not being properly adjusted for RL.

The screen shot goes down to 22nd place because of space. However, I have to go down to the 25th place where Wade Miley becomes only the second pitcher on the All-Time ERA list to have an OOTP RL ERA+ figure that is not as good as the ERA+ figure on BBREF.

[VanillaGorilla]

What is shown here is potentially a game breaker as far as historical recreations/emulations go.

If the neutralized data is skewed by not neutralizing the actual historical data properly and this results on player performance in a dissimilar simulated environment by the happenstance of playing the bulk of one's career in an environment conducive to that player's attribute, then the sim output becomes a capricious representation of a player's ability.

It is fine that a player can overperform his career numbers (or underperform) due to factors ranging from injuries, development, and a random number generator. But if the foundation of a player's ability is determined by the fact that he played in an extreme environment for his career, when the OOTP user has asked the program to neutralize for these factors, that is a problem.

I will explore data from previous leagues and see if this is, in fact, a bias that is skewing results or simply and aberration of a one league sample set.

[Scruff]

Don't forget, the top players are also going to be the ones who had the most luck (all things equal).

I think you need to look at everyone, not just the leaders. You leaders will have disproportionately outperformed their real life ability - the ones that underperform won't be on the leaderboard.

You could have something here, but I think more investigation is needed.

[VanillaGorilla]

Originally Posted by Scruff

Don't forget, the top players are also going to be the ones who had the most luck (all things equal).

I think you need to look at everyone, not just the leaders. You leaders will have disproportionately outperformed their real life ability - the ones that underperform won't be on the leaderboard.

You could have something here, but I think more investigation is needed.

The leaders will have often overperformed their RL ability. True. The question is whether or not this overperformance is influenced by skewed historical data in OOTP that is the basis for the overperformance output.

As I said earlier, we should see about half of the players with npa ERA+ above their pa ERA+ and half with npa ERA+ below pa ERA+.

If the sim is using npa stats to determine the ratings base, then what we will see on the leader boards is a disproportionate number of players with stats that would be lowered with proper neutralization.

[VanillaGorilla]

Since he is the player that started me down this path, I pulled up his BBREF page, again and compared the neitralized stats there with the neutralized stats in OOTP 14.

On his neutralized RL pitching page in OOTP, his ERA+ is listed as 110, down from the 121 on his main RL stats page. This is still much better than his BBREF ERA+ of 98.

His Neutralized ERA on BBREF is 4.44. In OOTP it is 4.24.
BBREF neutralized h/9 is 9.9. In OOTP it s 9.49.

On what the neutralization is based for the folks at BBREF in comparison to OOTP, I do not know. However, ERA+ IS a neutralized stat, and that number, even with the neutralized OOTP number of 110 is significantly better than his BBREF number of 98. The ERA+ number, if properly presented, should not change through neutralization, but it does in OOTP. There is a reason it isn't listed under neutralized stats on BBREF. That is because it would simply be redundant to do so, since it is listed in the primary display, already.

[VanillaGorilla]

I looked up the all-time Colorado Rockies leaders in IP to compare the data of those players on BBREF to that in OOTP.

The ERA+ display for RL stats was not in v13, so I had to use v 14 to look up the OOTP data. The first two players with the most IP in Colorado history have not appeared in my v14 league. Number three, Jason Jennings, has.

Jennings pitched 6 of 9 big league seasons as a Rockie. 941 of his 1128 1/3 IP in the bigs were as a member of the Rockies. His RL ERA+ on BBREF is 98 (coincidentally the same as Mike Torrez). His RL ERA+ in the OOTP RL stats display is 92.

Jennings neutralized ERA on BBREF is 4.31. In OOTP it is 4.43.
Jennings neutralized H/9 on BBREF is 9.3. In OOTP it is 9.41.

So, Mike Torrez, who pitched much of his career in St Louis, Montreal, Yankee Stadium (before the Winfield fences) and Shea stadium has a RL ERA+ in the OOTP database of 121, and Jennings has one of 92. They both should be 98. Because of this, Mike Torrez gets an arbitrary advantage in OOTP over Jennings. This is not acceptable.

[David Watts]

Originally Posted by VanillaGorilla

I looked up the all-time Colorado Rockies leaders in IP to compare the data of those players on BBREF to that in OOTP.

The ERA+ display for RL stats was not in v13, so I had to use v 14 to look up the OOTP data. The first two players with the most IP in Colorado history have not appeared in my v14 league. Number three, Jason Jennings, has.

Jennings pitched 6 of 9 big league seasons as a Rockie. 941 of his 1128 1/3 IP in the bigs were as a member of the Rockies. His RL ERA+ on BBREF is 98 (coincidentally the same as Mike Torrez). His RL ERA+ in the OOTP RL stats display is 92.

Jennings neutralized ERA on BBREF is 4.31. In OOTP it is 4.43.
Jennings neutralized H/9 on BBREF is 9.3. In OOTP it is 9.41.

So, Mike Torrez, who pitched much of his career in St Louis, Montreal, Yankee Stadium (before the Winfield fences) and Shea stadium has a RL ERA+ in the OOTP database of 121, and Jennings has one of 92. They both should be 98. Because of this, Mike Torrez gets an arbitrary advantage in OOTP over Jennings. This is not acceptable.

This stuff is all way above my head, but could Torrez be getting a bump due to his years pitching in Boston?

[VanillaGorilla]

Originally Posted by David Watts

This stuff is all way above my head, but could Torrez be getting a bump due to his years pitching in Boston?

It is the opposite. Torrez gets his numbers knocked down from the Boston years because there is not a park adjustment for the base stats.

Any neutralization effort will have a systemic bias, including the one in use on BBREF (based on Bill James work) and it is this neutralization process that OOTP uses, in part.

I say "in part" because for the historical neutralization the part of the formula that adjusts for the home ball park is simply left out.

I performed the same check for the OPS leaders in my current league that I did for the ERA leaders. 20 of the top 23 places (I only checked through 23...what number 24 is, I don't know) had neutralized slash lines in OOTP better than on BBREF. Reggie Jackson's slash for the neutralized data was exactly the same across the board (the universe revolves around Reggie???).

I will post more on this later. There appears to be a discrepancy in how hitters and pitchers are, respectively, neutralized, also.

One thing is clear: the neutralized data that is used for historical players (pre 2009...more on this later, also) is NOT adjusted for the actual home park the player pitched/batted in, and this is leading to biased results which favors hitters from hitting parks and pitchers from pitching parks.

[majesty95]

Are you using the Lahman DB or Spritze/Gambo? It sounds like you are using Lahman and selected "neutralized" stats when setting up the league? I was told not to do this although I'm not sure the reason. However, if not using neutralized stats in a historical replay then the Rockies hitters and Safeco pitchers are always going to be far ahead of even their RL bloated stats, right?

[VanillaGorilla]

Originally Posted by majesty95

Are you using the Lahman DB or Spritze/Gambo? It sounds like you are using Lahman and selected "neutralized" stats when setting up the league?

Yes. I used the default database and neutralized the stats.

Originally Posted by majesty95

I was told not to do this although I'm not sure the reason.

If the reason is because the stats are not park adjusted, that would be a good one. The game does not park adjust OPS+ or ERA+ that results from play, either, which is a separate issue from any database selection.

Originally Posted by majesty95

However, if not using neutralized stats in a historical replay then the Rockies hitters and Safeco pitchers are always going to be far ahead of even their RL bloated stats, right?

Of course. However, that is the case even using neutralized stats. Troy Tulowitski has better OOTP neutralized stats compared to BBREF and Ichiro has poorer neutralized OOTP RL stats compared to BBREF.

This is a problem. The user (ok, maybe it is just me) is led to think that by neutralizing stats he is putting historical players on a level playing field in regards to the era in which they actually played and the home park environment in which they actually played. The OOTP neutralization process only addresses the first aspect.

[majesty95]

Dang. Since I was told to pick "Real life stats" doubly inflated stats now? I was under the impression that the "neutralized stats" selection was only used if using the Spritze/Gambo DB. Sigh. Its all so confusing.

[VanillaGorilla]

"Neutralized Stats: The "Garlon and Spritze" option. Current player ratings will be assigned based on a player's neutralized statistics, giving perhaps the most accurate assessment of individual player ratings on an even playing field."

This was my selection, as opposed to selecting:

"Real Stats: The "standard Lahman" option. Current player ratings will be assigned based on a player's historical statistics. So, in this case, a player who played in a hitter-friendly park throughout his career could end up with inflated ratings."

These neutralized stats, as displayed on player pages, are not factored for ball park, as is inferred by the manual. They are adjusted for the year, but that is all.

[majesty95]

So what causes these kinds of inaccuracies:

1984 Sim

Dave Kingman 51 HR (real life 35)
Ken Phelps 47 HR (real life 24)
Rich Gedman 42 HR (real life 24)
Wayne Gross 41 HR (real life 22)
Harold Baines 40 HR (real life 29)

Dave Collins 83 SB (real life 60)

Juan Beniquez .372 AVG (real life .336)

These were all leaders in the league. Obviously the home runs were the most glaring. Outside of the Collins and Beniquez anomalies the rest of the averages and stolen bases seem reasonable. My question is, though, why weren't all of these players closer to their real life stats when using a one year recalc and optimizing for single season replay? I was OK with a 5-10% fluctuation one way or the other but these were all very extreme...

[JMDurron]

Were all of those players on their historical teams?

Ken Phelps hit 24 HRs in 1984, in only 101 games. That would be roughly 38.5 HRs over 162 games if my math is right, so that 47 becomes slightly more reasonable.

I can't even explain Rich Gedman's real 1984 season, nevermind your version. Maybe if he was in a LHH HR paradise of some kind, but not in Fenway Park.

Gross, like Phelps, would have had ~28 HRs over 162 games, so it's slightly less of a distortion than you'd think at first.

Dave Collins, similarly, had 60 SBs in 128 games IRL. That would be almost 76 over 162, so this doesn't strike me as an anomaly at all, and is within your tolerances.

I find your single-season numbers significantly less compelling as evidence of a problem than VanillaGorilla's career leaders and ERA+/OPS+ discrepancies. Some outliers, even freakish-looking ones, would have to be expected to occur during a completely real replay of any given season. Your results could be the symptom of an underlying issue, or just standard single-season variation, where offense as a whole was above the original 1984 baseline. You basically had 1987 happen in 1984.

[VanillaGorilla]

Originally Posted by majesty95

So what causes these kinds of inaccuracies:

1984 Sim

Dave Kingman 51 HR (real life 35)
Ken Phelps 47 HR (real life 24)
Rich Gedman 42 HR (real life 24)
Wayne Gross 41 HR (real life 22)
Harold Baines 40 HR (real life 29)

Dave Collins 83 SB (real life 60)

Juan Beniquez .372 AVG (real life .336)

These were all leaders in the league. Obviously the home runs were the most glaring. Outside of the Collins and Beniquez anomalies the rest of the averages and stolen bases seem reasonable. My question is, though, why weren't all of these players closer to their real life stats when using a one year recalc and optimizing for single season replay? I was OK with a 5-10% fluctuation one way or the other but these were all very extreme...

Back to my OOTP Fanboy mode....

Your league totals for HRs should be something close to 1980 HRs for the AL in 1984. 1980 HRs is the base for the league total. They will be hit. It is how the game is set up to execute, which makes it different in structure from Strat or APBA or DM. And it is a genius methodology that allows the game to be what it is.

Kingman: All things being equal, if he hits 10 HRs in April, we don't expect him to hit 25 HRs the rest of the season, we expect 30. If he plays 162 games we now have a 38 base vs 35 for our expectancy. Conversely, if he goes into May without a HR, we don't expect him to hit 35 (or 38) HRs in 5 months. At the same time, he is not prohibited from hitting 35, or 38 HRs simply because he hit 0 in April.

Player injuries will have an impact on output. If other sluggers go down, then the remaining sluggers benefit in OOTP because the ratings are relative to the other players in the league.

There are a ton of other factors. A few can be the pitchers faced (were they better or worse than the ones that they actually faced on a given day vs a given team? Is the game a blow out and the mop-up guy is tossing instead of Willie Hernandez for a particular PA?) You are looking at THOUSANDS of stats that are recreated. There will be outliers. This is not an indication of a game flaw. If there were not outliers, then that would be an indication that the game is not honest.

Did Jack Morris throw a no-hitter? He didn't? Is that 'unrealistic'?

When we look at a re-creation and look at dozens of stats from hundreds of players, they are not going to synch up in each and every category. Did Kingman miss a series in Boston in RL that he didn't in the sim?

How many PAs did Gedman get? Probably more than in RL.

What happened to Tony Armas? Did he get hurt? If so, then someone hit those league leading HRs, somewhere, and they would be distributed disproportionately to other power hitters who are now more exceptional power hitters in relation to the league by virtue of the fact that they no longer have Tony Armas to be evaluated against in comparison.

The game is brilliant, in this regard. My issue is that there is inappropriate biases in place that give some players a better evaluation than they deserve (and likewise some a poorer evaluation than what they deserve) which skews the output.

[majesty95]

Well, I'm using historical teams and historical transactions so all things should be even.

Kingman did play 162 games so that makes some sense but that's still a 38% jump in only 80 more AB (629)

Gross and Phelps I can kind of buy.

Baines hit 11 more HR in 63 more AB (632)

Part of the problem is that the AL hit 2155 home runs vs 1980 in real life and 8.8% increase. The NL had 1239 vs 1278 in real life. I thought by having the league total modifiers imported each season keeps the league totals the same buy allows variance in which players do what. I didn't think there should be an 8% increase?

I know its not the same issue but it came to mind when reading through this. I should have only had 1-2 guys hit 40+ home runs and quite a bit fewer home runs overall.

FWIW, hits were within 1.0% of real life and stolen bases were within 3.0%. I can live with those but the HR seem way too far off to me