Historical Accuracy Overrated?

[pstrickert]

Originally Posted by Le Grande Orange

But what about all the players those eight guys played against? You can't look at just those eight players in isolation; their stats are generated by their game interactions with all the other players in the league.

True enough. How would I go about assessing that? I would like to think that player ratings matter in OOTP. I'd like to think a team with a lineup of 4- and 5-star hitters would hit well against a pitching staff with 1- and 2-star pitchers. Based on what I've seen, though, I'm not inclined to believe that would happen. With the other variables reduced to a minimum, I would think we could predict that the lineup with the real good hitters would pound the team with exceptionally thin pitching. I'm not convinced, though. That 1-star pitcher could pitch like a 20-game winner (and do it all season long!). That 5-star hitter could produce average results. I'm not saying that the results should be entirely predictable. I'm simply exasperated that the results are all over the map. (In that case, what good is a historical league? The players may as well be entirely fictional if they have no semblance to their historical counterparts.)

[pstrickert]

How would these results affect someone who plays a fictional league? If you've got a 5-star closer on your team, but he's not putting up all-star type numbers, what's the problem? Is he unhappy? Has he hit the downward slope of his development curve? Or do his ratings, as good as they are, mean little in terms of game results? Sim that season one time, and he might end up with a 4.11 ERA with 16 saves. Rewind it (if that were possible) and sim it from that same exact point again, and he might end up with a 2.11 ERA with 46 saves. That kind of variation is possible, unfortunately, if what I've seen holds up to scrutiny. But wouldn't we expect a little more consistency than that (even in a fictional league)?

[Syd Thrift]

We've found that if anything pitchers tend to be too consistent from one year to the next. It may be that your league isn't weighting players particularly well in terms of stars and they just aren't being generated to be all that good. Mike Caldwell in particular was kind of a weird case: good in 1982 but IIRC at the time he was throwing an 80 mph heater and he didn't have a lot of success at all after then. If you're allowing players to age, a week after you started the league it's fictional anyway.

This is probably the hardest pill to swallow WRT real-life baseball. There is a tremendous amount of luck involved in terms of which pitcher turns into a stud and which one gets hurt or just doesn't develop. If the real-life Todd Van Poppels and Brien Taylors and David Clydes of the world didn't pan out, why should you expect the fictional Mike Caldwells and Lamarr Hoyts to?

[Le Grande Orange]

Originally Posted by pstrickert

I'm simply exasperated that the results are all over the map. (In that case, what good is a historical league? The players may as well be entirely fictional if they have no semblance to their historical counterparts.)

The results are variable because OOTP is open-ended. Each time you sim a game, the program is essentially flipping coins to determine the outcome of each game event based on the player ratings and other factors. You can't really expect those coin flips to produce the same results each time since it's a random process. But the bigger the sample size, the more likely the results will reflect the ratings. A single season is, in this case, too small a sample size to smooth out all the random variables, so that's likely why you get the differences in each run through of a season.

Of course, even decades-long historical sims won't produce the exact same results each time. But then, that's the point of OOTP, it's a career sim, it's open-ended, it isn't intended to repeat the exact same results every single time. That's not the design intent.

That the game generates results which mimic quite reasonably the numbers over the length of a player's career as well as it does is quite an achievement.

[pstrickert]

Originally Posted by Le Grande Orange

That the game generates results which mimic quite reasonably the numbers over the length of a player's career as well as it does is quite an achievement.

Does it, though? I know you'll cite examples of career stats that look remarkably close to RL stats. If we checked a random sample of 100 players in the career database, how closely would their OOTP career stats match their RL numbers?

[yabanci]

on the one hand, recalc is no good because everything is too predictable. On the other, recalc is no good because everything is too unpredictable. I'm feeling sorry for poor recalc. It just can't win.

[The Wolf]

Originally Posted by yabanci

on the one hand, recalc is no good because everything is too predictable. On the other, recalc is no good because everything is too unpredictable. I'm feeling sorry for poor recalc. It just can't win.

Feel sorry for the game as well, this guy has no intention of letting it win, either, even though it's a winner and we all know it.

[rastak]

Originally Posted by pstrickert

I've seen that. Again, it's not the career totals that bother me. Everything "evens out" in the end. It's the individual season totals that bother me. More often than not, they don't look very accurate.

I'm just curious, but is running all those tests fun?

[RonCo]

OOTP is not designed to make 100% of players in a historical environment finish each season with stats within a small tolerance of their actual numbers. It's also not designed to be repeatable from sim-to-sim...or maybe a better way to state this is that the results _are_ repeatable within the framework of randomization. Rollie Fingers, for example, is going to have highly variable results from sim-to-sim due to his relatively low IP. (pure math will tell you that 70-90IP will give a very high standard deviation: a 2.6 ERA pitcher in that number of IP probably has a 15%= chance of registering a 1.5 ERA or better...and a 15% chance of registering a 3.7 ERA or better, for example, by random chance).

In addition to random chance, players will miss targets due to development (assuming it is on) or injury (assuming it is on). Note, these misses can happen both ways. Recalc eases this a bit, but development happens from day 1 of each year. With a few hundred pitchers in a leaague, it is a mathematical sureity that several will take development hits early enough in the season to fade in ways they didn't in real life. It is not predictable from run-to-run which will take the hits...this is the charm of OOTP, it is also the reason for variability between runs. It is a feature of career mode, not a bug.

There are, however, game engine issues affcting this to a degree. We have already hashed out an issue with some pitchers falling off and getting dumped despite reasonable numbers. This is due to ratings dropping past a few key levels, or you could argue it is because of weak GM AI decision code. Either way, it's a known issue--note: It is not a large issue in scope (number of pitchers across an entire league), but it causes some people discomfort. We have also discussed the issue that the Lahman DB import is not really well-tuned to the new development curves. The issues are already TTed, and i'm sure Markus will work on them when he returns.

At the end of the day, this user is bothered by the data he's seeing. I think there are two problems:

1) I think he's re-reporting the issue I described above, and
2) I think he has an improper expectation of repeatability across the scope of a league that the game is able (or designed) to provide.

[briantheiceberg]

Originally Posted by pstrickert

How would these results affect someone who plays a fictional league? If you've got a 5-star closer on your team, but he's not putting up all-star type numbers, what's the problem? Is he unhappy? Has he hit the downward slope of his development curve? Or do his ratings, as good as they are, mean little in terms of game results? Sim that season one time, and he might end up with a 4.11 ERA with 16 saves. Rewind it (if that were possible) and sim it from that same exact point again, and he might end up with a 2.11 ERA with 46 saves. That kind of variation is possible, unfortunately, if what I've seen holds up to scrutiny. But wouldn't we expect a little more consistency than that (even in a fictional league)?

dude, its PROBABILITIES. A 5 star player has higher ratings than a 1 star player. Those ratings are simply probabilities. Looking at a person's seasonal statistics in ONE year of a 15 year career and comparing it with ONE year of a 15 year career in real life - the probablility of the years being the same are low. I'd bet if you took those players career numbers they'd be similar. Additionally, if you took their numbers for a season you could find a season where they had comparable numbers. But if you look at one season of 15 and compare it to one season of 15 then odds are, they're going to be different. Its the LONG RUN that the stats even out.

don't beleive me. Flip a coin 10 times. You should have 5 heads and 5 tails. More likely than not, you'll have 7 of one and 3 of the other. Does that mean that the coin is not 'rated' properly? I mean, in that small samplesize it wasn't 50-50. Do it 150 times and you'll be much closer to 50%.

[KGAGE]

Player ability in real life does not produce realistic results either.

If we went strickly by "player ratings" at the beginning of the 2006 season, the Detroit Tigers certainly would have never gone to the World Series with the players on that team. If you would have simmed the 2006 season and come up with that result, it would have probably seemed very unrealistic and a flaw in the game.

Real life is very unpredictable and the fact the OOTP can have the same unpredictableness to it, makes it more realistic in my mind.

As for me, I strictly play fictional to even further this unpredictableness so I probably do not belong in this thread anyways. I'll leave now.

[pstrickert]

Originally Posted by RonCo

OOTP is not designed to make 100% of players in a historical environment finish each season with stats within a small tolerance of their actual numbers. It's also not designed to be repeatable from sim-to-sim...or maybe a better way to state this is that the results _are_ repeatable within the framework of randomization. Rollie Fingers, for example, is going to have highly variable results from sim-to-sim due to his relatively low IP. (pure math will tell you that 70-90IP will give a very high standard deviation: a 2.6 ERA pitcher in that number of IP probably has a 15%= chance of registering a 1.5 ERA or better...and a 15% chance of registering a 3.7 ERA or better, for example, by random chance).

In addition to random chance, players will miss targets due to development (assuming it is on) or injury (assuming it is on). Note, these misses can happen both ways. Recalc eases this a bit, but development happens from day 1 of each year. With a few hundred pitchers in a leaague, it is a mathematical sureity that several will take development hits early enough in the season to fade in ways they didn't in real life. It is not predictable from run-to-run which will take the hits...this is the charm of OOTP, it is also the reason for variability between runs. It is a feature of career mode, not a bug.

There are, however, game engine issues affcting this to a degree. We have already hashed out an issue with some pitchers falling off and getting dumped despite reasonable numbers. This is due to ratings dropping past a few key levels, or you could argue it is because of weak GM AI decision code. Either way, it's a known issue--note: It is not a large issue in scope (number of pitchers across an entire league), but it causes some people discomfort. We have also discussed the issue that the Lahman DB import is not really well-tuned to the new development curves. The issues are already TTed, and i'm sure Markus will work on them when he returns.

At the end of the day, this user is bothered by the data he's seeing. I think there are two problems:

1) I think he's re-reporting the issue I described above, and
2) I think he has an improper expectation of repeatability across the scope of a league that the game is able (or designed) to provide.

Thank you, RonCo, for contributing helpfully to the discussion. I appreciate that. I'm not trying to be argumentative. I really love OOTP, but I'm disappointed with this part of the game. I understand the variability, but that may not entirely account for the problems I'm seeing. That's what I'm hoping to zero in on.

[phenom]

Personally, I actually very much enjoy variability like that which is presented here. Anomalies can be fun and interesting.

If I wanted true historical accuracy, I'd buy the Baseball Encyclopedia.

[pstrickert]

Originally Posted by briantheiceberg

dude, its PROBABILITIES. A 5 star player has higher ratings than a 1 star player. Those ratings are simply probabilities. Looking at a person's seasonal statistics in ONE year of a 15 year career and comparing it with ONE year of a 15 year career in real life - the probablility of the years being the same are low. I'd bet if you took those players career numbers they'd be similar. Additionally, if you took their numbers for a season you could find a season where they had comparable numbers. But if you look at one season of 15 and compare it to one season of 15 then odds are, they're going to be different. Its the LONG RUN that the stats even out.

don't beleive me. Flip a coin 10 times. You should have 5 heads and 5 tails. More likely than not, you'll have 7 of one and 3 of the other. Does that mean that the coin is not 'rated' properly? I mean, in that small samplesize it wasn't 50-50. Do it 150 times and you'll be much closer to 50%.

I have no problem with probabilities. I understand that. But when I sim the same season multiple times (with player development OFF) and compare the results, I'm disappointed by what I see. Robin Yount, for example, had an MVP season in 1982. Sim after sim after sim, he comes up way short in production -- his power numbers, in particular. Is it too much to expect his stats to be somewhat close to RL? His BA can range from the .260s to the .360s for that season in OOTP. I expect some variation, but isn't that a bit much?

I'd love to see DMBers and PureSimmers convert to OOTP. But I know many of them enjoy replaying a single historical season with some degree of consistency with RL. From what I've seen, OOTP -- unless Markus can work some magic -- will not satisfy them.

[ctorg]

For one thing, with pitchers, don't look at things like ERA and W-L records. Look at K, BB, and HR ratios, because that's what OOTP is really doing. The rest of how a pitcher performs has to do with things like BABIP, which he doesn't really control.

For another, we'd really need to see some baseline numbers to compare these to. How does this stack up against an established game like DMB? If the deviations are similar, I think it's safe to say that the game does a good job. There are so many things to take into account - players face different combinations of players than in real life, and there is still in-season ratings change, even when recalc is on.

For another thing, beyond all that, OOTP is designed to be less predictable than games like DMB and SOM. One star players should have occasional brilliant seasons, and superstars have occasional off years. The game is not a season simulator.

[phenom]

I always thought of OOTP as a way to recreate history, not replicate history.

[RonCo]

Originally Posted by pstrickert

Thank you, RonCo, for contributing helpfully to the discussion. I appreciate that. I'm not trying to be argumentative. I really love OOTP, but I'm disappointed with this part of the game. I understand the variability, but that may not entirely account for the problems I'm seeing. That's what I'm hoping to zero in on.

I expected you understood the variability issue, though I'm thinking most people don't quite grasp the magnitude of the swings that random chance can provide. I expect that groups like DMB are using less than pure randomization in their approach in order to ensure precision in their final stats. I have no idea if I'm right, but if their numbers are always spot-on, it tells me that the dice are loaded.

Turning off development might help you, and changing the amount of random development might help, too. The game still has the two issues I noted, though.

[briantheiceberg]

Originally Posted by phenom

I always thought of OOTP as a way to recreate history, not replicate history.

excellent summation

[RonCo]

Originally Posted by ctorg

For one thing, with pitchers, don't look at things like ERA and W-L records. Look at K, BB, and HR ratios, because that's what OOTP is really doing. The rest of how a pitcher performs has to do with things like BABIP, which he doesn't really control.

For another, we'd really need to see some baseline numbers to compare these to. How does this stack up against an established game like DMB? If the deviations are similar, I think it's safe to say that the game does a good job. There are so many things to take into account - players face different combinations of players than in real life, and there is still in-season ratings change, even when recalc is on.

For another thing, beyond all that, OOTP is designed to be less predictable than games like DMB and SOM. One star players should have occasional brilliant seasons, and superstars have occasional off years. The game is not a season simulator.

All that is true. But let's not ignore the areas where the game does need to be improved in or process of explaining away issues.

[pstrickert]

Originally Posted by ctorg

For one thing, with pitchers, don't look at things like ERA and W-L records. Look at K, BB, and HR ratios, because that's what OOTP is really doing. The rest of how a pitcher performs has to do with things like BABIP, which he doesn't really control.

For another, we'd really need to see some baseline numbers to compare these to. How does this stack up against an established game like DMB? If the deviations are similar, I think it's safe to say that the game does a good job. There are so many things to take into account - players face different combinations of players than in real life, and there is still in-season ratings change, even when recalc is on.

For another thing, beyond all that, OOTP is designed to be less predictable than games like DMB and SOM. One star players should have occasional brilliant seasons, and superstars have occasional off years. The game is not a season simulator.

Good points. Based on my experience, DMB has some variation, too, especially pre-1980 seasons, I think. Most of the time, the results compare better to RL than OOTP. Part of the challenge of taking a historic team is to see if your coaching decisions make a difference in the season outcome. Can you win the pennant with a team that finished 6 or 10 games out of first IRL? In DMB, that's a fair challenge. In OOTP, based on what I've seen, it's not the same challenge. Even with player development (and other variables) off, it's hard to know if your decisions as a manager really affected the outcome. It's not uncommon to sim a historical season and see a last place team IRL win the pennant in OOTP.

Here's another thought: Some people like to set up a league in OOTP with the greatest teams in baseball history. Were the 1927 Yankees the best team ever? Based on what we know about OOTP, would we really be able to trust the results? I'm not so sure.