Historical Accuracy Overrated?

[phenom]

Originally Posted by pstrickert

Good points. Based on my experience, DMB has some variation, too, especially pre-1980 seasons, I think. Most of the time, the results compare better to RL than OOTP. Part of the challenge of taking a historic team is to see if your coaching decisions make a difference in the season outcome. Can you win the pennant with a team that finished 6 or 10 games out of first IRL? In DMB, that's a fair challenge. In OOTP, based on what I've seen, it's not the same challenge. Even with player development (and other variables) off, it's hard to know if your decisions as a manager really affected the outcome. It's not uncommon to sim a historical season and see a last place team IRL win the pennant in OOTP.

Here's another thought: Some people like to set up a league in OOTP with the greatest teams in baseball history. Were the 1927 Yankees the best team ever? Based on what we know about OOTP, would we really be able to trust the results? I'm not so sure.

If I'm not mistaken, DMB season disks cost nearly as much (if not more) than OOTP itself does. Is that correct? That would lead me to believe that much more time is focused on the accuracy therein with each season, with anything from gameplay modifiers to... who knows what?

I think that, while OOTP does have year-specific modifiers and the like, the scope of the game was not to meticulously detail those factors in minutiae, whereas DMB does so because, frankly, that IS the game.

[RchW]

Originally Posted by KGAGE

Player ability in real life does not produce realistic results either.

If we went strickly by "player ratings" at the beginning of the 2006 season, the Detroit Tigers certainly would have never gone to the World Series with the players on that team. If you would have simmed the 2006 season and come up with that result, it would have probably seemed very unrealistic and a flaw in the game.

Real life is very unpredictable and the fact the OOTP can have the same unpredictableness to it, makes it more realistic in my mind.

As for me, I strictly play fictional to even further this unpredictableness so I probably do not belong in this thread anyways. I'll leave now.

Funny you should reference Detroit. A 6.5 league I was in decided to re-start (not my idea) for the 2006 season. I took Detroit out of curiosity. They sucked. Started the season 4-24. Almost all of the players who were producers IRL were 1 to 1.5 star slugs. I-Rod was batting .176 and had allowed about 80% SB against. The closer, forget his name, had an ERA of 10+.

I can't get past expectations when I play with real players. Thats why I prefer fictional.

[pstrickert]

Originally Posted by RonCo

I expected you understood the variability issue, though I'm thinking most people don't quite grasp the magnitude of the swings that random chance can provide. I expect that groups like DMB are using less than pure randomization in their approach in order to ensure precision in their final stats. I have no idea if I'm right, but if their numbers are always spot-on, it tells me that the dice are loaded.

Turning off development might help you, and changing the amount of random development might help, too. The game still has the two issues I noted, though.

Here's a link to a detailed description of DMB:

http://www.diamond-mind.com/products/inside.htm

OOTP takes much of the same statistical research into account. Maybe you're right about DMB using "less than pure randomization." Maybe I'll inquire about that on the DMB boards.

[BaseballMan]

Originally Posted by phenom

Personally, I actually very much enjoy variability like that which is presented here. Anomalies can be fun and interesting.

If I wanted true historical accuracy, I'd buy the Baseball Encyclopedia.

and even the Baseball Encyclopedia isnt always accurate. remember the changes made to Ty Cobb's record for hits? And then theres other records being changed and changed back again

[BigCity]

I've been involved in discussions similar to this for the entire 6 years I've been associated with OOTP. In the end, it comes down to this...

If any season out of MLB history was replayed by the same players immediately after they finished the first one, the results would not be the same, and in some cases would be significantly different.

A major league season, month, week, even game, is unique, and a one time occurance.

DMB is designed to "mirror" the actual results as close as possible without resorting to simply subtracting each stat as it occurs.

OOTP was never designed to do anything like that, and even in its current form, is only measurably better than it used to be.

Asking OOTP to do anything more is to ask it to replicate history. How far do you carry that? Do you want your replay to have the exact number of strikes, balls, singles, HRs, etc, as in real life? Then you need a simulation that simply subtracts each stat from the real-life totals as they occur in your simulation.

Anything else is an approximation, which is what OOTP does very well.

[RchW]

Originally Posted by RonCo

I expected you understood the variability issue, though I'm thinking most people don't quite grasp the magnitude of the swings that random chance can provide. I expect that groups like DMB are using less than pure randomization in their approach in order to ensure precision in their final stats. I have no idea if I'm right, but if their numbers are always spot-on, it tells me that the dice are loaded.

Turning off development might help you, and changing the amount of random development might help, too. The game still has the two issues I noted, though.

I think he did in the first post. If not he should.

I expect you're right about the less random state of DMB. That's why I can't get excited by "replay" sims. Is the die cast?

Somebody suggested the attraction of DMB was to see if you could "manage" a contender to the pennant. What I find strange is that books like Baseball Between the Numbers clearly state that managers have little effect and if they do it is usually negative. So if one were to win with an also ran in DMB would that not blow its stated statistical purity (superiority) out of the water?

[DaddyO]

Originally Posted by pstrickert

Here's a link to a detailed description of DMB:

http://www.diamond-mind.com/products/inside.htm

OOTP takes much of the same statistical research into account. Maybe you're right about DMB using "less than pure randomization." Maybe I'll inquire about that on the DMB boards.

This is funny. On the DMB boards we get the same complaints, and the answers are the same. It is not that DMB loads the dice. They specifically say they do NOT do so. Some OTHER companies have admitted in the past doing so, and DMB rejects this approach.

Invariably the answer is small sample size. There are simply not enough ABs and IPs in a single season to replicate season stats with a consistently high degree of accuracy for any given player. Even with real life transactions and lineups turned on, so your batters face exactly the same starters as they did in real life, there can be for some players significant variations.

But when you run 1000 samples, the accuracy of DMB is outstanding.

I completely agree that you have to take into account the purpose of each game. OOTP seems to have taken a large step forward this year in historical accuracy, but it is not designed primarily to replay single seasons accurately.

If DMB doesn't satisfy your season replay hopes, nothing will at this point. They go to a great deal of trouble to ensure that each season disk will replicate the season as accurately as statistically possible. Beyond the statistically possible, they cannot go. Neither can anyone else.

[pstrickert]

Here's another question: When I imported the 1982 season multiple times from the Lahman DB, sometimes I would see Mike Caldwell rated 1-star, sometimes 1-1/2 stars. The same thing would happen with other players, too. If the same stats are being imported, why the difference? (I'd have to check, but I assume Caldwell's Stuff, Movement, and Control ratings differed a bit, too. Thus, the difference in "star" rating.)

[satchel73]

I'd agree that ordinary, predictable, statistical variation accounts for a lot of the sim-to-sim "inconsistency" you found. There might be something else going on, though. I was thinking it could be the DIPS-based engine in the case of the pitchers, particularly Moose Haas vs. Pete Vuckovich. It seems like, in your sims, Haas usually did better than IRL, while Vuckovich did worse.

Here are their real-life stats:

Moose Haas 1982
IP: 193.3
H-HR: 217 / 10.10 per 9 innings
HR: 15 / .70
BB: 39 / 1.82
K: 104 / 4.84
ERA: 4.47

Pete Vuckovich 1982
IP: 223.7
H-HR: 220/ 8.85
HR: 14 / .56
BB: 102 / 4.10
K: 105/ 4.23
ERA: 3.34

In the defense-independent categories, Vuckovich narrowly edged Haas in home runs allowed (by .14 a game), while Haas struck out .61 more batters per game and had *much* better control, walking batters at less than half Vuckovich's rate (amounting to a rather large difference of 2.41 walks a game!).

The other big difference between them is on hits on balls in play--precisely the stat that's heavily influenced by fielding. According to DIPS (to simplify it greatly), when two pitchers on the same team have a big difference on hits allowed, it's probably due mostly to chance. So, agree or disagree with the DIPS approach, OOTP might be telling you that Haas was really a better pitcher than Vuckovich that year, and that Vuckovich's Cy Young was something of a mirage.

[rastak]

Originally Posted by pstrickert

Here's another question: When I imported the 1982 season multiple times from the Lahman DB, sometimes I would see Mike Caldwell rated 1-star, sometimes 1-1/2 stars. The same thing would happen with other players, too. If the same stats are being imported, why the difference? (I'd have to check, but I assume Caldwell's Stuff, Movement, and Control ratings differed a bit, too. Thus, the difference in "star" rating.)

Isn't the star view in this game really your scouts interpretation?

[pstrickert]

Originally Posted by rastak

Isn't the star view in this game really your scouts interpretation?

Scouts OFF

[BaseballMan]

For me, OOTP is accurate as it can be. Do you expect the 1927 Yankees to always win a greatest team tourney? If so why run the tourney? I am not sure how much more accurate Markus can make the game considering that it wasnt designed to be a historical accurate type baseball game. I think more or less he probably did it as a favor to us historical players. I mean at least now we dont have to use a Babe Ruth and a George Ruth. Now Sandy Koufax turns into well Koufax and not a journeyman reliever.

Sure i want a player like Ruth to hit close to 50-60 hrs but if i sim the 27 season and he has different results it doesnt bother me unless he ends up with only 20 homers 75% of the time. I just want the players in the ballpark as far as historical stats. From there anything can happen. Yeah i agree with the if i wanted perfect accurate stats i would go to baseball reference or read the baseball encylopedia. The only way to have perfect accurate historical stats is to have the exact schedule, ballparks, weather, lineups, trades injuries etc etc. You know theres no way that can happen so i just accept it if Ty Cobb bats .358 in one career and .331 in another. Just as long as he doesnt bat .217 all the time.

[RchW]

Originally Posted by pstrickert

Here's another question: When I imported the 1982 season multiple times from the Lahman DB, sometimes I would see Mike Caldwell rated 1-star, sometimes 1-1/2 stars. The same thing would happen with other players, too. If the same stats are being imported, why the difference? (I'd have to check, but I assume Caldwell's Stuff, Movement, and Control ratings differed a bit, too. Thus, the difference in "star" rating.)

Star ratings compare players to others at the same position. Seems to me the variation in the ratings calculated for each player may be sufficient to cause a 1/2 star change from time to time. Perhaps a swing man type of pitcher was an MR in one import and an SP in another.

You should try it with 20-80 instead of stars to better quantify the difference.

[pstrickert]

Originally Posted by satchel73

I'd agree that ordinary, predictable, statistical variation accounts for a lot of the sim-to-sim "inconsistency" you found. There might be something else going on, though. I was thinking it could be the DIPS-based engine in the case of the pitchers, particularly Moose Haas vs. Pete Vuckovich. It seems like, in your sims, Haas usually did better than IRL, while Vuckovich did worse.

Here are their real-life stats:

Moose Haas 1982
IP: 193.3
H-HR: 217 / 10.10 per 9 innings
HR: 15 / .70
BB: 39 / 1.82
K: 104 / 4.84
ERA: 4.47

Pete Vuckovich 1982
IP: 223.7
H-HR: 220/ 8.85
HR: 14 / .56
BB: 102 / 4.10
K: 105/ 4.23
ERA: 3.34

In the defense-independent categories, Vuckovich narrowly edged Haas in home runs allowed (by .14 a game), while Haas struck out .61 more batters per game and had *much* better control, walking batters at less than half Vuckovich's rate (amounting to a rather large difference of 2.41 walks a game!).

The other big difference between them is on hits on balls in play--precisely the stat that's heavily influenced by fielding. According to DIPS (to simplify it greatly), when two pitchers on the same team have a big difference on hits allowed, it's probably due mostly to chance. So, agree or disagree with the DIPS approach, OOTP might be telling you that Haas was really a better pitcher than Vuckovich that year, and that Vuckovich's Cy Young was something of a mirage.

This is exactly what I was hoping to generate: a productive discussion about the issue.

That season, Vuckovich did pitch out of a lot of trouble. Drove his manager crazy, I'm sure. Was there really, though, a 4-1/2 star to 1-star difference between Haas and Vuckovich?

[RonCo]

Originally Posted by DaddyO

This is funny. On the DMB boards we get the same complaints, and the answers are the same. It is not that DMB loads the dice. They specifically say they do NOT do so. Some OTHER companies have admitted in the past doing so, and DMB rejects this approach.

Ok. I only made the comment because of my impression from others that DMB always had essentially perfect season-to-season replication of numbers. If that impression is wrong, then I retract my caveated comment.

[RonCo]

Originally Posted by pstrickert

Here's another question: When I imported the 1982 season multiple times from the Lahman DB, sometimes I would see Mike Caldwell rated 1-star, sometimes 1-1/2 stars. The same thing would happen with other players, too. If the same stats are being imported, why the difference? (I'd have to check, but I assume Caldwell's Stuff, Movement, and Control ratings differed a bit, too. Thus, the difference in "star" rating.)

I admit I haven't paid much attentino to small differences in star ratings. I like stars, but basically just use them as visual milemarkers for players on the screen. I'll try it a few times. Got a few other specific players?

[RonCo]

Originally Posted by pstrickert

This is exactly what I was hoping to generate: a productive discussion about the issue.

That season, Vuckovich did pitch out of a lot of trouble. Drove his manager crazy, I'm sure. Was there really, though, a 4-1/2 star to 1-star difference between Haas and Vuckovich?

Assignment of stars has always been odd in OOTP. My very first guess is to wonder if there was a large endurance difference in their imported ratings.

[pstrickert]

Originally Posted by BaseballMan

i just accept it if Ty Cobb bats .358 in one career and .331 in another. Just as long as he doesnt bat .217 all the time.

I could accept that, too. However, I'd be worried if Cobb's BA ranged from .260 to .360. I'd also be worried if Ruth's SLG% consistently fell short of his RL number (in some cases FAR below it). Unfortunately, that's what I've seen with R. Yount, for instance. Part of the problem could be Lahman import-related. We'll have to hope Markus looks into that.

[pstrickert]

Originally Posted by RonCo

I admit I haven't paid much attentino to small differences in star ratings. I like stars, but basically just use them as visual milemarkers for players on the screen. I'll try it a few times. Got a few other specific players?

Import the 1974 season and look at the Milwaukee roster. I looked at this before and saw a difference from one import to the next of 1 to 2 stars for a few different players.

[pstrickert]

Originally Posted by phenom

If I'm not mistaken, DMB season disks cost nearly as much (if not more) than OOTP itself does. Is that correct? That would lead me to believe that much more time is focused on the accuracy therein with each season, with anything from gameplay modifiers to... who knows what?

I think that, while OOTP does have year-specific modifiers and the like, the scope of the game was not to meticulously detail those factors in minutiae, whereas DMB does so because, frankly, that IS the game.

DMB, as fine a product as it is, is dying a slow death IMO. Look at their message boards -- they're beginning to eat their own! Puresim also has fallen on hard times. Now's the time for OOTP to strike!!!