Historical replay accuracy

[struggles_mightily]

I do think, as Le Grande Orange has intimated, that there lies behind this issue a deeper philosophical question of how one consciously defines "plausibility", and also how such conscious definitions are impacted by sub-conscious expectations.

There's no reason, a priori, why "plausible" should be defined as "resembling real-life events." Many events have occurred in baseball history that appeared unlikely before unfolding ; many situations seen as plausible or even likely have failed to come to pass (cf. recently -- the 'can't miss' Tigers of last season). Indeed, how many events at the macro level would have seemed 'implausible' before the season began? Washington Nationals, 2009 World Champions? Very unlikely, sure. But "not plausible"? I'd hesitate before using those words. I'm not sure that any possible permutation of World Series representatives could be called "implausible". (Other than, like, Red Sox vs. Devil Rays and so on.)

At the micro level, there are things that deserve to be called "implausible." I'm not sure that anyone would, or should, buy Alex Cintron as the 2009 N.L. Batting Champion. But once one starts considering the fortunes of teams rather than players and general outcomes ('wins') rather than more specific values ('batting average'), the scope of plausability becomes much more extensive.

Earlier in this thread, someone suggested that the minimal demand of historical players as regards talent evaluation should be that players import "correctly." That is, Ken Griffey Jr. should import as a big slugger who plays some D.; Craig Biggio should import as a valuable singles hitter with no pop who plays a premium position etc. But is it really valid to locate the moment of entry into the majors (since that's the point at which most simmers import their historical players) as the time when player talent levels are "set"? I'm not sure that it is, exactly.

I absolutely agree that the fun of accurate replay is marred by odd A.I. decisions. Unless a team has an atrocious (general) manager, it should not be fielding players without defensive ratings at positions and watching them put together a 1:1 games:errors ratio, nor should it be trading Garvey for Montanez straight-up (probably -- that's a little more debatable). But those are A.I. issues that affect all areas of OOTP, and aren't strictly to do with historical play. (Some aspects of defensive mishandling are, sure -- the Lahman data and the lack of a good trans-historical defensive metric is the root cause of those.)

It has been said that OOTP is "bad" for historical players. It is indeed bad for certain types of historical play -- strict replayers are unlikely to feel that their needs are being met and (as Styx has said) might well be better served by looking elsewhere. I mostly play historically, however, and I find OOTP to be fantastic. I like to hit the pinball and then be unsure of where it might travel -- to, as far as I can, shake-off fallacies of pre-determined outcomes. I think that the crimes of other games -- no Negro Leagues; no Japanese leagues; no minor leagues full-stop; no A.I. trading -- subtract much more from the "historical" experience than do OOTP's issues.

That's not intended to suggest that my way of playing is "better" or deserves primary consideration. But I think it is worth pointing out that "historical play" covers many different sorts of experiences and desires.

[dashby]

Again, probably been said many times before but, in my book, plausible is playing a season with MLB rosters and getting a statistical/standings outcome that is very similar to what happened in real life. I don't believe anyone expects every team to end up in the same positions. I don't believe anyone expects every pitcher and batter to have identical stats. Don't believe anyone is suggesting "identical".

Obviously anyone can look it up themselves but this is "plausible" - having an appearance of truth or reason.

We should see most of the players having stats within a 5-10% statistical range of what they really did. There might be spikes that vary tremendously from real life. But, if these spikes are the trend and the 5-10% range is the exception there are problems with that game's statistical engine. It doesn't process statistics "plausibly" over 162 games.

For example, I played the 2006 season in OOTP 9 10 times. Cleveland finished 1st in the Central 1/2 of those times (they were in reality 4th and 18 games out). Minnesota, who won the division, finished in 1st only once. Player stats were just all over the place...except close to where the real season indicated they should have been. Many more similar anomalies.

Just takes all the fun out of playing seasons that you know what the "True" outcome was.

[struggles_mightily]

Originally Posted by dashby

Again, probably been said many times before but, in my book, plausible is playing a season with MLB rosters and getting a statistical/standings outcome that is very similar to what happened in real life.

The Major League Baseball season is a really bad test of the abilities of baseball teams, though, isn't it? It plays itself out only once. It doesn't control for extraneous variables such as injury, unexpected poor performance, or similar. The Major League Baseball season is not really intended to divine the "true talent" levels of the participating teams, and cannot do this with tremendous accuracy.

I don't believe anyone expects every team to end up in the same positions. I don't believe anyone expects every pitcher and batter to have identical stats. Don't believe anyone is suggesting "identical".

Neither did I suggest that posters had such expectations. The word "identical" appears nowhere in my post. I'm not interested in playing the "well, what percentage deviation from real life is acceptable?" game. I'm trying generally to question the usage of "real life" as a reference point with which to measure the "plausibility" of OOTP's simulations.

We should see most of the players having stats within a 5-10% statistical range of what they really did. There might be spikes that vary tremendously from real life. But, if these spikes are the trend and the 5-10% range is the exception there are problems with that game's statistical engine.

Again, I'm not sure that real life should be used as a basis. I wonder what results would be gained by comparing OOTP outputs with, say, PECOTA projections? I'm sure that OOTP probably does have some problems -- its A.I. is off in a number of areas.

It's also important to divide roster-set problems (mis-rated players and so on) from OOTP problems, though, isn't it? (Admittedly, the lines have been blurred in terms of responsibility since OOTP started endorsing roster sets. But these issues still have to be seen as separate, I'd have thought.)

For example, I played the 2006 season in OOTP 9 10 times. Cleveland finished 1st in the Central 1/2 of those times (they were in reality 4th and 18 games out). Minnesota, who won the division, finished in 1st only once.

Interesting that you should pick that season and those teams. Did you know that Cleveland under-performed their Pythag that year by 12 games? That's incredible. They 'should' have finished 90-72. Minnesota slightly over-performed their Pythag (three games) and 'should' have finished 92-70.

The Central had four really good teams that year -- Cleveland, the Twins, Detroit and Chicago all finished with Pythagorean Records which included 90 or more wins. Any division winner in that context would be very, very "plausible" -- though Kansas City probably shouldn't be finishing first any more than, say, 5% of the time.

I certainly don't think it implausible that an Indians team loaded with guys like Travis Hafner, Victor Martinez, Grady Sizemore and CC Sabathia could win an, albeit strong, division 10-20% of the time. That projection might even under-estimate Cleveland slightly, though they did get a lot of important contributions from fringy players that year (Ben Broussard, Eduardo Perez, Jeremy Sowers etc).

Were you also upset when the Cardinals didn't win it all or even make the playoffs? That was a mediocre team that won a World Series. I'd hate it if OOTP faithfully recreated that outcome in a majority of repetitions.

Player stats were just all over the place...except close to where the real season indicated they should have been. Many more similar anomalies.

In my historical play, I don't usually see player stats that I would call "all over the place." Could you give some examples?

Just takes all the fun out of playing seasons that you know what the "True" outcome was.

I think that people place far too much emphasis on "real" events. OOTP's task should not be to approximate closely the results of the real baseball season -- a bad, un-controlled, un-repeated "test" -- but to produce results which are "plausible" with reference to the talent levels of teams and players.

[Theonlystd]

I dont have a problem with how historical leagues play out..

Not really interested in near mirror mages of what happened in reality ..

[knockahoma]

Hey Struggles,

Just to clarify, I wouldn't want the game to import a player at his peak performance. Take Bonds, for example. I'd like to see him imported as he was in those early years with the "potential" set for what he "becomes".

I'm not as advanced as some on how the engine works, but I thought that was its intent with potential, and that potential is still based on "later seasons" as provided by Lahman.

And the Dodgers traded for lefty-Montanez while keeping Garvey. The Dodgers traded away one of their starting pitchers to obtain Montanez (.280-ish in June), apparently to platoon with Garvey (.320)?

You know, it would have been easier for me to invent my own little story if the two teams had just traded 1B. I'd borrow the story-line of the Padres/Cardinals swapping super shortstops Ozzie Smith and Garry Templeton. But to give up a starting pitcher for a 1B when your own 1B is an all-star hitting .320 in mid-summer? Apparently, LA accepted the trade because Monty was a left-handed hitter.

Hard to think up a story for that one ... unless Garvey was on his way out! But, no... no, Garvey stayed in LA. Okay, time to re-draft the story...

[struggles_mightily]

Originally Posted by knockahoma

Just to clarify, I wouldn't want the game to import a player at his peak performance. Take Bonds, for example. I'd like to see him imported as he was in those early years with the "potential" set for what he "becomes".

Sure, I get what you're saying. But is it accurate to want his "potential" set so that he will probably become a 750 HR uber-monster? Aren't there a whole bunch of potential career paths for Bonds at the moment of his entry into the Majors at the age of 21?

(Note: this brackets out the question of whether a poorly-developing Bonds reflects a flaw with Lahman imports even if it's desirable / "plausible" for him to scuffle in some replays. It also brackets out questions about steroids -- which, like... I don't even think I want to throw into the equation.)

I'm not as advanced as some on how the engine works, but I thought that was its intent with potential, and that potential is still based on "later seasons" as provided by Lahman.

I'm not an expert on this, either.

Is Bonds a good example to use in any case? Is he one of those guys known to struggle in historical replays? That's a genuine question -- I've never played that period myself. I know Clemente is one guy that a lot of people complain about.

And the Dodgers traded for lefty-Montanez while keeping Garvey. The Dodgers traded away one of their starting pitchers to obtain Montanez (.280-ish in June), apparently to platoon with Garvey (.320)?

Haha -- that's even worse, then. OOTP's trading AI has many flaws, and I'm certainly not going to argue that point.

Did the GM involved have really bad stats? Were GMs even on? Just wondering -- I'd like to know how much effect front office personnel have on player decisions.

Thanks for the clarifications in any case. I think this is / could be a really interesting thread.

[dashby]

Struggles, no offense, but I have a difficult time grasping your logic. It's probably my fault.

For example,

The Major League Baseball season is a really bad test of the abilities of baseball teams, though, isn't it?

I just can't really begin to assimilate the relevancy of this statement. It's like trying to eat jello with chopsticks. Bet you're already thinking up a way to tear that one down and dissect it.

They 'should' have finished 90-72. Minnesota slightly over-performed their Pythag (three games) and 'should' have finished 92-70.

If you darted out and took a look at Bill James Relative Power Index for 2006 as basis for this (I believe RPI is based loosely on Pythagorean theorem)...your statement was indeed plausible...but not accurate. In fact this index noted Minnesota EXPW-L at 94-68. Probably just a type-o on your part.

Just to save a whole lot of bickering I'm just going to say you're correct and I'm absolutely incorrect with my concept of what is plausible and what is not. Let's just say you're logic is deeply profound and I'm sim-tistically challenged.

[struggles_mightily]

I just can't really begin to assimilate the relevancy of this statement. It's like trying to eat jello with chopsticks. Bet you're already thinking up a way to tear that one down and dissect it.

"Tear down and dissect" what? It's not like you've made a point or anything.

In saying:

"The Major League Baseball season is a really bad test of the abilities of baseball teams, though, isn't it?"

I meant:

"The Major League Baseball season is a really bad test of the abilities of baseball teams, though, isn't it?"

But I'd be happy to draw the lines a bit more clearly.

i) Many people do not base their ideas regarding what consitutes a "plausible" outcome upon considerations of "true talent." Instead, they measure OOTP's "accuracy" by comparing its outputs to the "actual result" (i.e. the result of the relevant Major League Baseball season or seasons).

ii) However, expecting OOTP's outputs closely to resemble the "actual result" is misplaced and (to me, anyway) odd. This is because a single Major League Baseball season is not a good means of figuring out the "true talent" of teams and, to a lesser extent, players ("to a lesser extent" just because there are fewer variables in the second case).

iii) I think that what we should expect from OOTP is that it produces a result which is plausible based on the true talent of Major League Baseball players and (thus) teams. "True talent" is obviously a very slippery fish to grasp. But, by that phrase, I basically mean "the abilities of baseball players based on their record to date and discounting extraneous irrelevancies such as park factor, injury, aging and luck on balls in play." There may be other "irrelevances" that I'm missing, and some may dispute the degree to which BABIP is a result of luck. And the precise calculations and adjustments to be made will be highly contentious. But I think that what I mean by "true talent" (at least as differentiated from "actual result") is clear.

To expand on ii):

Think about it: if you wanted to figure out how good the 30 MLB teams were, really and truly, what would you do?

-- You'd play the 162 game seasons many times. At least a hundred. You yourself ran ten test seasons, so obviously you understand why this is.

-- You would turn off injuries. They're not really relevant to determining how good a team is. (Or, what you should really do, is distribute injuries to a pre-decided level based on a player's historical injury-proneness.)

-- You would ensure that all players had a league average BABIP, or a BABIP in line with their own career norms.

There are probably other adjustments.

The MLB season does not do any of this. It features wildly unpredictable injuries, schedules a paltry 162 games per club and leaves the fate of balls in play up to chance (to a degree). The MLB season is a really bad, unscientific way of determining which teams are the "best." We shouldn't expect OOTP to replicate its results.

Expecting that would be kind of like simming 100 seasons in Baseball Mogul, selecting one of those 100 seasons at random, and then demanding that a single replay of the same season on OOTP closely resemble that one BM replay.

If you darted out and took a look at Bill James Relative Power Index for 2006 as basis for this (I believe RPI is based loosely on Pythagorean theorem)...your statement was indeed plausible...but not accurate. In fact this index noted Minnesota EXPW-L at 94-68. Probably just a type-o on your part.

I was looking at Baseball Reference, which gives the value I quouted. I'm sure there are more sophisticated ways of calculating ExWL. My general point should be clear.

Just to save a whole lot of bickering I'm just going to say you're correct and I'm absolutely incorrect with my concept of what is plausible and what is not. Let's just say you're logic is deeply profound and I'm sim-tistically challenged.

Glad to have established some common ground. (Winky face).

[dashby]

One final statement. I truly enjoy OOTP. Been playing it for years. Don't even recall what version I started with. I'm not one of those folks that nostalgically remember their first OOTP version like their first blowjob. God bless those diehards though.

One of the reason's I choose it over some other VERY good MLB sims is the price. With OOTP you get a pretty good game for around $30. The Fictional play has wasted more precious hours of my life than I like to even think about.

BUT, and this is important Struggles, if games like Strat-o-Matic and Dave Koch do get the numbers real close (closer than plausible)....why can't OOTP do the same thing when fed rosters with real-life players and stats?

I'm not promoting these games I'm merely using them as a pretty definitive benchmark as to how a properly engineered software sim can nail real-life stats. Maybe that's why they charge so much extra for their season discs. 'cause they do produce realistic numbers and that may take a little extra savvy and work...I don't know.

And, this isn't to suggest that OOTP sucks...it just doesn't appear to be (imo) geared towards giving an adequate depiction of non-fictional play. At least when compared to a couple other games that do.

And, my whole point is....why can't OOTP do that?

[struggles_mightily]

BUT, and this is important Struggles, if games like Strat-o-Matic and Dave Koch do get the numbers real close (closer than plausible)....why can't OOTP do the same thing when fed rosters with real-life players and stats

I know virtually nothing about those other games, so I can't really give an answer. I imagine that the broad issue is one of purpose. OOTP doesn't market or present itself as a Strat or DMB-style "relive the past" game.

I know that games like Strat have options such as "limit player to [x] ABs / IP". They also have "season disks" (don't they? most of them...) and therefore seem to be aimed at the re-production of really small periods to a great degree of accuracy.

Couldn't I easily ask "why can't Strat give me the malleability and unpredictability of OOTP"? I could, but people would tell me (quite rightly) to piss off. I don't think that the close-simmers should (be told to) piss off, but they will probably have to come to realise that OOTP sacrifices the many elements that such players love about Strat etc. in order to open up many possibilities to other sorts of players.

I think.

It's a good question, and not one I can do much justice to. Maybe an avid player of Strat, DMB and OOTP can give some more precise reasoning.

e: Dashby, or any other Strat player, what is the content of the game's "season disks"? I'm guessing it's something slightly more sophisticated than the Lahman. I'd imagine that the content of those, if that's where the game draws its inner statistical workings from, would be the key to answering the question of why Strat achieves things that OOTP cannot.

[gglennwill]

Originally Posted by struggles_mightily

There's no reason, a priori, why "plausible" should be defined as "resembling real-life events." Many events have occurred in baseball history that appeared unlikely before unfolding ; many situations seen as plausible or even likely have failed to come to pass (cf. recently -- the 'can't miss' Tigers of last season). Indeed, how many events at the macro level would have seemed 'implausible' before the season began? Washington Nationals, 2009 World Champions? Very unlikely, sure. But "not plausible"? I'd hesitate before using those words.

The Washington Nationals of 2009 winning the World Series is implausible, unlikely, far-fetched, inconceivable, doubtful, unthinkable, dubious, and fantastic. Furthermore, it ain't gonna happen. (thank you thesaurus.com).

[struggles_mightily]

Originally Posted by gglennwill

The Washington Nationals of 2009 winning the World Series is implausible, unlikely, far-fetched, inconceivable, doubtful, unthinkable, dubious, and fantastic. Furthermore, it ain't gonna happen. (thank you thesaurus.com).

It's very, very unlikely. But how many times do you think it would happen if the season were played a thousand times? Zero? How about if it were played a million times? Maybe none, but it doesn't seem 'obvious' to me.

(Also -- I don't think you should thank thesaurus.com too vigorously, since something can be "unlikely" without being "implausible." Nationals as 2009 champs is the former, but I'm not sure it's the latter.)

But the Nationals comment was just me picking the most extreme example to illustrate a point. Even if I concede that their becoming champions is implausible, I'd only have to weaken my proposition to "in any year, like 20 teams could plausibly win it all."

In 2006, a Cardinals team that was -- by Pythag -- about .500 (in the NL) won it all. How many teams will be worse than that this year (i.e. will be < .500 in the NL or <, say, .480 in the NL)? Maybe 10-12?

[Theonlystd]

Originally Posted by dashby

One final statement. I truly enjoy OOTP. Been playing it for years. Don't even recall what version I started with. I'm not one of those folks that nostalgically remember their first OOTP version like their first blowjob. God bless those diehards though.

One of the reason's I choose it over some other VERY good MLB sims is the price. With OOTP you get a pretty good game for around $30. The Fictional play has wasted more precious hours of my life than I like to even think about.

BUT, and this is important Struggles, if games like Strat-o-Matic and Dave Koch do get the numbers real close (closer than plausible)....why can't OOTP do the same thing when fed rosters with real-life players and stats?

I'm not promoting these games I'm merely using them as a pretty definitive benchmark as to how a properly engineered software sim can nail real-life stats. Maybe that's why they charge so much extra for their season discs. 'cause they do produce realistic numbers and that may take a little extra savvy and work...I don't know.

And, this isn't to suggest that OOTP sucks...it just doesn't appear to be (imo) geared towards giving an adequate depiction of non-fictional play. At least when compared to a couple other games that do.

And, my whole point is....why can't OOTP do that?

You kind of answerred your own question

If you have to buy season disks.

How do you assure rather low margins of error in stats in simming a season of baseball. With all the variables,luck and such able to have a pretty big impact considering they play 162 games?

Im not sure myself.. But im guessing they try to take out as much luck/randomness as possible in the sim..

[StyxNCa]

Once again, OOTP is not for real historical play, period, and should quit using that term when describing the game. It is totally for fictional play. Even when using real players and teams, if the results are fictional, it can't be called historical.

Mentioning the Washington example below reminded me that 2 times the 1969 Expos went to the World Series, winning once. Not plausible. Definitely not. When has any expansion team ever gone to the playoffs their first year? Anyone who sees something like that or the Washington example as plausible has one heck of a suspension of disbelief and really don't understand what historical players are saying.

Anyone who has been giving explanations as to why this game isn't exactly made to be historical is actually making the case for historical players and their issues and for that I thank you. It's saved me a lot of typing.

[bababui]

Originally Posted by StyxNCa

Once again, OOTP is not for real historical play, period, and should quit using that term when describing the game. It is totally for fictional play. Even when using real players and teams, if the results are fictional, it can't be called historical.

LoL

Any game that doesnt regurgitate the exact events of any past season is fictional.

[Base Stealer]

I cannot be considered a person that knows a lot about baseball. I follow the game only from a couple of years and OOTP helped me increase my interest in this beautiful sport. I've read all this thread and i think it all comes down to everyone's taste (as everthing in life). Personally i love the ability of being able to play some baseball history with the game. The results are not accurate, even a person that know so few things about baseball history can recognize that in the game...and i'm not referring to players that in game get a serious injury so that their carrer end sooner that expected. One of the great problems in the past was, for me, the fielding ratings. I love the defensive part of every sport game. I'm a "defense wins championship" guy. And i was frustrated by how great players that won gold gloves were horrible in the game. But then version 9 made a step forward. Now i hope the new pitching system will fix a few more problems in this part of the game.
All i wanted to say is that the "historical" part of the game is what made me want to know more about baseball history, looking on the internet for infos about Cy Young, about Ruth playing pitcher and then become what he has, about Phil Niekro, about knuckleballs....i've seen this part of the game getting better and better with years. And this will continue in this and future version.
Take this post for what it's worth in a sea of posts made by people that lived of baseball for years and that would like to live the feeling of those years again. I love the feeling i get playing the game....and that's a good thing

[struggles_mightily]

Originally Posted by StyxNCa

Once again, OOTP is not for real historical play, period, and should quit using that term when describing the game. It is totally for fictional play. Even when using real players and teams, if the results are fictional, it can't be called historical.

This, to me, seems imperious. The meaning of the term "historical play" is up for debate and is contestable. I don't think it has one 'obvious' definition that OOTP can either live up to or fail to realise.

Mentioning the Washington example below reminded me that 2 times the 1969 Expos went to the World Series, winning once. Not plausible. Definitely not. When has any expansion team ever gone to the playoffs their first year? Anyone who sees something like that or the Washington example as plausible has one heck of a suspension of disbelief and really don't understand what historical players are saying.

I wish I'd never mentioned the 2009 Nationals. But isn't it very difficult to judge what sort of job OOTP is doing whilst confining oneself to the question of only whether or not [x] team as World Champion is "plausible"? Doesn't the plausibility depend partially upon how these teams are achieving these feats?

[Aytumious]

I just want to say struggles mightily, you've made some excellent posts in this thread. I was going to comment on some things, but you already made the case.

[RonCo]

Originally Posted by struggles_mightily

In saying:

"The Major League Baseball season is a really bad test of the abilities of baseball teams, though, isn't it?"

I meant:

"The Major League Baseball season is a really bad test of the abilities of baseball teams, though, isn't it?"

Nice discussion. Most people, and I mean this nicely, have no real idea how random "random' is.

[Cryomaniac]

Originally Posted by ronco

most people, and i mean this nicely, have no real idea how random "random' is.

+1