Historical replay accuracy

[EnderCN]

DMB and Strat do not end up with statistics anywhere close to the real major league numbers btw. The last few times I ran full DMB seasons for the Brewers I ended up with Braun with as few as 25 HR and as many as 60 HR.

Strat as a system is completely flawed and will never give realistic results because it is still based on the card game (a game I absolutely loved). The way the results come out does not lead to an accurate system because it cannot deal with extreme players. Guys who just don't walk anyone, guys who don't take walks etc, their stats never come out realistically.

[RchW]

Originally Posted by EnderCN

DMB and Strat do not end up with statistics anywhere close to the real major league numbers btw. The last few times I ran full DMB seasons for the Brewers I ended up with Braun with as few as 25 HR and as many as 60 HR.

Strat as a system is completely flawed and will never give realistic results because it is still based on the card game (a game I absolutely loved). The way the results come out does not lead to an accurate system because it cannot deal with extreme players. Guys who just don't walk anyone, guys who don't take walks etc, their stats never come out realistically.

This is what I understood too, but I have no details. Since those that don't like OOTP's historical results seem to have the bit between their teeth, I challenge them to show me actual offensive stats from these other games that show close correlation with RL stats. It's great to talk about OOTP's flaws but to paraphrase the movie, "show me the numbers."

The point I do agree with (from Knockahoma) is the random assignment of poor defensive ability (Belanger). The Lahmann DB doesnt have this information and that will affect historical results. The only fix I see is to become more like DMB Action or Strat. That will result in a price increase to cover the cost of obtaining and translating the information needed.

I'll repeat what I say in every thread like this. You can't compare these games without including the cost of playing. Knockahoma tells us that DMB will offer player editing and player creation, wow! Where have they been for 7 years? Raking in way more cash than OOTPD I think.

I find it curious that OOTPD is being criticized for not being as good as some other games in 3-4 specific areas (historical, PbP, in-game AI?). These games offer no customization no flexibility no real fantasy leagues and cost up to 10 times the $$ depending on how you play. I hope some of you are on their boards asking the developers similar questions about what is missing in their game.

[Wrathchild]

My understanding is that Puresim does a pretty good job of getting close to reality sometimes. I've seen it myself and the following is an example I posted at their boards (note that this was just one player I picked out; this type of close performance is not a rare occurrence):

I was reviewing my 1906 season and it's amazing how similar some of the data is to what really happened in real life. Here's one really good example.

Here's Hal Chase. The first number is what he did for me on my 1906 St. Louis Cardinals. The second number is what he did in real life for the 1906 New York Highlanders. Some of the numbers for my data was affected by the fact that he was playing for such a crummy team.

G 153 / 151
AB 632 / 597
R 56 / 84
H 192 / 193
2B 26 / 23
3B 8 / 10
HR 5 / 0
RBI 56 / 76
BB 39 / 13
SB 8 / 28
OBP .342 / .341
SLG .394 / .395
AVG .304 / .323

Maybe the purists wouldn't think that these numbers were close enough, but I was a bit surprised to see them in a game where I wasn't really attempting to duplicate reality.

[knockahoma]

DMB and Strat do not end up with statistics anywhere close to the real major league numbers btw.

Really? Boy, I played a gazillion seasons with Strat, the same years over and over. I was always impressed with the accuracy. I can tell ya this: Phil Niekro never finished the 74 or 76 season with an ERA over 5.00. My recollection was that he was always around 30 or 40 points of his actual ERA. Memory, though.

I've never played Diamond-Mind. But, after reading it touted as a replay, I did some research and found this. It's using Diamond Mind as a predictor of coming seasons, which suggests a LOT of work getting the players right. I'm still exploring for stat samples of seasons from DM and Strat.

Diamond Mind: Midyear projection review

Summing up
In 2006, our spring simulations correctly identified five of the six division winners and came within a tie-breaker of going 6-for-6. Last year, we had our best year by far, according to statistical measures that evaluate how well we projected the win totals of all 30 teams. Thus far in 2008, the real standings have diverged from our projections by a little more than we've experienced in recent years.

I'd like to see a season comparison on numbers. Seems to me that STRAT used to put those out and they were very impressive.

But, going back to my concern. I just ran the 1974 seasons 6 times. I was curious about how 2 players were imported/and or treated by the AI.

Phil Niekro was an ace during that period of time. Yet, he's repeatedly imported as a "Don't concern yourself over this guy. He's minor league filler."

Mark Belanger was consistently rated at about an 18, or a 19 defensively (scale of 20).

"rated ABOUT". Strangely, with three year calculation on, I noticed that Niekro and Belanger were both imported six times with at least two different ratings sets. (scouts off, which I understand means you're seeing the actual ratings given?)

Niekro:

stuff 8
mov 15
con 12

or

stuff 7
mov 12
con 11

These from Niekro's real numbers in a 3 year period :

1973 13-10, 3.31, 21HR, 89 BB, 131 K, .250 BABIP

1974 20-13, 2.38, 19HR, 88 BB, 195 K, .253 BABIP

1975 15-15, 3.20, 29HR, 72BB 144K, .284 BABIP

Sim after Sim, Niekro had numbers like this from OOTP:

15-20, 5.34, 26HR, 112BB, 151K, .309 BABIP

or

16-19, 5.31, 44HR, 103BB, 150K, .301 BABIP

Niekro's best sim year was:

16-16, 3.97, 25HR, 82BB, 174K, .297 BABIP

Meanwhile, Mark Belanger, was rated as an 18 or 19 shortstop and continually lost his job to Frank Baker. Looking at Baker's real life stats, I couldn't figure out how Baker got his ratings. He was sub-par with bat and glove in 290 ABs as a major leaguer. Belanger was either banished to the minors each season, or traded. The Orioles then played SS by committee. For example, one season had this lineup:

Frank Baker starter. Every 10 games, Larry Milbourne. Every 10 games Brooks Robinson???

I thought, maybe the AI was making good choices based on the ratings. Sacrificing a great glove for a better bat with at least an adequate glove. So, I looked.

Belanger 19 field rating, 9 contact
Milbourne 10 f, 11 contact
Baker 12f, 8 contact
robinson, 1F, 10 contact.

To re-state my feelings on OOTP. I love its flexibility. I'd be happy if I could simply manipulate for the desire results in the OOTP universe with all its great bells and whistles.

So, I'm wondering about how OOTP imports. Why does Niekro keep coming up a Kreepo. Why do Niekro and Belanger get at least two different sets of ratings through six imports when scouts are off?

[knockahoma]

By the way, in the 6 seasons simmed, player development was turned off.

For the developer's edification: I like the small chance that Phil Niekro might veer off into early obscurity, or even the ability to make it a large chance. But I don't like him being imported as "filler" and "nothing to concern yourself over". If he's imported correctly, I think that might ease some of the pain, even if he washes out in the following years.

[RonCo]

Knockahoma should be a beta tester next version.

[knockahoma]

I sense there's sub-text in that statement that I'm failing to pick up, Ron.

I just sense that OOTP is close to something really great with its historical, if not coming out of the box, then at least with the proper manipulation and patience in editing.

I suppose the only way to secure Belanger's rightful place as a starter in Baltimore is to suck the hell out of Frank Baker's ratings. What's funnny is that even when I force the AI to start Belanger at SS by gunpoint at the checkbox, it still gets the last laugh by making someone like Baker start every other game.

But the Niekro thing is a mystery to me. Why would a guy who was a top-tier pitcher in his three year ratings period repeatedly come up as "filler"? That seems to be a translation issue between Lahman and OOTP.

[Aytumious]

Its very possible Niekro is off because he was a knuckleball pitcher.

[Cryomaniac]

Originally Posted by Aytumious

Its very possible Niekro is off because he was a knuckleball pitcher.

That would make sense.

[Theonlystd]

Originally Posted by Aytumious

Its very possible Niekro is off because he was a knuckleball pitcher.

Aye im guessing its pretty hard to import a knuckler in properly just based on there stats considering how knuckler stats usually are.

[Jah]

Originally Posted by RchW

Since those that don't like OOTP's historical results seem to have the bit between their teeth, I challenge them to show me actual offensive stats from these other games that show close correlation with RL stats. It's great to talk about OOTP's flaws but to paraphrase the movie, "show me the numbers."

Okay…in past posts, I’ve hesitated going into specifics because I didn’t want to talk too much about another game on the OOTP boards…but here it goes, purely for informational purposes. (Usual disclaimer…I love OOTP and still use it for fictional play, but recently moved on to Action for my historical replay needs.)

Action makes it easy to compare real vs. computer replay results in a stock report - see the .pdf attachment for the 1969 Orioles, batting and pitching. Realistic results…definitely some variance, but plausible and believable in my book. Now I know it's only one team from one season (I can provide other examples), but the results are good across the board – and I like the fact that it isn't an exact rerun of reality, yet it's believable in the context of the players and teams for the year in question.

Although I used real-life transactions for the 1969 replay, I didn't use actual daily lineups or starting pitchers – instead, I just ran the computer manager on all teams and let it fly. IMHO, pretty damn good results out of the box vs. real-life stats – without having to tweak and test settings ad nauseam. Defensive stats also show a close correlation with real-life stats.

Also, even though nothing matches the depth of OOTP, Action does offer some customization – you can edit or create players (by entering stats and then generating ratings), create a draft league (releasing all players into a draft pool), or set up custom leagues with teams from different eras, etc. There’s also an encyclopedia function to track individual season and cumulative player stats – kind of cool if you want to see how someone did in their career, etc. Despite being a historical replay sim, I must say there are a lot of features in the game I wasn’t expecting, and therefore have been pleasantly surprised.

Like you said, RichW, the replay sims are a different animal compared to OOTP because of the need to purchase season disks…but – with Action at least – “the numbers” are there, if that’s your thing.

[risp2out]

But I don't like him being imported as "filler" and "nothing to concern yourself over".

Can someone shed some light on this? I thought that the scouting reports were given by your scout or the ubiquitous scouting function, OSA. It's my understanding that these are deliberately flawed to some degree to add to the gamesmanship of figuring out who the better players are. Is there an infallible Pope-mode scout? If not, why not just chalk up these comments to scouting mistakes? Just curious. In my limited playing time with historical leagues in OOTP, I've definitely had players who my scout was unimpressed with set the league on fire when I've played them.

I've only been playing OOTP a few months (I'm completely hooked, both with historical players and fictionals). I've played Strat and Pursue the Penant since I was a little kid. I didn't know how much I would enjoy fictional leagues, but I've really had a ball and now I find myself thinking about Carlos "Hoot" Chavez and some poor Tommy "Ratbreath" Maxwell as if they were real players. When I've played with historical players, I've been interested in letting the players move around as opposed to replaying seasons. I was surprised the first time one of my players retired early or had a CEI, but I really like what that brings to the game.

So, I'm not someone who has a horse in this race about stat accuracy, but love reading the posts from both sides and really love learning more about how the game works.

[actionjackson]

Originally Posted by PhillieFever

I'm sorry for getting back so late,I was talking about ballpark factors.I use the default DB most times when I play.

Don't worry Phillie. I'm honoured to have had my query buried in the most interesting thread I have read on this site to date (I'm sure the DIBS, DIPS etc thread was very interesting as well, but it was quite looooong and somewhat intimidating by the time I got to it) and to be quite honest I was surprised to see my name attached to your post, as I had completely forgotten my original post and had to go back and find it for context's sake.

Thanks for the settings - will try them out once I get the game.

As for the discussion, I have to admit to being torn. As a lad I always fiddled with designing games at a very simple level which never produced reasonable/desirable results. I realized quite quickly I didn't have the brains for all the randomness of baseball. Then around age 11, I discovered Strat and was hooked through most of my teen years until discovering other interests. Picked up Strat again in my mid-20's and moved on to the computer version as soon as I had a viable PC. Unfortunately, I ran into a really snot nosed, arrogant as all get out customer service rep one day and started looking for another game.

Diamond Mind was next in my quest for "replay perfection". IMHO, it's a better game than Strat, although I could be biased due to encounter with said dickhead rep and my experiences with the phenomenal Diamond Mind support team. From there it was Baseball Mogul, which was great for a while. I loved the ability to move through history, rather than sim season, buy new version, sim next season buy new version...me want earlier seasons: must buy more season disks...hey this is costing a lot of friggin money.

As for Mogul, it was fun and fast and addictive. Then 2010 came out and it was a disaster at the beginning. IMO, it should have been delayed and polished. I have yet to play a game in it and have only run test sims so far. I won't play until I see some improvement in the results, there has been some, but wow what a mess at the start. I actually got 24 players in a 1901-2008 sim that hit under .200 in more than 3000 PA. To be fair it has improved greatly since then, but that was...well, just sad.

Which brings me to OOTP. Fiddled with version 9 demo and downloaded free version 5 and have messed around with that. I was always intimidated by the learning curve, but it's not too bad. The thing I find is I keep starting dynasties, discovering a new feature that I gotta have in there and starting over again.

I would love to find optimal settings for reasonable (note: not perfect) historical play because I love learning about baseball history while playing a game. I don't want to see Ozzie Smith's name on the HR leaderboards nor John Olerud's name on the SB leaderboards...ever. Can't say I've ever seen that in any other game though. Through my evolution with these games I've gone from: "Hey c'mon that's not realistic!" to "How do I know what the heck realistic is?". In fact, I'm absolutely sure that if you were able to gather everyone into a time machine and replay whatever season you're interested in over and over, you would never get the same results twice and you certainly wouldn't be able to carbon copy said season. This is not meant as a shot across the bow of Styx et al who want more historically accurate results, it's just a fact.

Baseball is a game of skill which hinges on an incredible amount of luck. Pop the ball up in (whatever Oakland's stadium is called these days) and a lot of times you're out. Hit the same popup in Petco and it's into the crowd and you've got another life. Hit a ball that would normally go 400 feet on a line to left in Fenway, you'll dent The Monster and you might get a single or get thrown out at second, or if you're really fast and lucky a double. That ball is a homerun everywhere else. Think of the amount of variables there are within every PA. Hit the sweet spot, most likely life is good unless it's right at someone. One quarter of an inch above and you've got an uninspiring popup, maybe somewhere in between you've got a flare, bloop hit. One quarter inch below, look out could be a DP or maybe it sneaks through that crappy Yankee infield for a hit.

Point is there are tons of variables within each PA and there are what...70-80 PA per game between two teams and 4,860 (approx) games in a regular season. Where are the numbers people? How many different possible outcomes are there over the course of your average baseball season? I would say plenty enough to never have the same season twice and no baseball simulation on the planet, no matter what they say, will give you the same season twice. Even if all the league wide numbers are similar, the results will be different due to the timing of scoring that occurs for each team over the course of a season.

WRT Phil Niekro and Mark Belanger/Ozzie Smith/Mario Mendoza, that is very disappointing and demonstrates the need to a) corral that nasty little BABIP beast once and for all and absolutely related to that b) fix the way the game rates defense. Until the defense is fixed, it will be impossible to get satisfactory pitching, hitting, and of course fielding stats because if any part of the engine is "off" every other aspect of the game is affected.

With Niekro, maybe the game has the Braves defense set up to be far worse than it really was. Thus, a low strikeout pitcher like "Knucksie" gets shelled repeatedly. Belanger being treated like a second class citizen by the AI rather than the defensive wiz that helped seal off the left side of the infield for those Baltimore teams with Brooks Robinson, is going to strongly affect the pitching staff's ability to maintain their excellence. I can't speak to the make-up of those staffs, but it goes without saying that the non-strikeout groundball type pitchers are going to pay for the AI's general silliness.

In short (hah! ya right) in baseball the most seemingly insignificant outcome can determine a ballgame and thus a pennant race and possibly even a World Series. Everything affects everything else. That's what makes it such a brilliant game (except to the dull-minded), but that is also what makes it so damn difficult to simulate in any meaningful, representative way. This is probably the best, most comprehensive and most flexible sim out there. For now, my money's here, but as you can see from my history, I'm a bit shifty and restless, so hopefully someone with more brains and coding experience (nil) than me comes up with the all-important defensive adjustments and fixes that need to be made. Easier said then done, but I'll bet there are a lot of things currently in the game that the veterans here once thought would never make it in.

[struggles_mightily]

Originally Posted by Aytumious

Its very possible Niekro is off because he was a knuckleball pitcher.

Aren't knuckleballers more likely to have "real" control over their own BABIP? They mostly survive on inducing weak contact -- their peripherals often aren't all that great. So OOTP's DIPS system may not be able to capture the skill that makes good knuckleballers effective. Does the game also struggle when importing sinkerballers?

[knockahoma]

Action,

Thanks for that post. Very informative. Jah, your post of the Orioles numbers, computer versus real life, were compelling. When I played Strat cards in my teens and 20s, those are the types of numbers I remember seeing.

I do hope OOTP can get to the point where it can duplicate such numbers. Start there, and then allow the user to dial down to his desired variance. Combined with OOTP's other bells and whistles (fictionals, flexibility, etc) and ....wow!


Well, with any luck and a lot of development, all this will be moot in a few days.

Struggles, looking at the real Niekro numbers and those of his re-incarnation, the bloated over 5 ERA appears to come from his inflated BABIP at first blush, but maybe there's more to the story. The Braves defense does not import anywhere near stellar support. Then again, it was quite average to even sub-par in real life, too. Niekro's HR, BB, K, numbers are reasonably within range of actual numbers in those categories in most of the sims. He did get pounded for over 40 homers in one sim. Most sims he was around 25 HRs. But, he gave up 29 real-life homers in 75 (one of the seasons considered during import for stats) and still came in with a 3.20 ERA that year.

But, Knucksie isn't the only top tier pitcher from 74 regularly suffering with reincarnation. Apparently, Tommy John Surgery isn't as advanced in this universe. And Catfish Hunter often should be nick-named Jim "Sucker-fish" Hunter. It seems to be a combination of import issues, reading pitcher stats incorrectly, while suffering from the imperfections of defense-based stats.

[NHSOXFAN]

Originally Posted by actionjackson

Baseball is a game of skill which hinges on an incredible amount of luck. Pop the ball up in (whatever Oakland's stadium is called these days) and a lot of times you're out. Hit the same popup in Petco and it's into the crowd and you've got another life. Hit a ball that would normally go 400 feet on a line to left in Fenway, you'll dent The Monster and you might get a single or get thrown out at second, or if you're really fast and lucky a double. That ball is a homerun everywhere else. Think of the amount of variables there are within every PA. Hit the sweet spot, most likely life is good unless it's right at someone. One quarter of an inch above and you've got an uninspiring popup, maybe somewhere in between you've got a flare, bloop hit. One quarter inch below, look out could be a DP or maybe it sneaks through that crappy Yankee infield for a hit.

Great post actionjackson. You not only provided a nice, tidy overview of various baseball gaming products but, even more impressive, reminded me, once again, of why I love the game of baseball.

In fact, and I'm not sure why it is, but OOTP best represents the entire baseball experience for me. I am a long time DMB player and I don't believe there is a game that better represents the "in-game," experience, however, there is no "off the field" components for the solitaire gamer.

I suppose it is the ability to lose oneself in a fictional baseball universe that creates the feeling of immersion into a very real (yet fictional) experience.

Ok, I'm rambling but this makes a lot more sense in my mind than on the screen

[knockahoma]

Can someone shed some light on this? I thought that the scouting reports were given by your scout or the ubiquitous scouting function, OSA. It's my understanding that these are deliberately flawed to some degree to add to the gamesmanship of figuring out who the better players are.

I considered that, too. It's my understanding that with scouts off, you're seeing the actual numbers-- so you can edit if need be.

But, if anything, the scout is correct when evaluating Niekro in most of these sims. Niekro is filller. He shouldn't be bothered with. He does suck. But his import seasons (73,74,75) show he was a top tier pitcher. So....?

[knockahoma]

I suppose it is the ability to lose oneself in a fictional baseball universe that creates the feeling of immersion into a very real (yet fictional) experience.

I'm just now beginning to feel the magic of that. I started a fictional league, and went over every name so that it rolled off the tongue as if the player had a mom who gave half a damn about her son. I used a lot of Irish, German, Scottish, Spanish, Polish, eastern european, a few asian etc., last names to get what I considered a perfect mix for a mid 70s league. Then I took the names of real life NL teams, set them in a 3 division format for a total of 12 members, with a wild-card.

Just played the first season in God mode (my preference). And the favorites are already coming to light. For example, a young 23 year old CF for the 3rd place Cubs named Gabrieli, who stole a league high 50 bases.

The name "Lahman" has been banned from this universe, never to be spoken aloud, along with the other forbidden word: DH.

[Flying Dutchmen]

I'm by no means an expert on historical play, but here are some thoughts on the Niekro example. The following are some of his stats from 1973-5 and the NL league pitching stats for that period:

Year   ERA   WHIP   BABIP  ERA+   FIP   L-ERA   L-WHIP
1973   3.31  1.237  .256   120    3.76  3.67    1.331
1974   2.38  1.115  .259   159    3.04  3.63    1.357
1975   3.20  1.295  .288   118    3.81  3.63    1.360

These numbers say to me that Niekro really only had one stellar year of the three mentioned - 1974 - and that his success in 1973 and 1975 was due to the defense backing him. The WHIP numbers in those years don't show someone much better than league average, especially if one considers that the league ERA and WHIP include all pitchers, including those marginal pitchers who made only a few appearances.

It looks more as though defense is the culprit here, but that ties in with well-documented concerns by many on this board regarding how defense is calculated and operates in historical play.

[knockahoma]

These numbers say to me that Niekro really only had one stellar year of the three mentioned - 1974 - and that his success in 1973 and 1975 was due to the defense backing him.

Hi FD. Well, I think you're right in that the defense needs some OOTP overtime attention, especially where it comes to importing correctly.

It may be what you suggest. Two considerations.

The WHIP numbers in those years don't show someone much better than league average, especially if one considers that the league ERA and WHIP include all pitchers, including those marginal pitchers who made only a few appearances.

Even if Niekro were only slightly better in the two seasons (is that right?), with one "stellar" season, I would argue against him rightfully being imported as "filler" talent, and his ERA shouldn't be up around 5.30. I think the import process is being unfair to Knucksie.

I did a quick run to baseball-reference. Actually, in 73 and 75, Atlanta appears to have had a horrible defense ( I know they did as portrayed by strat in the old days). They finished at the bottom of the league in those years in fielding and defensive efficiency. So, it appears any success Niekro had may have been hampered by his defense, at least according to those stats.

In 1974, Atlanta's defense was at or near the top of the league in both categories, perhaps helping elevate Niekro to a 20 win season.