Historical replay accuracy

[pstrickert]

I, for one, agree about randomness. I would hope, however, that we do not attribute all unexpected results to randomness, when they may, instead, be the product of a flawed game engine, fielding model, etc.

[RonCo]

If a .300 hitter gets 600 AB 10,000 times:

1) How many times do you expect him to hit .300 on the nose?
2) How many times do you expect him to hit between .295 and .305?
3) How many times do you expect him to hit between .290 and .310?
4) What's the lowest you can expect him to hit?
5) What's the highest you can expect him to hit?

[RonCo]

Originally Posted by pstrickert

I, for one, agree about randomness. I would hope, however, that we do not attribute all unexpected results to randomness, when they may, instead, be the product of a flawed game engine, fielding model, etc.

True. But understanding what randomness can provide is important with regard to analyzing what problems really exist, too.

[pstrickert]

Of course.

[Sweed]

Originally Posted by Aytumious

I just want to say struggles mightily, you've made some excellent posts in this thread. I was going to comment on some things, but you already made the case.

+1

I've enjoyed reading his posts.

[knockahoma]

If a .300 hitter gets 600 AB 10,000 times:

1) How many times do you expect him to hit .300 on the nose?
2) How many times do you expect him to hit between .295 and .305?
3) How many times do you expect him to hit between .290 and .310?
4) What's the lowest you can expect him to hit?
5) What's the highest you can expect him to hit?

This is a very reasonable question, especially to elicit feedback for the developer. I actually like the flexibility of OOTP. I was talking to my brother about it the other day. While I enjoyed Strat-o-matic immensely, I was bothered by great players having off years. In 1976, Johnny Bench suffered through his worst year, hitting .234, 16 Homers, if memory serves (that's how much I loved that game ). If I'd had the 1975 set, I'd have switched out Johnny Bench cards to get the better player.

In other words, in that world, Johnny Bench just wasn't... Johnny Bench! It's funny. There's a lot of math about why this or that player performs differently from year to year. But, how much of that is in the head and confidence of a player from year to year? It's weird to think that Individuals are in a very real sense a "different player" from year to year, based on statistical performance. As different as two players named Jones and Smith, one who hit 15 homers and one who hit and 25. Johnny Bench of 75 was not the same guy who wore the same name-tag in 76.

Anyway.

If a .300 hitter gets 600 AB 10,000 times:

1) How many times do you expect him to hit .300 on the nose?

Not applicable to me.

2) How many times do you expect him to hit between .295 and .305?

Not applicable.

3) How many times do you expect him to hit between .290 and .310?

Applicable. I'd like to see that about 50% of the time. That seemed to be what I experienced with Strat.

4) What's the lowest you can expect him to hit?

Almost not applicable, but let's say... .230-.250. Why not applicable? Because with Strat, you'd have that occasional disappointment, but 20 of 25 players performed damn close to their real life stats. It was pretty cool. Just enough variance that a few players might over-perform, or under-perform, leading to possibly a different place in the standings.

5) What's the highest you can expect him to hit?

.340-ish, and see above.


Here's why I've been replaying the same seasons. I'm hoping to (A) understand what settings get me the desired results and (B), then manipulate the settings to bring about my desired result.

But, that's the potential beauty with the flexibility. Maybe I'll get tired of how I answered the questions above. Maybe I'll want a bit more variance. Then, I simply manipulate from there. But, I need access to numbers, percentages, quirks, etc., in order to manipulate the engine.

[RonCo]

Not bad...I wrote a tiny program to test this. Here is what I got:

		Run 1	Run 2
.300		3.44%	3.40%
.295-.305	23.99%	24.52%	
.290-.310	43.50%	44.12%	
Highest		.368	.361
Lowest		.240	.240

So your 50% number is just a little high. In addition, 56% of .300 hitters will hit lower than .290 or higher than .310 (half above and half below). This is with 600 AB. With considerably less AB, the spread is quite a bit higher.

Another way of saying this is that if 28% of true .300 hitters _don't_ hit between .240 and .290 (and between .310 and .360), then the game is probably not random.

[Pdubya64]

RonCo:
I assume each run you did is 10,000 executions? (Thx btw)

[Siv]

The beauty of baseball is that we see and feel differnet things. I am in an industry that feeds off these same type of situations (slot machines).

In the slot machine industry machines have to pay back a certain percentage (85-95% typically) over the course of "the machine life cycle". This life cycle is 10 million handle pulls.

The average gamer cannot play a machine for 10 million handle pulls as they are don't have enough money or time to sit on that one machine to do this. The machine doesn't know what it's last handle pull was, it just has the formula's stored in it's memory and the random number determines the result. Nor does the machine store any of it's results for consultation later.

In the first 1000 handle pulls the variance to payback is 50%. But if a player hits a jackpot on that machine in 100 handle pulls their perception is that this is a great machine. As the machine gets more handle pulls the variance gets tighter. At 10000 handle pulls the variance is 30% from it's payback. 1 million it is at 10% and finally at 10 million games it is within half a percent.

This whole time a varied group of customers are playing this machine and are perceving different results from a lucky machine to a tight one.

This i believe is similar to a baseball player and it's fans. With only 600 AB's a season, one cannot capture all the variables that go into the "life cycle" of a baseball player, but the fans have a perception they believe to be true. Thus you take to heart how great or pooer this player is and many times what they have done in "critical" games and situations because as the fan you experienced these results in a live format and felt all the anticipation and excitement that goes with it.

Now that the fan or customer has that set it there mind, their expectations will be further jaded and thus not allowed to take in the big picture!

With that said the feeling and analytical parts of a fan will always out weigh what a baseball player will be able to do based on not having enough opportunities to complete his life cycle in a particular season. Even 12000 AB's in a career isn't enough to normalize the variables that go into a baseball players career to give any idea of an anticipated result.

A player is still within the 20-30% variable which is the best we can see in his 20 year career. We will have a good idea, but since every session (game) we see of a player is only 4-5 AB's it further strews our belief in what they can do and coats our memeory with inaccuracies that we will keep forever.

It is no wonder that baseball players like gambler's are supersticious and always believe that they will hit the big one or win the game. It is probably why I enjoy baseball so much, because no matter when, where or how; there is always a chance to win.

[struggles_mightily]

Originally Posted by pstrickert

I, for one, agree about randomness. I would hope, however, that we do not attribute all unexpected results to randomness, when they may, instead, be the product of a flawed game engine, fielding model, etc.

Absolutely. I hope that my statements in this thread don't seem like an attempt to shout down criticism of OOTP. There are many areas in which it's not perfect, and some in which it's far from that. I just think that some of the criticism in this thread is misguided.

Fielding is a very good example. I'm not sure that OOTP handles fielding very well at the moment. But its struggles are compunded by issues that it's not responsible for, too. Importing historical players in a very un-sophisticated stats-based way will always be ultimately unsatisfying. There just isn't a good trans-historical fielding metric available. There's FRAR, but that is heavily criticised by people who know far more than I about the issue.

Not bad...I wrote a tiny program to test this. Here is what I got:

Thanks a lot for doing that RonCo, and thus fleshing out my rhetorical skeleton.

Oh, and thanks very much also to the posters who said they have enjoyed my posts in this thread.

[RonCo]

Originally Posted by Pdubya64

RonCo:
I assume each run you did is 10,000 executions? (Thx btw)

Yes, each run had the .300 hitter encounter 600 AB 10,000 times.

[RonCo]

Great comparison, Siv.

[Carlton_Willey]

Siv makes great points and leads to an addition to all the current sports sims I play--OOTP, FOF, and FM, mostly--that I'd love to see implemented in some form.

I'd like an indication of the die roll and whether or not it was a lucky one. These games have lots of die rolls for each event, I think and not one 'handle pull' but I'd still like to see a comment about luck when an almost-off-the-chart event occurred.

A lot of the perception of bad design would go away if the player could see the die rolls. I play board games, and its improbable to roll three sixes three times in a row, but I've done it and seen it done. When this happens we don't blame the game, we blame the dice

Perhaps a 'luck factor' column added to individual totals at the end of the season so we could determine the impact of the die rolls on the player's performance. It could be based on standard deviation, and done broadly. One, two or three STDs up or down, from expected would be a useful indicator for the die roll impact.

[struggles_mightily]

Originally Posted by Carlton_Willey

Perhaps a 'luck factor' column added to individual totals at the end of the season so we could determine the impact of the die rolls on the player's performance. It could be based on standard deviation, and done broadly. One, two or three STDs up or down, from expected would be a useful indicator for the die roll impact.

Won't we sort-of get this in OOTPX, via the new feature which provides player ratings based on entered stats? That way, we'll know that -- all being equal -- a player with a particular set of ratings 'should' be an .800 OPS player (or whatever). Combine that with park factors and league averages, which are already figurable, and the numbers will be there for someone to do a comprehensive study linking ratings with statistics-in-a-vacuum.

[knockahoma]

Thanks, Ron, Siv. Nice posts. I wonder if STRAT and Diamond mind have run such numbers on accuracy for their games?

Fielding is a very good example. I'm not sure that OOTP handles fielding very well at the moment. But its struggles are compunded by issues that it's not responsible for, too. Importing historical players in a very un-sophisticated stats-based way will always be ultimately unsatisfying. There just isn't a good trans-historical fielding metric available. There's FRAR, but that is heavily criticised by people who know far more than I about the issue.

One of Strat-o-matic's best features, IMO. I like the way they scour numbers AND professional opinions. Two quick stories. I used to interview on a regular basis an infield coach for the Brewers and Angels named Sam Suplezio. One day we got into the nitty gritty of player's arms, range, etc. Sam was a spring training infield coach for the Milwaukee at that time. We were discussing Franlin Dubbs, a natural 1B and decent lefty hitter, who was being tried out in RF. Remembering Strat's arm rating for Dubbs, I said, " Isn't Franklin's arm a little weak for RF?'

Suplezio looked at me and said, " You must be a HELLUVA baseball fan!"

I once read an article recounting a meeting between the creator of STRAT and CF Bobby Dernier (Philly and the Cubs). Range ratings are between 1-4, and few get the coveted 1. Dernier got a 2 that year. Like many major leaguers, Dernier was an avid strat fan. So, he hunted down the creator to argue his case. Based on numbers and scouting, the latter gave a very detailed explanation in defense of the 2 rating given to Dernier.

Now, that's effort in getting the defense right. It's why I borrow the ratings from STRAT when I edit OOTP. Of course, Strat has different intentions as a game. But, I wish there was some down-loadable source for OOTP that married stats and scouts together for defensive ratings, some source that would make a Sam Suplezio say, "Wow!"

[knockahoma]

A lot of the perception of bad design would go away if the player could see the die rolls. I play board games, and its improbable to roll three sixes three times in a row, but I've done it and seen it done. When this happens we don't blame the game, we blame the dice

So true! LOL. I haven't played strat in years, but I understand they kept the dice roll in the computer game because players loved it so much?

Even though STRAT uses a step-by-step approach (where you see the dice roll, then look at the SS's defense rating to determine if he has the range to reach the ball), the method's allure is the vision it puts in your head. Now, you see that grounder that few but Belanger could have gloved. It's more than "Great play!". Average shortstops make "great plays" too. The magic was "seeing" a play that no one but a 1-rated shortstop could make.

[struggles_mightily]

One of Strat-o-matic's best features, IMO. I like the way they scour numbers AND professional opinions

Yup, this is the essential reason why OOTP will not satisfy historical re-players. Games which seek historical replay are able to devote much more time and effort to ensuring that a player's performance in a particular season passess the smell test. With OOTP, you have the Lahman and that's it.

Raw stats offer an acceptable -- or better -- recreation of historical players' abilities in most areas. Defense is one of those areas where this is not so.

It's a "season disk" versus "stats database" question. And, whaddayaknow?, the issue of which is "better" comes down boringly to the matter of 'personal preference.'

[knockahoma]

Apparently, the Alzheimer's is advancing faster than I thought. If Sam were alive to read my last post, instead of saying, " You must be a helluva baseball fan!", he'd say, " Franklin Dubbs? Whose Franklin Dubbs? Do you mean Franklin STUBBS?"

It's a "season disk" versus "stats database" question. And, whaddayaknow?, the issue of which is "better" comes down boringly to the matter of 'personal preference.'

I gotta say, I'm surprised this community hasn't collected a library of seasons edited for the historically hysterical. I'd be happy to do a few (74,76, 83, 84, 85, 86), but, candidly, I'd just be using STRAT ratings.

But even with that, some defensive guys would continue to be under-valued by the AI. I really hope Marcus fixes how the AI looks at SS, for example. My experience has been that wonder-gloves like Belanger often get snubbed, even after their ratings are edited, for average to poor fielders who hit .260 with 5 homers. In other words, an iron glove gets the job for batting 15 points higher than Belanger.

[Siv]

Originally Posted by knockahoma

So true! LOL. I haven't played strat in years, but I understand they kept the dice roll in the computer game because players loved it so much?

Even though STRAT uses a step-by-step approach (where you see the dice roll, then look at the SS's defense rating to determine if he has the range to reach the ball), the method's allure is the vision it puts in your head. Now, you see that grounder that few but Belanger could have gloved. It's more than "Great play!". Average shortstops make "great plays" too. The magic was "seeing" a play that no one but a 1-rated shortstop could make.

Wow I haven't thought of Strat O Matic in years! I would love to see the dice roll and cards, man that was awesome to play. i would sit for hours playing the season I had (1979) and keeping all the stats. The anticipation of the dice roll and looking at the card were awesome. The control of the pitchers and whether you were trying to get a hit off their card was another cool feature. I honestly thought they were out of business.

i don't know that OOTP could do that, because it isn't based on the dice. But who knows.

[RonCo]

Originally Posted by Siv

i don't know that OOTP could do that, because it isn't based on the dice. But who knows.

Dice are just a way to manually create random numbers. OOTP is based on random numbers the same way that Strat or any of the old board games are.