How well-calibrated are the new "Superstats", really?

[hjrrockies]

I am running my first save in OOTP 27, and I just ran a quick test that gave some concerning results. Midway into June, the league average wOBA is 0.311 (fine), but the league average xwOBA is just .295 (a 17 point deficit). I computed these values by doing a weighted average with player wOBA/xwOBA multiplied by PA, then divided by the total league PA. This gives the averages with respect to plate appearances, which is important.


Obviously there's going to be *significant* player-to-player variation in wOBA vs. xwOBA. But at a bare minimum, the average xwOBA should be close to the average wOBA.


I know that the Superstats are not part of an overhaul of the underlying engine, and I emphatically do not want OOTP to switch to a computationally infeasible physics-based model. That said, I am left a bit concerned with the possibility that the Superstats, as implemented, are not calibrated enough to be useful.

[luckymann]

Originally Posted by hjrrockies

I am running my first save in OOTP 27, and I just ran a quick test that gave some concerning results. Midway into June, the league average wOBA is 0.311 (fine), but the league average xwOBA is just .295 (a 17 point deficit). I computed these values by doing a weighted average with player wOBA/xwOBA multiplied by PA, then divided by the total league PA. This gives the averages with respect to plate appearances, which is important.


Obviously there's going to be *significant* player-to-player variation in wOBA vs. xwOBA. But at a bare minimum, the average xwOBA should be close to the average wOBA.


I know that the Superstats are not part of an overhaul of the underlying engine, and I emphatically do not want OOTP to switch to a computationally infeasible physics-based model. That said, I am left a bit concerned with the possibility that the Superstats, as implemented, are not calibrated enough to be useful.

So, what's the exact calculation you use here? I have a save that's just about to finish its first full season on 27 and will give you my results if you can clarify that formula for the mathematically challenged. Where does one even find the league-wide wOBA or xwOBA?

Thanks

G

[WhiskyTango]

I used to worry about the usefulness of the stats but I don't anymore thanks to the interventions of forum moderators and helpful forum members.

[hjrrockies]

Originally Posted by luckymann

So, what's the exact calculation you use here? I have a save that's just about to finish its first full season on 27 and will give you my results if you can clarify that formula for the mathematically challenged. Where does one even find the league-wide wOBA or xwOBA?

Thanks

G

Easiest way to do it on a default OOTP game:


1. Open the "Batting Stats 2" view on the "Sortable Stats" tab for MLB.
2. Customize the view to include xwOBA from "Batting Superstats".
3. Make sure all players are displayed (no filters, no "batting qualifiers").
4. Export the table to a CSV file.
5. Open the CSV file (you'll need to find your OOTP export folder)
6. Make a new column in the CSV file that is equal to the PA column multiplied by the wOBA column. This makes it so we are computing an average weighted by plate appearances, not by player count.
7. Make another new column that is equal to the PA column multiplied by the xwOBA column.
8. Compute the sum of the two new columns using the SUM function. Also compute the sum of the PA column.
9. Divide the sum of the two new columns by the sum of the PA column. This gives you the league-average wOBA and league-average xwOBA, respectively.

[BaseballATeam]

Just an fyi, it's not uncommon for the delta of MLB wOBA and xwOBA to reach .009. It's happened often enough that it's probably a 1-1.5 SD event. So, .017 isn't necessarily remarkable.

I think we should also plan to be patient about the realism of these stats. irl, we build these from the physical inputs of what makes a hit/out. In OOTP, they're automagically generated to look plausible, but the physics engine doesn't work in a manner that would allow an apples-to-apples relationship between the real and game metrics.

Here's an old conversation about just such a gap. https://www.reddit.com/r/baseball/co..._so_different/

[luckymann]

Originally Posted by hjrrockies

Easiest way to do it on a default OOTP game:


1. Open the "Batting Stats 2" view on the "Sortable Stats" tab for MLB.
2. Customize the view to include xwOBA from "Batting Superstats".
3. Make sure all players are displayed (no filters, no "batting qualifiers").
4. Export the table to a CSV file.
5. Open the CSV file (you'll need to find your OOTP export folder)
6. Make a new column in the CSV file that is equal to the PA column multiplied by the wOBA column. This makes it so we are computing an average weighted by plate appearances, not by player count.
7. Make another new column that is equal to the PA column multiplied by the xwOBA column.
8. Compute the sum of the two new columns using the SUM function. Also compute the sum of the PA column.
9. Divide the sum of the two new columns by the sum of the PA column. This gives you the league-average wOBA and league-average xwOBA, respectively.

Great, much appreciated, leave it with me.

G

[luckymann]

OK, I got the csv but made it more comprehensive, then gave it to Chat for analysis. Here's the response I got, FWIW.

I kept the full file intact, including the two-way players, and then calculated the league environment three ways: with no PA cutoff at all, with a 50 PA minimum for a cleaner player pool, and with a 100 PA minimum as an even more stable reference point. That gives you both the pure league environment and the cleaner signal once the smallest samples are stripped out.

League environment

With no cutoff, the file contains 503 players and 152,819 total PA. The PA-weighted league-average line comes out to .3363 wOBA against .3135 xwOBA, .3426 OBP, .2745 AVG, .4360 SLG, .3035 BABIP, 100.4 wRC+, 101.4 OPS+, 8.83% BB, 15.63% K, 28.80% chase rate, 77.82% Z-Contact, 29.24% whiff rate, 83.78 EV, 114.60 max EV, 10.85 launch angle, 7.43% barrel rate, 23.71% hard-hit rate, and 11.78% HR/FB.

At 50 PA and above, the pool drops to 375 players and 150,769 PA, which means you are barely losing any plate appearances and mostly just removing noise. The weighted league line becomes .3370 wOBA against .3140 xwOBA, .3432 OBP, .2750 AVG, .4370 SLG, .3040 BABIP, 100.8 wRC+, 101.8 OPS+, 8.86% BB, 15.63% K, 28.77% chase, 77.81% Z-Contact, 29.24% whiff, 83.82 EV, 114.71 max EV, 10.85 launch angle, 7.47% barrel rate, 23.79% hard-hit rate, and 11.85% HR/FB.

At 100 PA and above, the numbers barely move again. The weighted line is .3377 wOBA against .3146 xwOBA, .3438 OBP, .2754 AVG, .4383 SLG, .3042 BABIP, 101.3 wRC+, 102.3 OPS+, 8.88% BB, 15.60% K, 28.76% chase, 77.84% Z-Contact, 29.22% whiff, 83.87 EV, 114.78 max EV, 10.89 launch angle, 7.52% barrel rate, 23.90% hard-hit rate, and 11.92% HR/FB.

The important point is that the environment is very stable once you remove the tiniest samples. That is a good sign in itself.

Actual versus expected tracking

This is where the file gets more interesting. Leaguewide, actual production is running ahead of expected production.

With no cutoff, league wOBA is .3363 versus .3135 xwOBA, so the weighted gap is +.0228. At 50 PA and above, the gap is still +.0230. At 100 PA and above, it is +.0231. That consistency matters. This is not a tiny-sample artifact. The sim appears to be producing a league where actual wOBA sits about twenty-three points above xwOBA.

By contrast, slugging is almost perfectly aligned. League SLG is .4360 versus .4309 xSLG with no cutoff, a gap of only +.0050. At 50 PA and above the gap is +.0053, and at 100 PA and above it is +.0053 again. That is very tight.

Batting average also runs a little hot, but not wildly so. AVG is .2745 versus .2672 xBA with no cutoff, a gap of +.0074. At 50 PA and above it is +.0076, and at 100 PA and above it is +.0080.

So if you are looking for the headline takeaway, it is this: the sim’s expected slugging and expected batting average are tracking fairly tightly to the realised league environment, but xwOBA is running meaningfully below actual wOBA at the league level.

Player-level spread

Using the 50 PA pool, the average player-level wOBA minus xwOBA gap is +.0203, and the median is +.0230. The 10th percentile is about -.019 and the 90th percentile is about +.0596. More importantly, 54.4% of players are more than +.020 above xwOBA, while only 9.6% are more than -.020 below it. That is a very lopsided distribution. If the expected model were fully centred, you would normally expect something more balanced around zero.

For SLG minus xSLG in the 50 PA pool, the average gap is only +.0022 and the median is +.0060. The 10th percentile is -.066 and the 90th percentile is +.0706. That looks much more naturally centred. There is real variance, but not an obvious directional bias.

For AVG minus xBA, the average gap is +.0023 and the median is +.0030. The 10th percentile is -.0396 and the 90th percentile is +.0410. Again, that looks much healthier than the xwOBA distribution.

Relationship tracking

This is the other major realism check. I focused on the 50 PA pool here, because it gives you cleaner player-to-player signal.

wOBA versus xwOBA has a correlation of 0.663. SLG versus xSLG is 0.689. AVG versus xBA is 0.549. EV versus xwOBA is 0.673. Barrel% versus SLG is 0.574. BB% versus OBP is 0.597. K% versus Z-Contact is -0.868. Chase rate versus BB% is -0.477.

Those are actually pretty encouraging. The stats are behaving in the right directions, and in most cases with healthy strength. The strongest relationship in the set is K% versus Z-Contact, which is exactly what you would want to see. EV and barrel rate are also mapping sensibly to damage production. So the internal logic of the metrics looks broadly baseball-like.

What this suggests

My read is that the simulation’s modern stats do not look random or decorative. They are interacting in coherent ways. Contact quality is mapping to power. Zone contact is strongly tied to strikeout rate. Chase is negatively tied to walks. Expected slugging is close to realised slugging. Those are all good signs.

The main red flag is xwOBA. On a leaguewide weighted basis it is consistently about twenty-three points lower than actual wOBA, and the player-level deltas are skewed heavily toward overperformance. That suggests either the xwOBA model is too conservative, or the sim is generating enough non-contact or sequencing value that wOBA is systematically outrunning the expected version. Either way, that is the one metric in this file that looks least well calibrated.

The practical verdict

If I were grading this file purely on realism behaviour, I would say the contact-quality and shape metrics look broadly believable, xSLG looks pretty well calibrated, xBA looks acceptable, and xwOBA is the main area that appears biased low relative to realised outcomes.

So the best single sentence summary is this: the stat ecosystem looks internally coherent, but the xwOBA model appears to be understating leaguewide offensive value.

[Sweed]

Thanks hjrrockies for posting your data.

I'm with BaseballATeam, we need to be patient with this new feature. I would imagine as the game is played more and more, and users report what they are seeing, they will be tweaking this to more realistic numbers. Maybe in a patch? Or maybe not until v28? Users like hjrrockies posting their data will be helpful to the developer to make improvements to this new feature.

OOTP adding advanced stats to the game has finally got me reading a book, recommended by another user here, on the subject. Until now I vaguely knew how to use the newer stats that were already in the game, ie wRC+, wOBA, etc. while not taking the time to study the underlying math. The book isn't drastically changing how I viewed the stats I concentrated on, but it is interesting to see the formulas. The book is opening my mind to some of the newer stats I hadn't paid much attention to, giving me a bigger picture to look at to evaluate players.

My thought reading some of these threads is OOTP may not have the advanced stats scaled as close to reality as "those in the know" (not me, though I'm learning more and more from my book ), would like, but the advanced stats are all being generated by the OOTP engine. I'm not an expert on the subject, but in the context of my OOTP league, I think there is still value in the numbers being displayed even if they are off from what real life numbers are.

[Todd R]

I don't understand who there can be a mismatch. Ratings create performance and ratings create expected performance.