Extensive Tests on MLB Quickstart: Starters and Relievers need to be re-Balanced

hjrrockies · 04-10-2017, 04:24 AM

So I am a tad bit obsessed with the ratings and simulation engine. First off, I want to commend the devs for putting out an amazing product, as usual.

I started diving down the rabbit hole when I noticed that my favorite Rockies pitcher, Jon Gray, always had really poor strikeout rates in my sims. I posted here about it, and Lukas kindly buffed Gray a tad, in agreement with my assessment.

I've noticed a larger overall trend, however: in OOTP, starting pitchers are getting too few strikeouts and relievers are getting too many. I have data to prove it.

I ran 5 full-season sims with the 2017 MLB quickstart. I turned trading, injuries, morale, player development, and team chemistry off. At the end of each sim, I exported the pitching stats, split on "As Starter" and "As Reliever." With a little python code, I ran some analysis.

At the top end, the big-name starting pitchers are definitely getting fewer strikeouts than expected. Despite being projected by ZiPS for a K-rate above 30%, in the more than 1000 innings I simmed, Kershaw averaged a K-rate below 29%. Yu Darvish, owner of a career 30.1% rate and a ZiPS-projected 28% rate, strikes out only 24% across his 1000 innings. Very few starting pitchers match their ZiPS-projected K-rates.

Seeing this trend, here are the league totals:
SP K%, Simmed: 19.2%
SP K%, 2016 IRL: 20.2%
RP K%, Simmed: 25.2%
RP K%, 2016 IRL: 22.7%
Total K%, Simmed: 21.2%
Total K%, 2016 IRL: 21.1%

So, despite the league totals being correct (as set in the League Settings menu), there seems to be an imbalance in how the simulation engine treats starters vs. relievers. Starters are getting about 1 percentage point less than expected, and relievers are up by 2 and a half points. It would be interesting to hear from the developers on if there's a simple tweak that can rectify this imbalance.

hjrrockies · 04-10-2017, 04:34 AM

Just ran the numbers really quickly on walk rates:

Starting pitchers are spot-on with walk rates: 7.7% in game and IRL.
Relievers walks are a bit inflated in-game, 9.9% in-game vs 8.9% IRL.

Anecdotally, this lines up with my expectations, as I have noticed a distinct trend in OOTP for very high-walk relievers relative to real life.

Bunktown Ballers · 04-11-2017, 09:46 AM

Better Hitting?
Great thing about baseball is that ever stat is related to every stat. You start by saying Pitching K's down, but don't care to mention than hitters are striking out less. Are Homeruns Up/Down, Doubles Up/Down, How about walks, HBP...etc....Baseball does have trends & maybe just a new trend.

Just my 2 cents
BB

jpeters1734 · 04-11-2017, 10:03 AM

take it how you want it, but in my 88 game league, we just got to the halfway point and 2 pitchers are on pace to break the K's record by almost 40 K's. This is the first season on 18

Rain King · 04-11-2017, 10:04 AM

I'm curious if this is more an issue with the MLB Roster set or whether it is an issue (as you say) with how the game treats starters and relievers. What do these splits look like in a fictional league?

I believe they set the MLB ratings based on projected stats...is it possible they are not then adjusting for how the game treats the two?

Lukas Berger · 04-11-2017, 10:37 AM

Quote:

Originally Posted by Rain King

I'm curious if this is more an issue with the MLB Roster set or whether it is an issue (as you say) with how the game treats starters and relievers. What do these splits look like in a fictional league?

I believe they set the MLB ratings based on projected stats...is it possible they are not then adjusting for how the game treats the two?

If it is an issue, and I'd have to check into it further, I'd guess it is a roster set issue.

My initial take is that the percentages the OP is coming up with are actually very good, they aren't far off at all. If starters are off 1% and relievers 2.5%, that's not a cause for concern in my opinion, that means we're doing a great job and that this is almost perfect.

That's well within any reasonable margin of error and we're really not going to be able to get things any closer than that.

A few percentage points off in one direction or another is just how statistical distributions naturally fall. Since we're using actual math, not forcing results to conform to an exact template, there's always going to be a bit of play in the results.

These results are really, really accurate. I don't know how we could do any better without abandoning real math and forcing predetermined results on the game engine.

Lukas Berger · 04-11-2017, 10:48 AM

Something else that's important to note is that having injuries off is going to skew the results. It's not an accurate way to test the engine, since it's not really how the game was designed to be played.

It results in the pool of players as a whole being stronger than it's intended to be with injuries on, particularly the bottom of the pool.

So because of how the game works, what that means is that more positive results are going to be distributed to the lowest rated players than would naturally happen if injuries were on, and more replacement players were getting more playing time.

Thus that takes some pieces of the pie away from the top pitchers, if you will, and is going to result in them performing worse than they would in a typical environment, where injuries etc are on. That's just how the game is designed.

BigRed75 · 04-11-2017, 11:29 AM

I'll be honest with you, five simmed seasons is a really small sample size. I know it might seem like a lot from the perspective of 12,150 games, but it's really not. Think of the variances that can occur IRL in a five-year span.

I'd be more curious of the numbers in, say, 100 seasons. But like Lukas said, the variances with five seasons are probably within the margin of error anyway

hjrrockies · 04-12-2017, 01:56 AM

Quote:

Originally Posted by Lukas Berger

If it is an issue, and I'd have to check into it further, I'd guess it is a roster set issue.

My initial take is that the percentages the OP is coming up with are actually very good, they aren't far off at all. If starters are off 1% and relievers 2.5%, that's not a cause for concern in my opinion, that means we're doing a great job and that this is almost perfect.

That's well within any reasonable margin of error and we're really not going to be able to get things any closer than that.

A few percentage points off in one direction or another is just how statistical distributions naturally fall. Since we're using actual math, not forcing results to conform to an exact template, there's always going to be a bit of play in the results.

These results are really, really accurate. I don't know how we could do any better without abandoning real math and forcing predetermined results on the game engine.

I hope I didn't come across as being overly critical: you guys do a fantastic job, seriously.

I re-ran my tests to use default ratings all across the board (AKA injuries on, player morale on, etc), 10 sims this time. Starter strikeout rate remained the same, and reliever strikeout rate declined a half-point.

Anyway I totally understand that this is the lamest of all possible concerns. Thanks for taking the time to respond.

04-10-2017, 04:24 AM	#1
hjrrockies Minors (Single A) Join Date: Mar 2016 Posts: 78	Extensive Tests on MLB Quickstart: Starters and Relievers need to be re-Balanced So I am a tad bit obsessed with the ratings and simulation engine. First off, I want to commend the devs for putting out an amazing product, as usual. I started diving down the rabbit hole when I noticed that my favorite Rockies pitcher, Jon Gray, always had really poor strikeout rates in my sims. I posted here about it, and Lukas kindly buffed Gray a tad, in agreement with my assessment. I've noticed a larger overall trend, however: in OOTP, starting pitchers are getting too few strikeouts and relievers are getting too many. I have data to prove it. I ran 5 full-season sims with the 2017 MLB quickstart. I turned trading, injuries, morale, player development, and team chemistry off. At the end of each sim, I exported the pitching stats, split on "As Starter" and "As Reliever." With a little python code, I ran some analysis. At the top end, the big-name starting pitchers are definitely getting fewer strikeouts than expected. Despite being projected by ZiPS for a K-rate above 30%, in the more than 1000 innings I simmed, Kershaw averaged a K-rate below 29%. Yu Darvish, owner of a career 30.1% rate and a ZiPS-projected 28% rate, strikes out only 24% across his 1000 innings. Very few starting pitchers match their ZiPS-projected K-rates. Seeing this trend, here are the league totals: SP K%, Simmed: 19.2% SP K%, 2016 IRL: 20.2% RP K%, Simmed: 25.2% RP K%, 2016 IRL: 22.7% Total K%, Simmed: 21.2% Total K%, 2016 IRL: 21.1% So, despite the league totals being correct (as set in the League Settings menu), there seems to be an imbalance in how the simulation engine treats starters vs. relievers. Starters are getting about 1 percentage point less than expected, and relievers are up by 2 and a half points. It would be interesting to hear from the developers on if there's a simple tweak that can rectify this imbalance. Last edited by hjrrockies; 04-10-2017 at 04:25 AM.

04-11-2017, 09:46 AM	#3
Bunktown Ballers All Star Starter Join Date: Apr 2013 Location: Louisiana Posts: 1,598 Infractions: 0/2 (3)	Could It Be... Better Hitting? Great thing about baseball is that ever stat is related to every stat. You start by saying Pitching K's down, but don't care to mention than hitters are striking out less. Are Homeruns Up/Down, Doubles Up/Down, How about walks, HBP...etc....Baseball does have trends & maybe just a new trend. Just my 2 cents BB

04-11-2017, 10:03 AM	#4
jpeters1734 Hall Of Famer Join Date: Jun 2014 Location: Juust a bit outside... Posts: 6,216	take it how you want it, but in my 88 game league, we just got to the halfway point and 2 pitchers are on pace to break the K's record by almost 40 K's. This is the first season on 18 __________________ "Cannonball Coming!" Go Bucs!! Founder and League Caretaker of the Professional Baseball Circuit, www.probaseballcircuit.com An Un-Official Guide to Minor League Management in OOTP 21 Ratings Scale Conversion Cross-Reference Cheat Sheet

04-11-2017, 10:48 AM	#7
Lukas Berger OOTP Developments Join Date: Aug 2007 Location: Nice, Côte d'Azur, France Posts: 21,362	Something else that's important to note is that having injuries off is going to skew the results. It's not an accurate way to test the engine, since it's not really how the game was designed to be played. It results in the pool of players as a whole being stronger than it's intended to be with injuries on, particularly the bottom of the pool. So because of how the game works, what that means is that more positive results are going to be distributed to the lowest rated players than would naturally happen if injuries were on, and more replacement players were getting more playing time. Thus that takes some pieces of the pie away from the top pitchers, if you will, and is going to result in them performing worse than they would in a typical environment, where injuries etc are on. That's just how the game is designed. __________________ lukas@ootpdevelopments.com PreOrder Out of the Park Baseball 26! Need to upload files for us to check out? Instructions can be found here

04-11-2017, 11:29 AM	#8
BigRed75 Hall Of Famer Join Date: May 2009 Posts: 2,430	I'll be honest with you, five simmed seasons is a really small sample size. I know it might seem like a lot from the perspective of 12,150 games, but it's really not. Think of the variances that can occur IRL in a five-year span. I'd be more curious of the numbers in, say, 100 seasons. But like Lukas said, the variances with five seasons are probably within the margin of error anyway __________________ Mainline team SPTT team Was not a Snag fan...until I saw the fallout once he was gone and realized what a good job he was actually doing. - Ty Cobb

04-10-2017, 04:34 AM	#2
hjrrockies Minors (Single A) Join Date: Mar 2016 Posts: 78	Just ran the numbers really quickly on walk rates: Starting pitchers are spot-on with walk rates: 7.7% in game and IRL. Relievers walks are a bit inflated in-game, 9.9% in-game vs 8.9% IRL. Anecdotally, this lines up with my expectations, as I have noticed a distinct trend in OOTP for very high-walk relievers relative to real life.

04-11-2017, 10:04 AM	#5
Rain King Hall Of Famer Join Date: Nov 2005 Posts: 3,110 Infractions: 0/1 (1)	I'm curious if this is more an issue with the MLB Roster set or whether it is an issue (as you say) with how the game treats starters and relievers. What do these splits look like in a fictional league? I believe they set the MLB ratings based on projected stats...is it possible they are not then adjusting for how the game treats the two?