Analyzing RP

[PSUColonel]

This is becoming a serious challenge for me, as many of the RP are very similar in abilities. It's like trying to put a puzzle together when building a bullpen. (this is a great thing BTW as it mimics real life)

My question is does OOTP have a stat that measures how a RP performs? ERA is not good, as it doesn't charge the pitcher for inherited runners scored...is there a stat like IR-A, or SIERA?

[alphington]

Originally Posted by PSUColonel

This is becoming a serious challenge for me, as many of the RP are very similar in abilities. It's like trying to put a puzzle together when building a bullpen. (this is a great thing BTW as it mimics real life)

My question is does OOTP have a stat that measures how a RP performs? ERA is not good, as it doesn't charge the pitcher for inherited runners scored...is there a stat like IR-A, or SIERA?

I have been using SIERA and it seems to correlate pretty well with overall performance as far as I can see. Small sample sizes of IP is a perennial issue I suppose with RP evaluation so I tend to look at the last year as well or even further back. I read somewhere that SIERA scores for RP assessment are slightly different (lower) to SP. Here's the scale I use for RPs (sorry cannot remember the source):

2.5 elite
2.85 v good
3.35 above average
3.5 normal
3.8 below average
4.1 poor
4.6 v poor

[MathBandit]

SIERA was added this year (though I think it was either sporer or pfholden that noticed some weirdness with it and wondered if it was a 'real' SIERA - which would be more accurate than FIP - or a reverse-engineered one that wouldn't be). IRS% is also in the game (if you mean number of inherited runners from other guys a pitcher allows to score), but I don't know if the inverse is there (number of inherited runners from a pitcher that others guys allow to score).

Honestly in general I feel like one of the biggest shortfalls with OOTP's current engine (which is still better than the competition, but not perfect) is a lack of process-based stats to help isolate performance signal from noise. I end up having to rely on K%-BB% pretty heavily for pitchers.

[PSUColonel]

I did notice IRS%, which combined with SIERA could be a good way to evaluate RP.

[Anonymous Ghost]

I look at Shutdowns versus Meltdowns (see Expanded Pitching Statistics 2).

Also, this is not exactly what you're asking, but there is a way to evaluate yourself as a manager using WPA and pLI, both of which the game calculates.

First, let's look at it from a player's perspective. We need to start with the definition of "Clutch," which is how much better or worse a player does in high leverage situations than he would have done in a context neutral environment. We can define it mathematically as:

Clutch_pa=WPA/pLI-WPA/LI_pa

where pa indicates the plate appearance (i.e., Clutch and LI vary by playe appearance). Some other quick definitions:

• WPA:=Win Probability Added
• LI_i:=leverage index of the situation
• pLI:=average leverage index of the player

The idea here is that in a neutral leverage plate appearance (i.e., pLI=LI=1), then Clutch is 0. Note that LI (and pLI) are not normally distributed: about 10% of plate appearances have LI>2 while 60% have LI<1. Also note that it's possible for LI<0 and pLI<0, in which case Clutch can be negative. However, WPA is always bounded by (0, 1).

Further note that you need to adjust WPA by LI and pLI to account for the fact that a player has no direct control over the situation that he faces himself in.

A player's Clutch_season is simply the sum across all PA. Unfortunately, the game doesn't calculate that for us.

However, we can evaluate a manager using just the weighted average of a team's WPA and pLI.

Let's start with a proposition that performance in high leverage situations is due to either Clutch or Manager, as in the manager making the right call with which RP to use. So then:

WPA-WPA/LI=Clutch+Manager

We can do a little algebraic manipulation and the definition of Clutch to solve for Manager:

Manager=WPA-WPA/LI-Clutch
Manager=WPA-WPA/LI-(WPA/pLI-WPA/L)
Manager=WPA-WPA/pLI

So if you wanted to grade your usage of RP, you simply calculate WPA-WPA/pLI for all of your RP and then find the TBF-weighted average. The easiest way to get these data is to export to CSV and go into the file "players_career_pitching_stats.csv."

Manager tends to be approximately (-1, 5), with a median of about 1.5 and 1.75 mean. So if your Manager stat is above 1.75, you're doing a better than average job managing your RP while anything below that means you have some work to do.

[PSUColonel]

Yes, I am using shutdowns vs Meltdowns also....as far as I can tell, these, SIERA and IRS% seem to be the best measurements of a RP with traffic on the bases.

[pgjocki]

Something I've seen that does mimic real life is that a reliever can be lights out one year then complete garbage the next. Which helps me not sign guys to multi-year contracts and I almost never pay a reliever top end money.

I do like the new stats and one thing is to look at how they fit within your league and ignore the real life application. Find the Top 10 Guys in your league and see what their SIERA, WHIP, and FIP is and see if it correlates.

[Cod]

Originally Posted by pgjocki

Something I've seen that does mimic real life is that a reliever can be lights out one year then complete garbage the next.

This is why I don't think SIERA is the right number to use for evaluating players year-to-year. SIERA does a great job of summarizing career numbers, but I've found it can be misleading when evaluating younger pitchers or those with less playing time.

I noticed in The 2023 Bill James Handbook (authored by SIS), they make no mention of specific advanced stats for relief pitchers. They focus on the following categories for relief pitchers: usage, inherited runners, saves, relief results. Each category then has multiple numbers below it. For example, usage has relief appearances, early entry, consecutive days, long appearances, and leverage index. I called out the usage category because in real life, usage is a huge piece of the puzzle for building a bullpen year-to-year. I'm not sure whether or not that translates to the game.

I can share the other stats for other categories as well if it's something y'all are interested in.

[Anonymous Ghost]

I don't believe OOTP produces it, but really you should use xFIP rather than FIP to evaluate RP given their limited samples in a single season. That is, HR/FB doesn't stabilize until ~400 FB, which is of course far more than any RP records.
https://library.fangraphs.com/principles/sample-size/

[PSUColonel]

I think (in OOTP) a good place to start is shutdowns...from there, I would say I like IR-S%. To me, a decent score here is 30 % or lower. Any higher, and you have a RP that may not have a high ERA (worthless for RP) but it shows he just can't putout fires. I have always like roundball pitchers since I play in Philadelphia (Citizens Bank Park) but I am beginning to think power type pitchers are the way to go. They are basically strikeout artists and rarely allow runners to advance due to this, and simply help you out of bad situations. I realize you aren't likely to find many roundball power pitchers (if any) but if they can't make contact, they can't drive in runs with traffic on the bases. Obviously movement is equally important as stuff with pitchers like this.

[PSUColonel]

What about individual pitches? Are certain individual pitches (or combination of pitches) more effective for a RP? Is a fastball/slider combo something to be covveted for example?

[PSUColonel]

I went on YouTube and found a video by SGT Mushroom outlining how individual pitches rank in OOTP. Here is an outline of what was discussed:


4 Pitch Pitchers:

Tier A

1. Slider
2. Knuckle Curve
3. Change-UP

Tier B

1. Forkball
2. Knuckleball
3. Screwball
4. Curveball

Tier C

1. Splitter
2. Sinker
3. Fastball

Tier D

1. Cutter

3 Pitch Pitchers (Starter)

Tier A

1. Splitter
2. Changeup
3. Slider

Tier B

1. Sinker
2. Knuckle Curveball
3. Curveball
4. Fastball

Tier C

1. Forkball
2. Screwball

Tier D

1. Cutter


3 Pitch Pitchers (Relievers)

Tier A

1. Knuckleball
2. Knuckle Curveball
3. Changeup
4. Slider

Tier B

1. Spliter
2. Curveball
3. Fastball

Tier C

1. Sinker
2. Forkball
3. Screwball

Tier D

1. Cutter

2 Pitch Pitchers

Tier A

1. Splitter
2. Knuckle Curve
3. Slider

Tier B

1. Changeup
2. Curveball
3. Fastball

Tier C

1. Sinker
2. Cutter

Tier F

1. Screwball
2. Forkball

[MathBandit]

I'd be careful taking anything from that source as gospel unless you've checked the underlying maths and the methods used to deduce it. When I first became aware of his popularity I saw two of his theories spread as gospel on Reddit (importance of Catcher Ability, and his re-ranking of the importance of various Infield Fielding ratings), and both were absolute bunk and full of holes that made the conclusion less than useless.

[PSUColonel]

Originally Posted by MathBandit

I'd be careful taking anything from that source as gospel unless you've checked the underlying maths and the methods used to deduce it. When I first became aware of his popularity I saw two of his theories spread as gospel on Reddit (importance of Catcher Ability, and his re-ranking of the importance of various Infield Fielding ratings), and both were absolute bunk and full of holes that made the conclusion less than useless.

OK thanks. I have to admit...putting a consistent bullpen together is very much a challenge (as it should be), so I am looking for any edge I can get.

[MathBandit]

Originally Posted by PSUColonel

OK thanks. I have to admit...putting a consistent bullpen together is very much a challenge (as it should be), so I am looking for any edge I can get.

I do think it's kind of interesting (in a good way) that regardless of the era you play in, based on the way the engine works, we're basically locked in the pre-Statcast era where you have to base everything on results-stats due to a lack of any process-stats - which by definition will make things foggier and harder to accurately assess.

[Pelican]

Am I wrong in using basic OOTP pitch ratings to staff a bullpen? I like a mixture of guys, some with with great stuff, great movement, or great control, even if they are lacking in one or two of the other attributes. My closer is typically the guy with the best of all [or at least all three] worlds. Setup guys need to have good control. OOTP uses “stopper”, which I like, and that dude is typically very high stuff, perhaps a risk on control. I then like “specialists” who are high movement guys. I’ve learned that they can be good antidotes to sluggers and home runs by the opposition. [Since I don’t and won’t use the three-batter minimum, I can bring these guys in to face one hitter.] And of course I need a long guy slash emergency SP, and a middle relief guy who can give me some innings, typically in a losing cause.

My ask is, am I misguided in basing this largely on pitch ratings, thus confidence in my scouting (high accuracy), as well as most recent year stats? [In evaluating stats, I will be guided by this thread into far more detail and secondary metrics, which is fine.]

The other thing I look for is RP who have a high hold number. I have found that this directly correlates to an ability to pick off the occasional runner, or at least prevent that extra base on a hit. Nothing better than the RP who comes on into a jam, and promptly picks off a runner. [I wish the game produced more pickoffs at 2B and 3B. For a RHP, those are more natural throws over, and, in particular situations, or counts, those pickoffs do occasionally happen IRL - and they can be devastating. I wonder if, in the new world of artificially-enlarged bases and pitch counts and limits on moves to the bases by the pitcher, runners will be more daring, and more susceptible to pickoffs?]

[Charlie Hough]

I don't think there is anything better for evaluating relievers and other pitchers than their actual stuff, movement, control, pitch ratings, groundball/flyball tendency, and velocity. If you have good scouting accuracy and rely on these indicators, you're always going to get great results.

Sabermetrics are just the output that measures and verifies the impact of these core indicators in OOTP.

If you find a reliever who has good ratings in all three core categories, good pitch ratings across all of his pitches, and he's a groundball specialist with good velocity, he's an ideal choice. If he has at least three pitches and he throws them all well, and he has some decent stamina, that's also helpful.

As soon as you start drifting into guys with weaknesses in their ratings or pitch ratings, you're going to have problems. If they're also flyball pitchers or don't have great velocity, that can make things worse. If they have significant limitations where they're too one-dimensional, can't throw all their pitches well, or don't have the stamina to pitch around good hitters and pitch through a longer inning without getting tired, that's another issue.

All it takes is one weak rating or wrong tendency in the wrong place, and you can end up having consistent problems with a reliever. I've seen it countless times.

As soon as you acquire relievers who have better ratings and tendencies in all the right places, these problems stop or become much more rare. In my experience, it's as simple as that.

[MathBandit]

I don't agree. I tend not to really look at most Ratings in general as anything more than a sanity-check in OOTP, but particularly for relievers I mostly throw them out the window entirely since basically every reliever is the same. Stamina is probably the one rating I do look at and find crucial for my bullpen, but everything else (Stuff, Movement, Control, etc) I don't really care as long as it's between say 40 and 75. Any lower and they might struggle (but also might not!) so it's worth keeping an eye on them, and anything higher and they might be elite (but are also probably not worth keeping).

[Charlie Hough]

Originally Posted by MathBandit

I don't agree. I tend not to really look at most Ratings in general as anything more than a sanity-check in OOTP, but particularly for relievers I mostly throw them out the window entirely since basically every reliever is the same.

Wow. The ratings and league totals work together to drive the statistical outcomes in the sim, so I can't imagine overlooking or ignoring ratings. Relievers are definitely not the same, and that's demonstrably false according to every measure, whether it's their actual ratings or their performance and statistical output from those drivers.

But I wish you the best with your approach.

[MathBandit]

Originally Posted by Charlie Hough

Wow. The ratings and league totals work together to drive the statistical outcomes in the sim, so I can't imagine overlooking or ignoring ratings. Relievers are definitely not the same, and that's demonstrably false according to every measure, whether it's their actual ratings or their performance and statistical output from those drivers.

But I wish you the best with your approach.

You can imagine or not imagine what you want. The fact is the game doesn't give us actual ratings to look at, which is why between the biased clouded view my Scout has (ratings) or being able to see and evaluate talent myself (stats), I will always lean on the stats and use ratings only as an error-check.

I also didn't say all relievers *perform* the same, but from a ratings perspective they do all tend to *look* about the same. I've attached a screenshot of the Pitcher Ratings (plus what I assume are the relevant other ratings) screen from my current offline sim (Feb 2023). I sure as heck wouldn't be able to decide on a single thing from looking just at that.