Is FIP actually fielding indepedent? I ran some tests

[Argonaut]

A short while ago I posted a thread on pitch types, and how they had more of an effect on OOTP than I thought. Here's the thread. On page two I actually ranked the different pitches.

Well I'm back at it again with more testing. This time I was wondering if FIP was actually fielding independent, or if it is a flawed statistic.

What got me thinking is that Nolan Ryan in my dynasty is striking out guys a lot less than he should be, and I wagered it was because my defense was too good. If my outfielder makes a great play for the third out, then Ryan doesn't get to strikeout the next batter to pad his stats, right? Makes sense.

Remember that the components of FIP are K's, BB's, and HR's. So if Nolan Ryan is striking out guys less, then does that mean his FIP, and therefore his WAR, is worse off? But then I also thought that if defense is good, then innings don't go on as long and the pitcher theoretically doesn't give up as many walks or homers either. So how does it all wash out in the end? I decided to do some testing.

For this test I used the 1995 Atlanta Braves, a model choice for pitching. I was watching Smoltz (strikeout artist), Glavine (finesse pitcher), and Maddux (all-time great). I also looked at Wohlers (lights-out closer with average control) for a reliever perspective.

I ran three simulations of 10,000 games against the San Diego Padres -- one with the vanilla Braves, one with a super defense, and one with a very bad defense. All numbers were "within reason" so it wasn't like maxed and minimum ratings across the board. I didn't touch catcher ratings, because they are really the biggest wildcard in OOTP and I didn't want to muck around with that variable.

Sidenote: Tony Gwynn batted .428 against the bad defense!

What I found was interesting, and I've attached the results in a photo. Conclusions:

• FIP really holds up for pitchers no matter what kind of defense is behind you, at least in OOTP. You'll see all the Braves pitchers had FIP numbers across the board that were very close.
• As expected, strikeouts were higher with a bad defense. But this was almost directly offset with a tiny bit higher HR/9 rate, and a noticeably higher BB/9 rate.
• If anything when it comes to a bad defense, a super elite pitcher like Maddux gets more of a chance to prove his dominance, while a finesse pitcher like Glavine will get killed by BABIP over and over because he doesn't strike guys out.
• In OOTP I would say that in the long run, FIP is definitely a measure of how good your pitcher is, so long as he is not a knuckleballer (see my other thread).

Now, I'm sure you're wondering the answer to the real question: After 30,000 simulations of the Braves against the Padres, did Greg Maddux ever strike out Tony Gwynn? Well, I didn't save the box scores, so I guess we'll never know.

[Jamee999]

Did you track batters faced? It might be informative to look at the K% and BB% of the pitchers.

[Argonaut]

Originally Posted by Jamee999

Did you track batters faced? It might be informative to look at the K% and BB% of the pitchers.

The simulation module doesn't seem to track batters faced. It actually doesn't track FIP either, I had to calculate that manually (without HBP which also isn't tracked, but that's probably a rounding error).

[CBeisbol]

First, thanks for doing these. It's always good and cool when someone takes the time to both study something and then tell people about it. Increasing people's knowledge is a noble endeavor.

Originally Posted by Argonaut

The simulation module doesn't seem to track batters faced. It actually doesn't track FIP either, I had to calculate that manually (without HBP which also isn't tracked, but that's probably a rounding error).

It's a shame that batter faced aren't included.

People can, though, easily see that with a worse defense K/9 is increased. Obviously, this is because less batted balls are being turned into outs, so a higher percentage of *outs* (not batters faced) are by strike out.

It is interesting that Maddux's FIP is lower with a bad defense and Glavine's is higher while Smoltz' and Wholers' remain unchanged. I'm trying to think of why this would be, but not being too successful.

Another interesting thing, is that each pitchers' WAR could be influenced by the defense, even in FIP doesn't really change. This is because the innings pitched totals are influenced by defense. Better defense=more outs=more innings pitched. Since WAR is based on innings pitched, not batters faced, better defense should lead to an increase in pitcher WAR. Which also means, that the defensive players themselves are being underrated by WAR - the pitchers are getting credit for what the defensive players did.

[NoOne]

heh, i just want to make fun of myself... i nearly wrote a book that wasn't even a tangent to the original post... i think i had a bit too much.

are you using the export stats function or opening stats in a browser and parsing?

if not, the export historical stats - picthing, batting, fielding files - should cover most anything the game displays. may need to tick a few more export files?

if so, still look through the export list... maybe something could contain it that is not selected currently. simple data link to the file will auto import etc or equivalent with a database program.

[Argonaut]

Originally Posted by NoOne

heh, i just want to make fun of myself... i nearly wrote a book that wasn't even a tangent to the original post... i think i had a bit too much.

are you using the export stats function or opening stats in a browser and parsing?

if not, the export historical stats - picthing, batting, fielding files - should cover most anything the game displays. may need to tick a few more export files?

if so, still look through the export list... maybe something could contain it that is not selected currently. simple data link to the file will auto import etc or equivalent with a database program.

I opened the data from the simulation module in my browser, yes. Is it possible to open it in Excel? I didn't see that option.

Or do you mean run actual seasons of full blown OOTP and collect the data? That would be a big time commitment, especially with changing variables. The simulation module only takes a minute or less.

[Argonaut]

Originally Posted by CBeisbol

First, thanks for doing these. It's always good and cool when someone takes the time to both study something and then tell people about it. Increasing people's knowledge is a noble endeavor.



It's a shame that batter faced aren't included.

People can, though, easily see that with a worse defense K/9 is increased. Obviously, this is because less batted balls are being turned into outs, so a higher percentage of *outs* (not batters faced) are by strike out.

It is interesting that Maddux's FIP is lower with a bad defense and Glavine's is higher while Smoltz' and Wholers' remain unchanged. I'm trying to think of why this would be, but not being too successful.

Another interesting thing, is that each pitchers' WAR could be influenced by the defense, even in FIP doesn't really change. This is because the innings pitched totals are influenced by defense. Better defense=more outs=more innings pitched. Since WAR is based on innings pitched, not batters faced, better defense should lead to an increase in pitcher WAR. Which also means, that the defensive players themselves are being underrated by WAR - the pitchers are getting credit for what the defensive players did.

Yes I agree, WAR will be higher with a good defense because starting pitchers will stay in longer. But also like you said, strikeouts will be lower. This is why I started the testing, because I was a bit disappointed Nolan Ryan is never hitting 300 K's per year.

But if overall FIP is constant, then where does the extra WAR go with a bad defense and fewer SP innings pitched? It'll go to the long and middle relievers. Wohlers wasn't a good example for this as he won't be closing out as many games with a bad defense. I should have included a middle reliever above, but I do recall them pitching more innings with the worse defense of course.

I hesitate to draw conclusions from the minor changes in FIP for Maddux and Glavine, but my hypothesis is this. Because Maddux is so elite, the best pitcher of the era, despite the innings going longer he's just not giving up more homers and walks -- at least not enough to offset his increased strikeouts. He's doing it all himself. Glavine on the other hand is letting his innings go on far too long, and isn't striking out enough guys to end things. So Maddux looks "better" with a worse defense, and vice versa for Glavine.

[RonCo]

You can determine the indirect influence of defense on FIP by simply playing with the FIP equation.

Put all a pitcher's performance figures in a per BF format, then vary BABIP, and you'll some subtle shifts in the values. So, no, technically, FIP is to 100% fielding independent.

[sabrtoothtiger]

out of curiosity how did you go about editing the good/awful defense figures on the 1-250 scale?

great work.

[Argonaut]

Originally Posted by sabrtoothtiger

out of curiosity how did you go about editing the good/awful defense figures on the 1-250 scale?

great work.

Generally it was a boost or subtraction of 30-50 points for all of range/error/arm depending on the player's ratings. It was enough to make a difference, but not so much as to be unrealistic.

Interestingly the Braves actually still won more games than the Padres with the bad defense. It came down to better pitching and more home runs (Justice/McGriff/Chipper).

[Argonaut]

EDIT: Disregard the following post's conclusions about FIP being arbitrary in some cases. There's a follow-up post below by me that offers explanation. The relevant data in this post pertains to a catcher's affect on strikeouts and walks.

NEW TESTING: FIP & CATCHERS

In doing some unrelated testing on something else, I seem to have come to the conclusion that FIP may have some element of arbitrariness, calling into question how I view FIP in OOTP.

It started by my opening the Pandora's Box that is Catcher Defense. I was looking at Ivan Rodriguez on the 1997 Texas Rangers, and simulated a bunch of games against the Milwaukee Brewers with and without his elite defense.

It was actually kind of eerie, as Texas had a mirror image record:

Normal Pudge: 5093-4907
Crippled Pudge: 4907-5093

There is some more detail in the attached results. A great catcher increases K/9 and decreases BB/9, and also definitely contributes positively to ERA and FIP.

But wait, I thought FIP was fielding independent? Well, normally it's pretty close as shown in my Braves testing. But catchers are a different story, and if their pitch framing increases K's and reduces BB's, that should be a net benefit to FIP because BABIP isn't involved. It's cool that this is in OOTP, and it reinforces my attraction to defensive catchers.

And then we get to the puzzling part... the Rangers with the normal Pudge gave up more home runs? I tested this over and over to make sure what I saw was real. It's true, having a better Catcher Defense actually increases home runs against. This doesn't make any logical sense -- if there's more strikeouts and fewer walks, then innings should be shorter and there should be fewer opportunities for home runs.

The only reasoning I can think of for the additional home runs is that there is some arbitrary invisible hand at play to keep FIP within a range. More strikeouts + fewer walks = more home runs. Having a great catcher is still a net benefit, but it looks like there's an illogical drawback.

In all of my previous testing I've come away more appreciative of OOTP, but this is the first time I've been disappointed. (EDIT: Disappointment not needed!)

[CBeisbol]

Originally Posted by Argonaut

NEW TESTING: IS FIP ARBITRARY?

In doing some unrelated testing on something else, I seem to have come to the conclusion that FIP may have some element of arbitrariness, calling into question how I view FIP in OOTP.

It started by my opening the Pandora's Box that is Catcher Defense. I was looking at Ivan Rodriguez on the 1997 Texas Rangers, and simulated a bunch of games against the Milwaukee Brewers with and without his elite defense.

It was actually kind of eerie, as Texas had a mirror image record:

Normal Pudge: 5093-4907
Crippled Pudge: 4907-5093

There is some more detail in the attached results. A great catcher increases K/9 and decreases BB/9, and also definitely contributes positively to ERA and FIP.

But wait, I thought FIP was fielding independent? Well, normally it's pretty close as shown in my Braves testing. But catchers are a different story, and if their pitch framing increases K's and reduces BB's, that should be a net benefit to FIP because BABIP isn't involved. It's cool that this is in OOTP, and it reinforces my attraction to defensive catchers.

And then we get to the puzzling part... the Rangers with the normal Pudge gave up more home runs? I tested this over and over to make sure what I saw was real. It's true, having a better Catcher Defense actually increases home runs against. This doesn't make any logical sense -- if there's more strikeouts and fewer walks, then innings should be shorter and there should be fewer opportunities for home runs.

The only reasoning I can think of for the additional home runs is that there is some arbitrary invisible hand at play to keep FIP within a range. More strikeouts + fewer walks = more home runs. The sanctity of FIP must be protected at all costs. Having a great catcher is still a net benefit, but it looks like there's an illogical drawback.

In all of my previous testing I've come away more appreciative of OOTP, but this is the first time I've been disappointed.

First, thank you, again, for doing these tests and sharing your results.

Second, this has driven me crazy because we are missing information. Specifically, batters faced.

There are problems with trying to compare things on a per inning basis since Normal Pudge faces less batters per inning (less walks, less balls in play - thus less hits).

If we equalize per inning we get
Normal Pudge: 9560 HRs, 25,304 BBs, 53,588 K's
Altered Pudge: 9381 HRs, 27,186 BBs, 50,549 K's
*I try not to use the term "crippled"

So, Normal Pudge, still less BB's, More K's and, oddly, more HR's

But, the more home runs might, actually, make sense.

Normal Pudge gave up 179 more home runs and 1882 fewer walks, while striking out 3039 more.

But, all those fewer walks means batters had *more chances* to hit home runs.

If we look at the ratios of HRs to K's and BB's we get

Normal Pudge
HR/K = .1784
HR/BB = .3778

Altered Pudge
HR/K = .1856
HR/BB = .3451

It's what we'd expect.

Normal Pudge gives up less HR/K (because he gets more K's) but more HR/BB (because he gives up less BB's). The HR could actually be the same because the denominators are changing.

Now, there could be a problem with the engine not simulating PA's and just saying there's a higher chance of a BB or a lower chance of a K but the same chance of a HR with Altered Pudge. But not accounting for the fact that BB's and K's occur later in the PA than a HR.

We could estimate BF's if you had more data - like hits, hit by pitches, double plays, etc

Otherwise it's hard to say if something's wrong or if it's ok.



EDIT: Giving more thought, I didn't factor in the increased strikeouts to remove opportunities to hit home runs.
I think, the most likely explanation is what I said earlier. The game engine doesn't simulate full plate appearances, just results. So, a good defensive catcher leads to more K's and less BB's per PA, but the same amount of HRs per plate appearance (as opposed to home runs per balls in play - that makes no sense). I don't know. We need batter faced data.

EDIT: How much did you alter Pudge? The differences are between .1 and .2 runs per 9. Which is, about, 15 runs per year (for a catcher), or a win and a half.

[Argonaut]

I'm not sure I'm following your logic on why more home runs makes sense... but that's sort of a moot point after looking at this again.

I've since done more testing, and now I'm convinced what I saw with the home run totals was just random variance. I did a few more sets of 10,000 games, and sometimes the HR totals for the 2 Pudge's were similar, and sometimes the edited Pudge gave up more home runs, sometimes fewer.

The K/BB numbers were always consistent though. I thought 10,000 games would be a big enough sample, but maybe you need more? I guess home run totals are really volatile, at least compared to other numbers.

So cancel the alarm, there's probably no invisible arbitrary hand guiding numbers towards expected FIP!

But at least I think I've answered the original question of this thread: Is FIP really fielding independent? The answer is yes, except for catchers who can affect FIP via K/BB rates.

I wager then that defensive catchers are undervalued in terms of WAR, as the extra WAR they generate will go to their lucky pitchers.

PS: I edited Pudge drastically from 218/200 Ability/Arm to 68/50. I also tried a middle ground of 118/100 which had in-between results.

[CBeisbol]

Oh man, I edited it out.

That was my original thesis, that the difference in home runs wasn't significant since it was a small difference, 200, compared to 2000 for BBs and Is. Then I got so caught up in trying to be "too cute" that I forgot about it.

��

[SkoCubs]

Originally Posted by Argonaut

EDIT: Disregard the following post's conclusions about FIP being arbitrary in some cases. There's a follow-up post below by me that offers explanation. The relevant data in this post pertains to a catcher's affect on strikeouts and walks.

NEW TESTING: IS FIP ARBITRARY?

In doing some unrelated testing on something else, I seem to have come to the conclusion that FIP may have some element of arbitrariness, calling into question how I view FIP in OOTP.

It started by my opening the Pandora's Box that is Catcher Defense. I was looking at Ivan Rodriguez on the 1997 Texas Rangers, and simulated a bunch of games against the Milwaukee Brewers with and without his elite defense.

It was actually kind of eerie, as Texas had a mirror image record:

Normal Pudge: 5093-4907
Crippled Pudge: 4907-5093

There is some more detail in the attached results. A great catcher increases K/9 and decreases BB/9, and also definitely contributes positively to ERA and FIP.

But wait, I thought FIP was fielding independent? Well, normally it's pretty close as shown in my Braves testing. But catchers are a different story, and if their pitch framing increases K's and reduces BB's, that should be a net benefit to FIP because BABIP isn't involved. It's cool that this is in OOTP, and it reinforces my attraction to defensive catchers.

And then we get to the puzzling part... the Rangers with the normal Pudge gave up more home runs? I tested this over and over to make sure what I saw was real. It's true, having a better Catcher Defense actually increases home runs against. This doesn't make any logical sense -- if there's more strikeouts and fewer walks, then innings should be shorter and there should be fewer opportunities for home runs.

The only reasoning I can think of for the additional home runs is that there is some arbitrary invisible hand at play to keep FIP within a range. More strikeouts + fewer walks = more home runs. Having a great catcher is still a net benefit, but it looks like there's an illogical drawback.

In all of my previous testing I've come away more appreciative of OOTP, but this is the first time I've been disappointed. (EDIT: Disappointment not needed!)

One of the reasons I think steal 3rd is much more successful in OOTP is because they could possibly base it off MLB stealing third successful rates. For example, IRL stealing third is usually only done when the baserunner sees something where the pitcher/catcher absolutely aren't paying attention. So there's a high success rate, same with double steal they'll only do it if they know they'll snag both bases.

Maybe more "batter must swing" counts correlates with more home runs? Something like how players like Javy Baez who doesn't walk usually has a high amount of hits?

Not sure if my logic makes sense, just spitballing.

[RonCo]

It is empiircally logical that a catcher who independently reduces a pitcher's BB rate will almost certainly increase his HR/9 rate.

[Argonaut]

Originally Posted by RonCo

It is empiircally logical that a catcher who independently reduces a pitcher's BB rate will almost certainly increase his HR/9 rate.

I don't think that's true and I can't follow the logic. Walks prolong innings. Shorter innings = fewer chances for home runs. Maybe you're right on a small scale per batter, but I can't see that being right per inning.

And strikeouts definitely shorten innings, so a catcher increasing K's should lower HR/9.

Anyway, after looking at several rounds of results I believe HR/9 isn't meaningfully affected by Catcher Defense in OOTP, and I'm happy with that.

[KBLover]

Originally Posted by Argonaut

I don't think that's true and I can't follow the logic. Walks prolong innings. Shorter innings = fewer chances for home runs. Maybe you're right on a small scale per batter, but I can't see that being right per inning.

Less walks = More PAs where a HR is a possible result?

If the game resolves BB/K/HR first (let's say in that order, check for a walk, check for contact, then if contact, check for HR), then if there's fewer walks, there's potentially more PAs that can get through to the HR, if they don't get trapped by the K.

Let's say there's 10,000 PAs and normally there's 3,000 BBs.

That leaves 7,000 PAs that could get through to HR, if they don't get caught by K's.

If there's 0 BB, then those 3,000 BBs can then "check" for K's, and if they get by that, they can become HR.

Assuming the same rate for K's, that's more PAs "checking for" HRs.

[Argonaut]

Originally Posted by KBLover

Less walks = More PAs where a HR is a possible result?

If the game resolves BB/K/HR first (let's say in that order, check for a walk, check for contact, then if contact, check for HR), then if there's fewer walks, there's potentially more PAs that can get through to the HR, if they don't get trapped by the K.

Let's say there's 10,000 PAs and normally there's 3,000 BBs.

That leaves 7,000 PAs that could get through to HR, if they don't get caught by K's.

If there's 0 BB, then those 3,000 BBs can then "check" for K's, and if they get by that, they can become HR.

Assuming the same rate for K's, that's more PAs "checking for" HRs.

I don't know, and I don't think anyone knows outside of the devs, in which order the dice are rolled so to speak. Maybe they just roll them all at once somehow?

If like you say, BBs are decided before HRs in the dice roll, then yes fewer walks could equal more homers. Per batter. But FIP is calculated per inning, and fewer walks equals shorter innings. And the more strikeouts from a good catcher would lower homers per batter, as well as per inning too.

Funny that people are sort of defending my initial results on HR/9 that were probably just random variance. But it's fun to theorize about it.

[RonCo]

With work you can ascertain that it is correct that an outside force that serves to reduce a pitcher's walk rates will increase his home run rates. It will not be big at all, but it will be.