Some manager/coach test results

[frangipard]

I was curious about the impact of managers and coaches, so I've done some testing. I created a two-team league, with each team fielding 25 identical 20 year old clones named "Joe Average" (all scores set at 100) and did some testing, and a bunch of managers and coaches that were all “neutral” or “conventional” in style. Some results:


Managers matter. … in theory

When I gave one team a very good manager & bench coach combo and the other team a very bad one of each (150 vs 50 on manage players/vets/rookies), the good-manager team won an average of 90-72 (three-season sequences repeated 3x each, 9 seasons total).

The interesting thing, though, is that the value consistently rose: in their first years, the good managers won an average 86 games, suggesting that a good/bad manager can make a difference of maybe 2-3 games just by fostering a positive environment. The real value of the managers came when that positive environment helped their young players develop: by year three, the good managers were winning 103-58.

Here's the caveat though: after I did my research, I went looking at the managers in the game, and found that actual average of existing managers isn't 100, but ~120, and that the band is fairly narrow, with very few outside a 100-165 range.

Coaches matter

I set the mangers to average and made the hitting and pitching coaches good/bad. These teams won an average of 92 games, and again the gains were greater as time went on, with the well-coached teams winning 97 games by year three.

Again, though: same caveat about actual personnel in the game being not as wildly differerent as the ones in the test.


Styles don't matter that much

I set all personnel at average skill level and set one team's manager/bench coach to “smallball.” I didn't change the actual sliders (everything was at median), just the label. Those teams won 82 games. “Sabermetric” managers won 77. No clear pattern, though, and the amount of variance suggests to me that the label made no difference. Didn't bother checking Tactician, since every one of the 50 Joes was a switch-hitter.

I set managers back to conventional and set one team's hitting coaches to “power” and pitching coaches to “groundball” while keeping skill levels at 100. It had minimal impact: these teams outhomered the opposition by about 23 per year after 3 years, and were getting 2% more groundballs. I set the coaches skill level to 200, and the impact rose, but not by a lot: 31 HR and 4% … and those teams also out-batting-averaged and had more K's, too. A good coach improves everything, it seems, regardless of style.

Another caveat here: “works well with power hitters” is not the same as “helps everyone hit for power.” All of the Average Joes were 100 rating, 150 potential in everything. It's possible that style comes more into play when you have different player types, and not 25 clones.

Personnel don't change

The skill level of the coaches doesn't vary much: even after the best/worst three-year runs, I never saw any coach or manager's score in anything drop, and they never rose more than a point or in one single case, two. Personnel don't get dumber, and they only get smarter very slowly.



Conclusions:

1) There are real differences produced by coaches and managers, and while actual results will be very dependant on the talent they were working with, if you have a sizable track record on a guy it will be meaningful. That, plus the fact that personnel learn slowly, is a strong argument for picking older staff: a long track record helps separate the signal from the noise. Even with the personnel in the quickstarts, spot checking the personnel ratings vs their reputations suggest that the more experienced the coach is, the more likely it is that his reputation is accurate. If he's got 20 years of experience and still shows as “inexperienced” or “poor” the odds are that he's not a guy you want to seek out (There are definite exceptions, though ...); if he only has a few years, there's a better chance that his rep is undeserved.

2) Manager style is irrelvant, except so far as it reflects the sliders. Probably don't worry about coach styles; a good coach is a good coach, period, and will help all his players. Just look for guys who have had success.

[IsaacR]

Could it be possible that management style is relevant in reference to the players. I.e. a power heavy lineup with a smallball manager wouldn't do very well.

[Bluenoser]

How big is your sample size? How many games/seasons did you sim?

[frangipard]

Originally Posted by Bluenoser

How big is your sample size? How many games/seasons did you sim?

For everything, three three-year sequences.

[garion333]

Originally Posted by frangipard

For everything, three three-year sequences.

That seems an awful lot like a small sample size. I'd suggest not making quite so many sweeping generalizations on three years worth of data.

Did you turn injuries off? What other factors did you try and normalize?

[ThePride87]

Originally Posted by frangipard

I was curious about the impact of managers and coaches, so I've done some testing. I created a two-team league, with each team fielding 25 identical 20 year old clones named "Joe Average" (all scores set at 100) and did some testing, and a bunch of managers and coaches that were all “neutral” or “conventional” in style. Some results:


Managers matter. … in theory

When I gave one team a very good manager & bench coach combo and the other team a very bad one of each (150 vs 50 on manage players/vets/rookies), the good-manager team won an average of 90-72 (three-season sequences repeated 3x each, 9 seasons total).

The interesting thing, though, is that the value consistently rose: in their first years, the good managers won an average 86 games, suggesting that a good/bad manager can make a difference of maybe 2-3 games just by fostering a positive environment. The real value of the managers came when that positive environment helped their young players develop: by year three, the good managers were winning 103-58.

Here's the caveat though: after I did my research, I went looking at the managers in the game, and found that actual average of existing managers isn't 100, but ~120, and that the band is fairly narrow, with very few outside a 100-165 range.

Coaches matter

I set the mangers to average and made the hitting and pitching coaches good/bad. These teams won an average of 92 games, and again the gains were greater as time went on, with the well-coached teams winning 97 games by year three.

Again, though: same caveat about actual personnel in the game being not as wildly differerent as the ones in the test.


Styles don't matter that much

I set all personnel at average skill level and set one team's manager/bench coach to “smallball.” I didn't change the actual sliders (everything was at median), just the label. Those teams won 82 games. “Sabermetric” managers won 77. No clear pattern, though, and the amount of variance suggests to me that the label made no difference. Didn't bother checking Tactician, since every one of the 50 Joes was a switch-hitter.

I set managers back to conventional and set one team's hitting coaches to “power” and pitching coaches to “groundball” while keeping skill levels at 100. It had minimal impact: these teams outhomered the opposition by about 23 per year after 3 years, and were getting 2% more groundballs. I set the coaches skill level to 200, and the impact rose, but not by a lot: 31 HR and 4% … and those teams also out-batting-averaged and had more K's, too. A good coach improves everything, it seems, regardless of style.

Another caveat here: “works well with power hitters” is not the same as “helps everyone hit for power.” All of the Average Joes were 100 rating, 150 potential in everything. It's possible that style comes more into play when you have different player types, and not 25 clones.

Personnel don't change

The skill level of the coaches doesn't vary much: even after the best/worst three-year runs, I never saw any coach or manager's score in anything drop, and they never rose more than a point or in one single case, two. Personnel don't get dumber, and they only get smarter very slowly.



Conclusions:

1) There are real differences produced by coaches and managers, and while actual results will be very dependant on the talent they were working with, if you have a sizable track record on a guy it will be meaningful. That, plus the fact that personnel learn slowly, is a strong argument for picking older staff: a long track record helps separate the signal from the noise. Even with the personnel in the quickstarts, spot checking the personnel ratings vs their reputations suggest that the more experienced the coach is, the more likely it is that his reputation is accurate. If he's got 20 years of experience and still shows as “inexperienced” or “poor” the odds are that he's not a guy you want to seek out (There are definite exceptions, though ...); if he only has a few years, there's a better chance that his rep is undeserved.

2) Manager style is irrelvant, except so far as it reflects the sliders. Probably don't worry about coach styles; a good coach is a good coach, period, and will help all his players. Just look for guys who have had success.

There's so much information missing from this research, and the fact you only used 3 years.....very amateur to speak in such absolutes. You need to dig much deeper than this to know for sure what the impact is.