Testing the Scouts - Preliminary Results

[Kiko1313]

With a lot of talk about the scouts being useless, I decided to take it upon myself to actually test how accurate they are. Below was my setup.

-All ratings listed below are on the 1 to 250 scale
-4 Teams in the league, so far I've only scouted my own organization (53 batters, 48 pitchers including minors - 3 minor levels).
-Made all players the same age (27)
-Made all players right handed batters and throw right handed
-Adjusted major league days to 344 and pro years to 2 and options used to 0
-For batters, all batting ratings, defensive ratings and speed ratings were set to 90 and potential ratings to 110
-For pitchers, all pitching ratings, defensive ratings, endurance and hold were set to 90 and potential ratings to 110
-Turned off injuries and financials
-I forgot to turn off player development (if you can - I forgot to even look), but since scouting takes less than a month, true values changed very little if at all (I didn't see any individual true rating change by more than 2 after the scouting was finished)
-Show all ratings on a scale of 1-100 and allow ratings over 100 to be shown

Assuming the scale is really 1 to 200, with ratings in the 201-250 range would actually be for ratings that would be above the max, I considered 100 to be average. Therefore I chose 90 for current ratings to make them slightly below average and their potential of 110 to be slightly above average. I suppose I could have just used 100 for everything, but decided to do it this way. I might consider doing another test using either very high or very low ratings.

I used all 6 scouts and edited their ratings such that for a given scout, all ratings are exactly the same. On the 1-200 scale for scouts I used the following (I only edited scouting ratings, didn't touch manager or doctor ratings - I also changed overall roster strategy bars to be in the center for each category):

Scout 1 - 200 across the board
Scout 2 - 170 across the board
Scout 3 - 140 across the board
Scout 4 - 110 across the board
Scout 5 - 80 across the board
Scout 6 - 50 across the board

I'm guessing some of the things above wouldn't make a difference like making everyone right handed, but I wanted as much as possible to be the same for these guys so we could account for scouting differences only and not have to worry about other possible things contributing to the variability.

This is the starting post. I'm going to add posts for different types of results.

[CHCfan]

Looking forward to it! My main grip with scouts is the complexity (albeit much improved this year) -- but accuracy/usefullness are a good concern, too. Thanks for doing all the work!

[Kiko1313]

These results are for batters only, and scouted using QUICK SCOUT for current ratings (this is important to note - these results are going to vary a lot more than a full scout). Keep in mind that current ratings should be close to 45 (entered 90 on the 1-250 scale with 200 actually translating to 100 on the 1-100 scale) and potential ratings close to 55.

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

Contact - overall average 44.86 (minimum = 43.31, maximum = 45.81)
Gap - overall average 46.47 (minimum = 45.17, maximum = 47.48)
Power - overall average 45.71 (minimum = 44.98, maximum = 46.46)
Eye - overall average 45.26 (minimum = 44.79, maximum = 45.88)
Avoid K - overall average 45.29 (minimum = 44.33, maximum = 46.69)

These look promising. At least the averages are close to what they should be regardless of how good or bad a scout is.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Contact - 5.60, 5.52, 6.90, 6.36, 9.31, 10.14
Gap - 6.77, 5.38, 6.36, 6.52, 9.60, 10.73
Power - 6.36, 7.25, 7.83, 6.88, 10.60, 8.88
Eye - 5.35, 6.89, 6.63, 7.82, 11.31, 11.00
Avoid K - 4.81, 6.19, 6.68, 7.69, 10.79, 10.43
All categories combined - 5.78, 6.25, 6.87, 7.10, 10.32, 10.28

For the most part, the standard deviations do increase as the scouts get worse. The thing that seems most apparent to me is that the first 4 scouts appear to be similar (still increasing but not huge differences), but then you get to Scouts 5 and 6 and there is a large increase in all categories. Their ratings are 80 and 50, respectively. It seems as if once you fall below 100 in your scouting abilities you are really bad and it's not necessarily a linear trend.

I'm sure most of you are familiar with interpreting standard deviations (SD), here is a general rule to keep in mind, ..., 68% of the data will be within 1 SD and 95% of the data will be within 2 SD's assuming a normal distribution (bell curve).

For example, looking at all categories combined, when Scout 1 rates a player with a true rating of 45, approximately 68% of the time he'll rate him between 39 and 51 (rounding his SD to 6) and 95% of the time he'll rate him between 33 and 57.

These may seem like large ranges, especially for a scout with the highest possible ratings, but most of the time you'll be within 12 points on a 100 point scale. This seems reasonable to me for a quick scout. I think it could be a little smaller, but I still wouldn't want it to be perfect by any means if you're using scouts.

The way I look at it, if you play fantasy sports at all, how close are you to predicting what a guy is going to do during a season? For the most part you'll probably do alright if you have a good bit of experience doing this, but there are enough booms and busts to make your predictions quite variable. Even experts don't see these things coming a lot of times (which is what Scout 1 would be considered).

Even if you had a slightly above average scout as in the case of Scout 4 (110 ratings) you're looking at 68% between 38 and 52 and 95% between 31 and 59. But when you get to a horrible scout you're going to have 68% between 35 and 55 and 95% between 25 and 65. So be aware when you have a really bad scout working for you.

Again, keep in mind these are quick scout results. I will be doing full scout results at some point (not sure about getting them done tonight - hopefully by Sunday - I'm busy all day tomorrow)

[Kiko1313]

These results are for batters only, and scouted using QUICK SCOUT. These are for potential ratings (previous post was current ratings).

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

Contact - overall average 59.04 (minimum = 56.42, maximum = 60.83)
Gap - overall average 58.08 (minimum = 54.79, maximum = 61.52)
Power - overall average 56.40 (minimum = 55.02, maximum = 59.21)
Eye - overall average 57.18 (minimum = 55.46, maximum = 58.37)
Avoid K - overall average 56.33 (minimum = 54.42, maximum = 58.12)

These look like they're all a little on the high side since they're all 1 to 4 points higher than they should be. These are potential ratings so maybe this is the main cause.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Contact - 11.19, 11.33, 11.36, 16.63, 17.08
Gap - 11.96, 11.20, 9.92, 11.79, 17.03, 17.73
Power - 12.16, 14.93, 9.65, 11.26, 14.32, 15.66
Eye - 10.02, 13.91, 10.69, 11.42, 16.24, 19.33
Avoid K - 10.28, 12.04, 9.63, 11.57, 15.81, 16.80
All categories combined - 11.13, 12.75, 10.28, 11.49, 15.99, 17.37

Comparing these to the current ratings, these are a decent amount larger as we would expect. Similarly to the current ratings, it appears that Scouts 1 to 4 are relatively similar and then there is a large jump when you get to scouts 5 and 6.

I don't have much else to say about these. I made most of my points about variability in the previous post. I am glad they're more variable than the current ratings at least though.

[Kiko1313]

These results are for batters only, and scouted using QUICK SCOUT. These are for fielding ratings. I am only going to show the results for infield ratings since it appears that outfield ratings are very similar excepct that the standard deviations are slightly higher for the outfield (probably less than half a point for each category). Catcher ability and arm were also similar to the infield results.

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

IF Range - overall average 46.11 (minimum = 45.5, maximum = 47.27)
IF Arm - overall average 46.02 (minimum = 46.02, maximum = 46.02)
IF Error - overall average 45.68 (minimum = 45.13, maximum = 46.35)
TDP - overall average 45.88 (minimum = 45.46, maximum = 46.27)

These look okay. Maybe a slight shift on the high side, but nothing to complain about. However one thing to note is that every scout gave a 46 IF Arm to every player except 1 where they gave a 47. The same thing happened with OF Arm except all players had a 46. Not sure why there is basically 0 variability for Arm. I guess whatever you see for Arm from a scout you can count on that number. I wonder if this is a glitch. This doesn't happen with Cather Arm.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

IF Range - 2.81, 5.54, 3.70, 4.43, 4.96, 5.75
IF Arm - 0.14, 0.14, 0.14, 0.14, 0.14, 0.14
IF Error - 3.41, 4.69, 4.72, 3.73, 4.99, 5.74
TDP - 3.67, 6.89, 4.34, 3.78, 5.44, 5.41

I would say just okay results as far as increasing by ability. Not sure what happened with Scout 2. He must have had a bad day. He's almost as bad as Scout 6. Looks like we see a jump again when we get to Scout 5, but not to the same degree we saw with batting abilities.

[Kiko1313]

These results are for batters only, and scouted using QUICK SCOUT. These are for speed ratings.

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

Speed - overall average 45.94 (minimum = 45.94, maximum = 45.94)
Steal - overall average 46.18 (minimum = 44.65, maximum = 48.04)
Baserunning - overall average 45.50 (minimum = 44.79, maximum = 46.44)

These look good to me. Every single player received either a 45 or 46 for speed, so almost no variability. I'm fine with this. Speed is probably the easiest thing to measure in the entire game since they could just time everyone in a sprint of some sort. To have variability here I think we would need a scout rating for how well they can use a stop watch.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Speed - 0.24, 0.24, 0.24, 0.24, 0.24, 0.24
Steal - 5.17, 5.76, 6.44, 6.13, 7.10, 10.31
Baserunning - 7.03, 8.09, 5.41, 6.46, 10.99, 8.21

Once again similar results as the batting categories I'd say. I like that baserunning is more variable than stealing. I would think that would be more difficult to assess in real life. I think these seem reasonable.

[Kiko1313]

That's all I have for now. On the to do list:

-Pitcher ratings
-Pitcher potential ratings
-Pitcher fielding, endurance and hold ratings
-All of the batter stuff already posted and pitcher stuff above I will redo using a full scout instead of quick scout.

I have an all day volleyball tournament tomorrow and birthday party at night, so I won't be able to post anything tomorrow. What I already posted should hold you over until Sunday hopefully.

Feel free to post any thoughts. If you'd like the results summarized another way, give me suggestions.

[sfgiants88]

Thanks for taking the time to do this. Quite enlightening.

[Kiko1313]

These results are for pitchers only, and scouted using QUICK SCOUT. These are for current ratings.

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

Stuff - overall average 45.84 (minimum = 44.21, maximum = 47.46)
Movement - overall average 46.22 (minimum = 44.56, maximum = 49.40)
Control - overall average 45.02 (minimum = 44.29, maximum = 45.81)
All categories combined - overall average 45.69 (minimum = 44.92, maximum = 46.28)

Looks good to me.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Stuff - 3.85, 5.31, 7.97, 7.56, 13.96, 13.26
Movement - 5.92, 5.43, 7.47, 8.82, 10.03, 14.10
Control - 4.94, 5.98, 5.96, 7.33, 11.17, 12.55

Pretty similar trends as the batters - increasing and then a jump at Scout 5.

[Kiko1313]

These results are for pitchers only, and scouted using QUICK SCOUT. These are for potential ratings.

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

Stuff - overall average 58.62 (minimum = 55.23, maximum = 62.94)
Movement - overall average 57.71 (minimum = 55.29, maximum = 60.88)
Control - overall average 57.14 (minimum = 54.04, maximum = 60.29)
All categories combined - overall average 57.82 (minimum = 54.85, maximum = 60.26)

Like we saw with the batters potential ratings, they are a little on the high side.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Stuff - 4.79, 7.46, 9.38, 9.48, 13.27, 12.44
Movement - 6.97, 6.51, 6.29, 9.82, 13.00, 13.50
Control - 6.53, 5.48, 8.97, 9.11, 14.31, 14.87

Looks like the variability for potential is only slightly higher than the variability for the current rankings. There was a pretty big difference in the variability for the batters. Perhaps it's easier to predict how good a pitcher is going to be compared to a batter. I have no idea, just throwing that thought out there. Anyone have a feeling on this?

[Kiko1313]

These results are for pitchers only, and scouted using QUICK SCOUT. These are for endurance and hold ratings. Both categories were set to 100 on the 1-250 scale (so 50 on the 1-100 scale).

When looking at the averages only for each scout we have the following

overall average is the average of all the scouts' averages combined minimum is the lowest average any single scout had
maximum is the highest average any single scout had

Endurance - overall average 50.43 (minimum = 48.33, maximum = 52.31)
Hold - overall average 51.12 (minimum = 50.88, maximum = 51.17)

Only a tiny shift on the high side for Hold. Definitely nothing to complain about.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Endurance - 3.47, 4.19, 6.03, 7.40, 10.29, 13.09
Hold - 0.95, 0.95, 0.95, 0.95, 0.95, 0.39

Endurance looks okay, but basically no variability for Hold.

[Kiko1313]

Results after this point are all for FULL SCOUTING reports.

I will not be reporting results for the means as I did before since there is not really any differences compared to the quick scouting. Current ratings are right on and potential ratings are a little on the high side. The more interesting thing to look at is the variability so I'll only be reporting that.

[Kiko1313]

These results are for batters only, and scouted using FULL SCOUT for current and potential ratings.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Contact - 1.41, 1.44, 3.00, 3.59, 5.56, 7.27
Gap - 1.40, 1.52, 3.54, 4.55, 5.84, 10.83
Power - 1.15, 1.68, 3.35, 4.64, 6.82, 8.83
Eye - 1.49, 1.69, 3.66, 5.00, 5.17, 10.22
Avoid K - 1.41, 1.51, 3.34, 4.47, 6.79, 7.86
All categories combined - 1.40, 1.62, 3.38, 4.50, 6.13, 9.11
<!-- / message --><!-- edit note -->
Pot. Contact - 1.99, 2.18, 7.25, 9.09, 13.24, 14.08
Pot. Gap - 2.11, 2.48, 8.58, 9.63, 12.23,18.51
Pot. Power - 1.85, 2.68, 7.89, 10.27, 12.68, 16.12
Pot. Eye - 1.80, 3.21, 7.84, 10.16, 11.41, 19.73
Pot. Avoid K - 1.83, 2.19, 7.60, 10.10, 13.32, 17.28
Pot. All categories combined - 2.29, 2.94, 7.91, 9.87, 12.77, 17.29

I think the basic take home message from the above numbers is that the better the scout the more it's worth making a full scout instead of a quick scout. If you have a bad scout, it appears that it's hardly worth sending them on a full scout, especially when considering the potential ratings. For Scouts 4, 5 and 6, there is only a slight improvement compared to quick scouting. But for the better scouts, there is quite a bit of improvement in their ability to rate the players.<!-- / message -->

[Mets Man]

Very interesting findings indeed, keep up the good work.

I'm curious, from your analysis, does this mean that when scouts are looking at potential ratings, they are never underrating the players? (always overrating). That's what I interpreted from seeing your results. The only time they underrate a player is for current ratings, am I reading this correctly?

[Mets Man]

Just to add, I interpreted that because when looking at your minimums and maximums for each potential ratings, none of them fall below 45. I would assume that you put every player rated a 45 (90 out of 200) for potential ratings just as you did for the current ratings, correct?

[Kiko1313]

These results are for batters only, and scouted using FULL SCOUT for fielding and speed ratings.

Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)
IF Range - 1.02, 1.17, 2.01, 3.20, 4.74, 6.05
IF Arm - 0, 0, 0, 0, 0, 0
IF Error - 1.10, 1.12, 2.19, 3.00, 4.43, 5.03
TDP - 0.89, 1.08, 2.23, 2.66, 4.95, 5.75
OF Range - 0.88, 1.04, 2.15, 2.86, 4.31, 5.10
OF Arm - 0, 0, 0.14, 0.14, 0.14, 0.19
OF Error - 1.00, 0.97, 2.25, 2.62, 4.43, 6.27
C Arm - 0.88, 1.06, 2.01, 3.14, 4.34, 4.85
C Ability - 1.12, 0.92, 2.07, 2.91, 4.05, 5.92

Speed - 0.41, 0.41, 0.44, 0.44, 0.44, 0.44
Steal - 0.91, 1.14, 2.01, 3.26, 4.36, 4.80
Baserunning - 1.24, 1.00, 2.17, 2.96, 4.10, 5.95

Similar to the batting categories, big improvement for the better scouts and only a small improvement for the worse scouts.

[Kiko1313]

Originally Posted by Mets Man

Just to add, I interpreted that because when looking at your minimums and maximums for each potential ratings, none of them fall below 45. I would assume that you put every player rated a 45 (90 out of 200) for potential ratings just as you did for the current ratings, correct?

No, not quite correct. I used 110 out of 200 for potential which translates to 55 on the 100 point scale. But even then the scouts ratings are a little higher, like 55-60. It's only a few points higher than what they should be. If they were supposed to be 45 and we were seeing 55-60, that would not be good. Overrating by a few points isn't a huge deal to me, but still worth pointing out.

Originally Posted by Mets Man

Very interesting findings indeed, keep up the good work.

I'm curious, from your analysis, does this mean that when scouts are looking at potential ratings, they are never underrating the players? (always overrating). That's what I interpreted from seeing your results. The only time they underrate a player is for current ratings, am I reading this correctly?

For the most part this is a correct interpretation, but as I mentioned above they are overrating only by a few points on average. That doesn't mean they are never underrating a player.

Does that clear it up?

[Mets Man]

Originally Posted by Kiko1313

No, not quite correct. I used 110 out of 200 for potential which translates to 55 on the 100 point scale. But even then the scouts ratings are a little higher, like 55-60. It's only a few points higher than what they should be. If they were supposed to be 45 and we were seeing 55-60, that would not be good. Overrating by a few points isn't a huge deal to me, but still worth pointing out.



For the most part this is a correct interpretation, but as I mentioned above they are overrating only by a few points on average. That doesn't mean they are never underrating a player.

Does that clear it up?

Thanks, yes it does paint a clearer picture. So judging by what you said and the above examples you illustrated, the only potential ratings where a scout averaged underrated the actual rating was for GAP (minimum = 54.79), AVOID K (minimum = 54.42), CONTROL for pitchers (minimum = 54.04). Every other potential rating category, it seems that every scout had them overrated on average.

My guess is that since these are only averages of what a scout gave, even for the scouts that overall averaged higher potential ratings than the actual (55 potential rating), they must've had some instances where they underrated the potential ability, just that the other instances outweighed these instances to bring the overall average to over 55. So in other words, for example Scout 1 may have said that a player was 51 in GAP potential in one instant, but because of several other instances, the overall average of his GAP potential rating ended up being 56.25.

So in short, for potential ratings, the scouts tend to overrate talent more than underrate talent. However, scouts still DO underrate talent, just not all that much. For current ratings, this does not hold true, for current ratings, scouts can underrate and overrate just as equally often (roughly).

[Kiko1313]

These results are for pitchers only, and scouted using FULL SCOUT for current, potential, endurance and hold ratings.
Standard Deviations - these are listed in order of Scout 1, Scout 2, ..., Scout 6 (i.e. best to worst, see original post)

Stuff - 1.13, 1.22, 2.41, 3.15, 5.81, 6.99
Movement - 1.57, 1.44, 2.50, 3.42, 4.94, 7.30
Control - 1.37, 1.21, 2.29, 3.43, 5.47, 6.48
All categories - 1.36, 1.30, 2.39, 3.32, 5.40, 6.89

Pot. Stuff - 2.12, 4.28, 5.38, 7.14, 10.24, 9.95
Pot. Movement - 2.32, 4.57, 5.67, 7.25, 8.65, 11.18
Pot. Control - 2.33, 4.52, 6.64, 6.36, 8.64, 11.05
Pot. All categories - 2.24, 4.43, 5.89, 6.93, 9.15, 10.77

Endurance - 1.09, 1.34, 2.90, 4.39, 6.56, 7.31
Hold - 0.53, 0.53, 0.53, 0.53, 0.53, 0.53

This seems a bit different from the batting ratings. It looks as though there is a pretty solid improvement for all scouts regardless of ability. Although still the better the scout the more the improvement, but at least even for the worst scouts it's worth doing a full scout compared to a quick scout.

[Kiko1313]

Originally Posted by Mets Man

So in short, for potential ratings, the scouts tend to overrate talent more than underrate talent. However, scouts still DO underrate talent, just not all that much. For current ratings, this does not hold true, for current ratings, scouts can underrate and overrate just as equally often (roughly).

I'm glad you brought this up. I meant to break down ranges of ratings and forgot about it. Below is a summary only for my best scout (this is a little too much to post on every scout) which shows how many players were rated within a specific range. That way we can see how many players were underrated and how many were overrated and by how far (gives a better idea of the distribution). For the ratings that were supposed to be equal to 45, there seemed to be a lot that were rated 46, so to prevent showing deceiving numbers I made a range of 44-46 (and 54-56 for the ones that are supposed to be 55). If I didn't do this, a lot more players would be categorized as overrated even though they were only off by 1.

Current Ratings:

PHP Code:

Range  Con Gap Pow Eye AvK
25-29   0   1   0   0   0
30-34   2   1   4   2   1
35-39   6   5   1   6   5
40-43   6   8   3   6   8
44-46  23  24  24  22  21
47-49   5   0   6   5  10
50-54   5   9   8   9   5
55-59   6   3   6   3   3
60-64   0   2   0   0   0
 
Under  14  15   9  14  14
Over   16  14  20  17  18 


Potential Ratings

PHP Code:

Range  Con Gap Pow Eye AvK
25-29   0   0   1   0   0
30-34   1   0   2   0   1
35-39   2   1   0   2   1
40-44   1   6   2   3   4
45-49   3   2   0   3   3
50-53   3   1   5   5   4
54-56   2  22  22  20  21
57-59  21   2   3   1   8
60-64   6   5   4   5   1
65-69   3   4   2   5   4
70-74   1   4   5   5   3
75-79   6   2   4   3   2
80+     4   4   3   1   1
 
Under  10  10  10  13  13
Over   41  21  21  20  19 


So, what does this all mean?

For current ratings, it looks like there is about the same number of players being overrated as there are underrated except for power. Overall, I'd say there are slightly more being overrated than underrated, but it's pretty close.

For potential ratings, except for contact it looks like approximately twice as many players are being overrated as there are being underrated. Plus as you can see there is only 1 player in a category rated lower than 30 (25 points underrated) while there are several over 80 (25 points overrated). For contact, for some reason the range with the highest number of players is 57-59 while the others are 54-56. Not sure why that is.

As I mentioned, there are obviously players being underrated, just not as many as there being overrated (for potential). Glancing at the other categories (fielding and speed) in the same fashion, I'd say there are more players being overrated than underrated, but again, not by a lot. Plus those means are fairly in line with what they should be so I don't think that's a big deal.

This should give a little better idea of what's going on.