AI evaluation settings test results

[Dutch Alexander]

I've been tinkering with my fictional league the past few days bringing it into it's final form. Before fine tuning the modifiers I first have to decide what scouting settings to use. I was thinking about something like:

low scouting accuracy
a rating scale of 2-8 with only potential ratings showing
no overall ratings showing
rare scouting reports
TCR of 150
AI evaluation setting: 25/30/25/20

I chose these settings in order to create some fog to make it more challenging. My fear however is that these settings might also negatively affect the AI teams too much. So I decided to run some 50 year tests, see two attachments. I used the same file for each test starting with the exact same conditions. In these tests I foccused on the performance of the top teams in the league. In my league there is financial imbalance so some teams are rich and others poor. What I'm looking for is settings that make it difficult for myself but still allow the top teams to dominate. My fear was that with low accuracy settings all teams would bunch up too much.

In the first test I used the settings shown above. In the second test I only added current ratings and in the third test I changed scouting accuracy to high. The next three tests were the same as the first three test but with AI evaluation setting: 55/20/15/10. looking at the results I think they are inconclusive. See first attachment. I don't see significant difference in performance of the top teams.

Next I decided to run some more extreme tests. First three test with scouting accuracy basically minimized and then three with scouting accuracy maximized. The results were surprising. See second attachment. It seems that with scouting accuracy minimized the top teams actually perform better, dominating more. Looking at the top two teams in both sets of tests their combined records are:

Low accuracy:
W: 11759 L:7795 T:255 Win%:.600

High accuracy:
W:11358 L:8181 T:267 Win%:.580

Not sure why this is. Perhaps with low accuracy good scouting becomes more important and the rich teams can hire better scouts. Or perhaps because of the low accuracy all teams make bad decisions and the rich teams can absorb the bad contracts better. Don't know what it is but I think it's an interesting result.

[Pelican]

That is way too much “fog” for me at least. I am sufficiently challenged with high scouting accuracy, comprehensive scouting reports, 0-100 ratings (actual and potential). On advice of experienced users, I now use TCR of 150, so, even with all that hard data, performance diverges. And I like AI evaluation setting of 40/30/20/10. This combination still yields plenty of surprises. As GM and manager I am competitive at best (I do usually take bad or mediocre teams), even with less fog and more information. I like the idea - from your testing - that high accuracy reduces the performance of the best teams. One goal in my modern sims is to boost the performance of the worst teams, by allocating more money, raising the minimum “floor” for spending, or a hard salary cap for the rich teams.

[PSUColonel]

He is definitely creating a "fog of war" here, but it's probably not the most realistic fog of war either.

I think the 150 TCR is probably a good move. (I have been told by the developers that a higher TCR and a higher ratings weight in the AI evaluation settings result in a more challenging game) The AI evaluations I think are way too low IMO. (I recommend using att least 55 ratings...but default is probably best)

I think the ratings scale of 2-8 is probably OK, but if you want it to be more realistic I'd probably use both current and potential.

I am all for fog of war, but I like the most challenging & realistic game I can get. I realize it's not always possible to have both, but I do my best for both when and where I can.

[Dutch Alexander]

Originally Posted by PSUColonel

I think the ratings scale of 2-8 is probably OK, but if you want it to be more realistic I'd probably use both current and potential.

I'm actually thinking about the 1-5 rating scale with no current showing. This means you have to rely more on stats to get a better picture. The rating will still show you what type of player it is and I think the difference between very low and very high scouting accuracy is not quite as high as people think. When I look at the league player rankings and then switch to 100% scouting accuracy the list remains mostly the same.

I think if you play this way the ratings are still accurate enough, it's not like scrubs are shown as stars or vice versa. And looking at the league history in tests the best players still make their way to the top teams and top teams are able to dominate. I played this way for two seasons in the previous version of this league and did not notice any drawbacks. My league is a bit different than most. There is no draft and trading is not allowed during the season. AI trading in the offseason is low as well. Maybe if you have a draft the rating becomes more important.

I am a bit worried that I will run into unforeseen issues however. I know you and others have been advocating for at least 55% focus on ratings.
Still I kinda think it should work and have not yet seen any issues that say otherwise. When I look at players ratings and compare it with their stats it matches very well. The combination of the ratings and the stats give a very good picture of the players ability. I would think that an AI evaluation of 25/30/25/20 would match well with that.

Although I'm still a bit hesitant I'll probably end going with these settings anyway. I'll just have to experience for myself if it works for me and my league. If I do run into problems, then so be it. I can always start over again.

[PSUColonel]

Originally Posted by Dutch Alexander

I'm actually thinking about the 1-5 rating scale with no current showing. This means you have to rely more on stats to get a better picture. The rating will still show you what type of player it is and I think the difference between very low and very high scouting accuracy is not quite as high as people think. When I look at the league player rankings and then switch to 100% scouting accuracy the list remains mostly the same.

I think if you play this way the ratings are still accurate enough, it's not like scrubs are shown as stars or vice versa. And looking at the league history in tests the best players still make their way to the top teams and top teams are able to dominate. I played this way for two seasons in the previous version of this league and did not notice any drawbacks. My league is a bit different than most. There is no draft and trading is not allowed during the season. AI trading in the offseason is low as well. Maybe if you have a draft the rating becomes more important.

I am a bit worried that I will run into unforeseen issues however. I know you and others have been advocating for at least 55% focus on ratings.
Still I kinda think it should work and have not yet seen any issues that say otherwise. When I look at players ratings and compare it with their stats it matches very well. The combination of the ratings and the stats give a very good picture of the players ability. I would think that an AI evaluation of 25/30/25/20 would match well with that.

Although I'm still a bit hesitant I'll probably end going with these settings anyway. I'll just have to experience for myself if it works for me and my league. If I do run into problems, then so be it. I can always start over again.

To me, having a 1-5 scale is just not realistic, as no MLB organization would use anything like this. Again, I also think current and potential grades should be used in an effort for more realism. You have to remember that the grades are not always perfect, and there is already fog of war built in. (scout opinions, money spent on scouting and development etc).

There is no reason why you can't use stats as a large part of YOUR evaluation, while still understanding what the strengths and weaknesses of each player may be. This is how it's done IRL. The AI however needs to rely mostly on ratings, since this is where the stats are driven from. I have done extensive studies on this over the years, and while I understand people want things to work differently, they don't. The AI needs between 55-65 ratings in the AI evaluation.

Obviously you can do whatever you want, and I am not trying to tell people what to do, but my experience tells me the best AI evaluation is likely the default. I also highly suggest using the most difficult trading setting, and keeping the prospects/vets slider at neutral.

I think you are fine with talent evaluation at 150....but for most things I would go default )in terms of settings) because they can change from year to year. I change things more like roster rules etc... The default numbers in 2022 won't necessarily be the same in 2023 or another year/version of OOTP.

[Dutch Alexander]

Thanks PSU, I appreciate your advise. My fictional league is not an MLB type league but rather it is designed as a low level International League which is why I'm thinking about a smaller 1-5 rating scale. Teams do not have MLB type scouting departments, scouting budget is like $25,000-$50,000 a year. In your experience will such a scale have negative effects on AI performance?
And what about scouting accuracy. Would using very low have negative effects on AI performance and would you recommend a minimum setting here.

Regarding the AI evaluation setting. What in your experience is the main problem with not using at least 55 ratings in the evaluation. What type of issues would I experience regarding AI behaviour? You know, during test runs everything looks good with my settings in regard to AI team performance and player performance. Top teams are able to dominate and win up to 60% of their games and the top performing players are ussually signed by the top teams. But in these test runs I mainly look at the broad picture of the league, which looks good. But I'm not seeing the day-to-day decision making of the AI in these tests. So I might be blind to one or more glaring problems.

Again my league is not an MLB type league. It's a low level International League that functions somewhat like European soccer leagues. To summarize there are 20 organisations in this world, 12 in the top league and 8 in the lower league. Between the two the bottom team straight up relegates and first place team straight up promotes. Each organisation has two Minor League teams, a reserve team and a rookie team with an age limit of 21. Each organisation also has an International Complex which functions as a domestic youth accedemy. The majority of players enter the world in the International Complex as scouting finds. There is no draft in this world. There are three other ways in which players enter the world. Each year there is a batch of 45-50 Amateur free agents that each of the 20 organisations can sign to a bonus. Signing cap is $100,000 (average payroll in the top league is about $1,000,000 and average player salary about $30,000). Then each year there is a batch of about 15 International Free Agents that enter the league and the game also creates 80 Free Agents each year the first day of the offseason. Creation of Free Agents is disabled in the top league, it is only enabled for the lower league to prevent star and superstar being created out of nowwhere. Trading is not allowed during the season only in the offseason.

In this league to be succesful AI teams need money and good scouts (which cost money). In my tests I see the rich teams dominate. With low accuracy they even seem to be dominating a bit more. Perhaps in the particular setup of this league the AI evaluation can be a bit different compared to a MLB type league? Since drafting and in season trading is not part of the equation. Perhaps not, maybe I'm overlooking some obvious glaring unwanted AI behaviour that would smack me in the face later. Man, too many options, too many settings to choose from. Complicated stuff.

[fuzzy_patters]

Originally Posted by Dutch Alexander

I've been tinkering with my fictional league the past few days bringing it into it's final form. Before fine tuning the modifiers I first have to decide what scouting settings to use. I was thinking about something like:

low scouting accuracy
a rating scale of 2-8 with only potential ratings showing
no overall ratings showing
rare scouting reports
TCR of 150
AI evaluation setting: 25/30/25/20

I chose these settings in order to create some fog to make it more challenging. My fear however is that these settings might also negatively affect the AI teams too much. So I decided to run some 50 year tests, see two attachments. I used the same file for each test starting with the exact same conditions. In these tests I foccused on the performance of the top teams in the league. In my league there is financial imbalance so some teams are rich and others poor. What I'm looking for is settings that make it difficult for myself but still allow the top teams to dominate. My fear was that with low accuracy settings all teams would bunch up too much.

In the first test I used the settings shown above. In the second test I only added current ratings and in the third test I changed scouting accuracy to high. The next three tests were the same as the first three test but with AI evaluation setting: 55/20/15/10. looking at the results I think they are inconclusive. See first attachment. I don't see significant difference in performance of the top teams.

Next I decided to run some more extreme tests. First three test with scouting accuracy basically minimized and then three with scouting accuracy maximized. The results were surprising. See second attachment. It seems that with scouting accuracy minimized the top teams actually perform better, dominating more. Looking at the top two teams in both sets of tests their combined records are:

Low accuracy:
W: 11759 L:7795 T:255 Win%:.600

High accuracy:
W:11358 L:8181 T:267 Win%:.580

Not sure why this is. Perhaps with low accuracy good scouting becomes more important and the rich teams can hire better scouts. Or perhaps because of the low accuracy all teams make bad decisions and the rich teams can absorb the bad contracts better. Don't know what it is but I think it's an interesting result.

I could be wrong, but I wonder if the cause of the better teams doing better with higher ratings might be because players tend to have ratings that stay close to what they originally were but are a little more off than they would be with less accuracy, and the better teams have more top players.

For example, let’s say that there are two players who are power hitters who would hit in the middle of the order if there is high scouting accuracy. Let’s say the good team’s power hitter has 8 power, and the bad team’s has 6. Let’s say that dropping the scouting accuracy results in their power numbers dropping by two ratings point. The good team’s player now shows 6 power and is still starting and hitting near the middle of the order. The bad team’s player now shows 4 and might be riding the bench depending on what his other ratings are, when his actual ratings might have had him in the middle of the order.

[Dutch Alexander]

Originally Posted by fuzzy_patters

I could be wrong, but I wonder if the cause of the better teams doing better with higher ratings...

No, it's the other way round. At low scouting accuracy the good teams are performing better than they do at high accuracy. This was surprising to me and I don't know why. Perhaps at low accuracy good scouting becomes more important and the rich teams are able to hire the best scouts.

[uruguru]

My impression of scouting is that it generally gives the player too much of an advantage because when a great scout comes along, players will sign him over anything else. And once that happens, the player has gained a permanent structural advantage over the AI when it comes to evaluating players.

What I do is turn scouting completely off (to level the playing field between AI & player), turn off potentials and set TCR to maximum.

With high TCR, potentials are much less useful anyway so this setting forcesyou to evaluate players on how they have performed and how old they are in order to estimate future growth. It's more of a "Moneyball" idea that scouts really can't accurately estimate potential as much as they think.

[fuzzy_patters]

Originally Posted by Dutch Alexander

No, it's the other way round. At low scouting accuracy the good teams are performing better than they do at high accuracy. This was surprising to me and I don't know why. Perhaps at low accuracy good scouting becomes more important and the rich teams are able to hire the best scouts.

Right. That’s what I meant. I made a typo. I was trying to say that if the scout is a little bit wrong about a really good player, the team is still going to start that player. If the scout is a little bit wrong about an average player, that player might get benched. Therefore, the teams with the best players will do the best with inaccurate scouts because they will still be playing their best players.

[PSUColonel]

I actually recommend keeping scouting accuracy at normal…I think it hurts the AI quite a bit when you lower it.

[Brad K]

Originally Posted by PSUColonel

I actually recommend keeping scouting accuracy at normal…I think it hurts the AI quite a bit when you lower it.

Scouting accuracy would be a good place to connect a "Help AI" switch.