Improving player evaluation (being able to calculate expected/projected performance)
 

I think this issue is what’s holding OOTP from reaching the next level right now: OOTP simply does not have a very good idea on how a player would perform ("true" value of a player; expected stats production of a player). To put it simply, OOTP does not know very well on what are the expected performance stats given certain ratings and what are the future expected performance stats given certain talents. In other words, OOTP simply does not have a very good grasp on rating to stats translation.

The benefits are several folds.

First, if CPU GM know the true value of a player, then CPU could become more cunning simply because AI sees better/has better awareness – without even making CPU GM smarter. In a way, computer GMs are made to be able to see better now. If the true value is known, then it benefits AI in the way that AI will know to what contract to give (not becoming grossly overpaying players) and know what players to keep and acquire (draft, trade, FA, waiver, etc.). This way, we can also possibly minimize sign-n-release/trade-and-release type of transactions. Computer AI will know the true weakness of the team and can find replacement talent better. “Moneyball” type computer GM would also be made possible since computer GM knows where to find comparable talent (within certain amount of confidence) for possibly less cost. For players who consistently perform over/under expected performance (ratings), computer GMs can also calculate how of a risk/gamble that they want to take by overpaying such players. They would also know how much of a risk is too big. This way, players who have great ratings but poor performance will not get grossly overpaid. Or computer may sign those players at a very low price as 'gambles'. Same thing can also be said to players with performance exceeding ratings.

Second, AI could be made to be more competitive due to the fact AI will have a better idea on whether AI's team could contend this year or not (just do a simple pyth. win projection based on expected production from players). This way, GM could switch between the "win this year" mode or the "win in X year" mode. In a way, “Dan's Campaign for a GM” type implementation could be truly realized. (Plugging in for Dan now) Another way of looking at this is that teams can look at theird own division and determine how much is required to win the division title. In a weak diviison (ala AL Central), computer GM will know that they don't need to participate in big-name FA war in order to win the title. This has the benefit of making computer GMs to be more economically conscious in terms of managing team's resources. Computer GMs can know when and where to spend and when not to spend.

Third, scouting report could be made better in which OOTP could show the expected performance to user instead of ratings. For example, instead of showing contact rating = 85 or “… has a smooth classic stroke that you see in the game’s best hitters”, you can opt to make OOTP show that “this player has the ability to hit .300-.320 when healthy”, “this player is projected to hit over .300 at major league level”, or something like that. This will be great for generating accurate (to a degree) scouting report and such.

Forth but not the least, OOTP can have virtual columnist who could make fake pre-season prediction/trade commentary/power ranking and stuff like that. (akin to PureSim's PSPN but more complex)

Entirely not being self-promoting here, but the one of the possible approach would be implementing a similar (and more complex/better) mechanism/code like my Rating Translator utility in future OOTP - an internal function that can translate rating into expected performance stats within satisfactory accuracy. My approach is advantageous in the way that rating to stats relationship is dynamically determined - tailored to each league. However, the drawback is that you willl need to 'sim ahead' in order to perform accurate rating to stats mapping. Now with OOTP7 is supposed to run much faster, I would think the above disadvantage will be minimized.

Well, this are just some thoughts after making Rating Translator utility. All feedbacks, opinions or even critiques are welcome. :)

Oh, I feel that I should add one more thing - the reason why a pre-determined mathematical formula will not be suffice to solve the player evaluation issue. (eg. pre-determined formula will be like contact 50 = 0.260 batting average or something like that). Here is what I wrote in my Rating Translator readme file:

"The translation between ratings and the expected performance is a problem that has no easy answer. The rating to stats translation needs to be done within context, not depending on pre-determined translation formulas. The actual translation depends on league total setting, rating itself and also the opposition rating. In short, pretty much every league will have its own unique rating translation formulas."

Basically, what this means is that the relationship between rating and stats is not fixed. Suppose a player has a contact rating of 50. If such player plays in a league full of great pitchers, then his expected batting average will be definitely lower than what he would've performed in a league full of crappy pitchers. You can think of competition level difference as the difference of playing in ML and AAA. A player with the same level of ability will post different number in major and minors. There are also other factors such as league total setting and others. Anyway, this post is just to demonstrate some of the complexity of the issue.

Again, please share opinions if you have any. :)

I have not used the ratings translator utility as personally I am not seeking scouting reports saying the likes of "John Basehit is likely to hit about .310." The more vague statements the game makes I actually sort of like. Just as I have taken to playing with few to no ratings, I am into the greater uncertainty.

That said, I do like the idea of something like described here, especially for the media and outside analysis possibilities. I also find appealing even if I did not utilize it the idea of being able to take teams with certain personalities (favoring high OBP, or big power, or ground ball pitchers, etc) and their scouts with their different skills and eyes and then producing more precise, though not necessarily more accurate, scouting reports.

Those things that I decribed after my frist (primary) point are all just added benefits. For people who play with rating off, it doesn't really matter either way. I also agree that keeping the vague scouting report is a good idea. I actually meant that as a possibiliy of extra info in the scouting report, not as a total replacement. Sorry that I misspoke about this.

You didn't make any comment on my primary point so I fear that I may not have got my points across. Well, I am hoping someone would comment on that. Anyway, I shall elaborate it once more here then. The thing that I am most concerned about and interested in is the potential benefit to computer AI competence. If the rest of the features don't make it in, that's fine by me - as long as computer AI is made to see better. I think OOTP really needs to address of 'quality of intel' obatined by computer AI. An analogy can be made that if a stock broker has a very good idea on when the stock price will rise and fall, then this stock broker could make killings all the time. As for it is now, OOTP AI is like a stock broker who doesn't really have a good idea on when the stock price will rise or fall. All OOTP AI could do is just guess and sometimes this results in a bizarre behaviour (sign/trade-and-release). No matter how smart this stock broker could be, he/she can only make the best decision based on the information available. If the information is crappy, then this stock broker really can not make any smart and informative decision from it. The quality of the decision-making by AI is only as good as the quality of information/intel obtained by AI - no matter how smart the AI is. Remember, "Garbage in, garbage out. (GIGO)".

Anyway, I hope I got my points across this time and not further confusing people. Thanks for reading.


p.s. It's too bad that you didn't try out Rating Translator. It is useful if you want to know the rough expected performance of a player - well, provided that you are playing with ratings on. :)

It would be nice if the AI did a better job of player evaluation, but I'm not sure this is the answer.

In terms of having tools to predict player performance...I'm guessing real GM's would like to be able to do that? I mean, if the game tells you what a player is going to do, that seems a bit like cheating. Most of the game involves using the vague scouting, stats, and ratings to make those evaluations yourself. Having the game tell you "this player will hit .320 with 23 homeruns and 85 walks" is akin to playing pacman and having an arrow pop up on the screen telling you which way to move. I like OOTP's customizability, but I can't imagine why anyone would play, even now, with the ratings set to anything but off or 1-5. If you're interested in an experience that parallels real life, that's about the level of detail you'd have to limit yourself to.

Alright, when a man starts quoting obscure programming axioms, you have to give the guy credit. :)

By the way, thanks for plugging in for me - we can all use that every now and then. I agree that this could have a rather large AI impact if handled properly. As GMO warned against, too much detail being provided through the scouting report could ruin a bit of the fun, but I don't see that happening here.

As the scouting system is redone for the next evolution of OOTP, I find myself wondering how it's going to be handled. I remember studying the concept of "strategic human resources" in my business classes back in my college days. In an odd sort of way, that's actually what we're discussing here:

Planning the business/team maneuvers based upon the views of analiysts from all levels. AI GM's already get basic "assessment reports" from their scouts and managers and plan for that season and the future based upon what they learn from them. With Jayzone's enhanced scouting system, these reports suddenly take on new life and importance, allowing for the crafting of particular team strategies.

The amount that GM's would weigh the scouting reports should also depend on how much of the season has already passed - there's a breakover point between a team's apparent potential and that same team's reality that occurs someplace between opening day and the end of the World Series. Think about it in the same vein as "the Artist Formerly Known as Henry's" scaled month-by-month player evaluation settings.

Similarly, a team's relative age should also be considered by an AI GM when viewing these reports. A simple equetion weighting total IP and PA by the respective players' ages could give valuable information regarding how to read the team's potential - an AI GM could then potentially have a good idea of when his team has peaked and know when to start moving the older players.

It's interesting, and I like it.

as for players ...

One other point I'd like to raise here is that real-life GM's actually get detailed information from scouts and managers. The proliferation of numbers, projections, and rankings is rife in the Majors.

Because of this, I think having more detail than we currently get on the scouting reports shouldn't kill the realism factor just so long as it isn't too precise or inhumanly accurate. Taking into account a player's potential, achieved ratings, and age could give some interesting results.

For example, let's take young slugger Joe Schmoe; "Joe looks like he could be a consistent .280/30/100 kind of hitter just so long as that hole in his swing isn't exploited any more than it already is. Right now, I'd also expect well over 100 k's in a year, but Schmoe's still young and there's room for improvement if he gets some good coaching and actually listens to it."

Or we could deal with aging superstar John Donuthole (I'm hungry - so sue me); "After several years, John most likely doesn't have many surprises left in him but at least there's plenty left in the tank. He can still give you a solid .280 batting average with enough power to knock out 30+ homers in a full year, but his best years are probably behind him. As pitchers everywhere know, he'll fish for some pitches and strikeout well over 100 times a year, but the threat of his bat is more than enough to keep them wary of putting in his hot zone."

To give realistic numbers, there would have to be some interaction with the league's base numbers and the ratings that would go on behind the scenes, right where it belongs. Scouts could still be wrong, and even if they're right the players could simply over- or under-acheive.

All-in-all, it's a good idea. I'm not sure with what ease it could be built into the game, but I'd be interested in finding out.

Jason,

This is the same debate as to whether using a more detailed rating is realistic or not. Well, I am not going to go into that argument as it is not the real point of this thread. The real point of this thread is to discuss how to make computer GM more competent.

However, I would still offer some counter arguments as some food for thought. You see, real-world does have access to performance projection system, although they may be not dead-on accurate. Real world GMs has access to PECOTA, ZIPS, or even in-house proprietary projection system, and others. Combing with in-house scouting report, it is not impossible to be able to predict a player's performance up to a certain accuracy (providing that the player is not hurted). So, using a more detailed rating is not without its basis of the real-world. Another thing is that rating itself is never everything. Players perform over/under ratings all the time. It's just that players are more *likely* to put up stats on the level of his ratings. You were worrying that the projection maybe to accurate that OOTP will just become a mindless excercise. Well, I happen to agree that pinpoint accuracy info should probably not be made available to human players, but this still should not prevent being able to be used by computer AI - if just to be used to make up the shortcoming in intelligence as compare to human GM. I just want to clearify that I, myself, switch between various rating scales depending on whether I want to play a fast or slow league. So I am *not* propsing more detailed rating all the way here.

The point here is that, IMO, the easiest way to make computer AI more competent is to let the AI have the abilility of more-or-less accurately predicting performance, or cheating - if you may. The way that I see it is that this is the easier solution than trying to make computer AI into a Deep Blue clone. Plus, a handicap (eg. scouting ability) could also be coded so that computer AI won't always have the the most accurate projection.

Darn, Dan. You totally beat me in posting. You even posted a longer message and in shorter time! :)

Yeah, I agree with what you have said and it seems that people have mistaken my main point in thread. My main point (probably have repeated this for the 100th time now) is to make comptuer GM more competent and competitive - by impoving how computer AI obtain player evaluation. I am just proposing a basic system shoud make it into the game, if just only available to computer GM. The further application of it is only limited by the imagination of Markus and co. Certainly, Dan has pointed out some potential advanced applications.

Well, as I have said before, one way to construct this enchance scouting system is to use a system similar to my Rating Translator. Basically, what it does it to look at what the league has done in the past, make rating to stats mapping, and then project what a player could've done given certain ratings. The mechnism is simple in the sense that the program just basically look at a table and generate the information, since the mapping is already adjusted to context (collected from what happened in real-world/league). The trouble is that stats projection is only as accurate as the mapping table (rating to stats mapping). With now that OOTP7 is supposed to sim much faster, we could even just 'simmed ahead' and obtain the most accurate rating to stats mapping.

By the way, one of my previous computer science class prof liked to use "GIGO" *a lot*. So it just kinda sticks. Well, it certainly sticks better than that other stuff that I have learned in that programming class! As for giving credit, I don't really remeber whose quote is it suppose to be.... hmm.... You see, that part does not stick either. How sad...

A problem which I'm curious about...

How does the game engine do these statistical projections? Since the statistics generated by the game are determined by a combination of a.) league totals and era settings and b.) the ratings of ALL players in the league (since they interact with each other - no player performs in a vacuum) I don't see a way for reasonable projections to be made. I'll check out the rating translator to see if I'm missing something, but I've been doing research on the OOTP engine since version 2, and I can't imagine how it could be done.

As I have mentioned before, the trick is that you can just *look back* at what happend in previous years. "History can be a great teacher.", my friend. :) (OK, I don't remeber who I am quoting or if I am quoting correctly this time....)

All those factors (league total, league setting, competition level, etc.) influencing rating to stats translation could be made directly encompassed if we can just look at what happened in the past - provided that future is similar to the past (no setting change, competition level is roughly the same, etc.). I believe this is the easiest way of doing rating translation. All the interactions between different factors - the results of simming/playing have already taken those into account. Of course, if there is a even slight change in the setting, then the mapping will not be accurate. This is why I would suggest that OOTP could opt to "sim ahead" if just to obtain the most accurate rating to stats mapping information.

p.s. There is a section (section 7) explaining how Rating Translator works in principle in the Rating Translator manual. You can take a look if you are interested.

I agree that scouting reports should be more specific, as was suggested. However, I don't have a problem with the quality of ratings translators available (although I don't use them), or with the 50 = .260 BA predetermined formula. I think they are accurate as they are.

I just looked at a player at random, in this case Rocky Colavito. Rocky has a lifetime BA of .266, so his contact rating should be about 52, for example. His scouting report should read: an above average hitter with exceptional power, but he strikes out a lot. Keep in mind, that he hit a low of .238 and a high of .303 (in seasons with over 100 games played). The translators/formulas, (as well as scouting reports), are more accurate over a career, rather than for a single season.

Ah, gotcha. I also did a little poking around the projector, it's a good idea (altho...it seems very conservative?). I doubt it would be particularly difficult to implement.

OTOH, I think I'm still missing how it would help the AI evaluate players in ways that using the available scouting wouldn't. I can think of dozens of ways to improve the AI player evaluation, so I'm not saying that it's good in its present form. :)

I suppose having some sort of value-projection system could help with the portion of AI eval that is based on stats. Ultimately, tho, I think the end result of having the AI use ratings to project stats in order to have more stats for player eval is basically the same as just eliminating the stat portion of the AI's player evaluation. I mean, if you're projecting stats from ratings in order to help the AI make better decisions based on statistics, you might as well just scrap the stat portion and make the AI's player evaluation totally ratings based, as it was prior to OOTP 6, because you're going to end up with similar results.

Now, if the AI were more aware of player progression and a player's rating/scouting history...

Edit: I think the easiest way to improve the AI, is to sit down, smack the computer's ass a bunch of times, and then teach the AI how you did it. For instance - the AI can be easily beaten by overpaying for a free agent, and then immediately trading him for someone of similar value who costs less. This could be fixed by having no-trade clauses. Another example...when a player starts to decline, they continue to decline 99.9% of the time. I regularly trade 32-35 year old players to the AI for prospects with similar immediate value (and who are, of course, within 6-18 months of being significantly better than the player I'm trading) because I recognize this. So teach the AI that. Fixing the AI really shouldn't be as difficult as it seems to be.

That's fine and dandy, unless you're playing a league that hits .300 as a league. A 50 rating = .260 in a league where a.) 50 is the average rating for hitters and b.) the league is normalized to hit .260. If the average hitter has a 40 hitting rating and the league hits .300, you're going to find a 50 rated hitter may hit .310 or .325.

(and what I just said isn't really relevant to Jayzone's Rating Translator, if it works as I understand it to)

^^ Exactly, I already wrote my take on this issue in post #2. That's why I said rating to stats translation is not as straightforward as you first think it would be.

And the answer is no. The above mentioned issue is not relevant/applicable to my Rating Translator as the translation formula is dynamically generated according to individual league environment, not pre-determined. This ensures the best match between ratings and stats.

Quite a few thing that I needed to address. I shall do this in address to specific comment format then.

**********************************

Glad you like the idea. It is quite easy to implement and this is why I am recommending this method here. Not quite sure/understand on the conservative part. The result is merely being league average. You will have to know that both park factor and localized competition level are not considered. Park factor is not included because I just want to do a true ability transaltion. Plus, you can add the effect of park factor pretty easily. Localized competition level (facing more tough/weak opponenets) is a complex issue, where there is really no simple & easy solution. Also, the number maybe conservative if you are using low rating scale. This is because the inherrent uncertainty in determining the exact abiltiy ratings when using lower rating scales.

I am not sure how many more ways I can continue to spin this. See my "GIGO" comment on this (post #4). I thought I have already got my points across. No matter, I shall try again below...

p.s. I don't know what "OTOH" stands for. What does it stand for anyway?

You have actually already brought up the key issue that I am trying to address in this thread. The key issue is: "How do you determine the value of a player?" The primary value of a player is run production - either in the way of helping team scoring runs or helping the team preventing runs being scored. But then, how can a computer GM know how many runs can a player produce or prevent? Computer AI is basically a complex number cruncher so we need to quantify player value(run production) in numbers for the AI. This is why a performance projection is handy in solving this issue. By enabling the computer AI to have the ability to determine accurately how many runs can a player produce/prevent, computer AI can come to know the *true value* of the player. As you have demonstrate the understanding that ratings by themselves, without context, are meaningless. How else will the computer AI know how many extra HR will be produced between a power-100 and a power-50 hitter? Then how else will the computer AI know how much more money should be paid to sign the power-100 hitters without overpaying.

As I said before, having a smarter AI is great and all. However, the quality of decision made can only be as good as the quality of the information supplied/obtained. Hencce the GIGO quote. It is just like us human - we can only make informed decision *if* we are well-informed. Thus, I am proposing a way to help computer AI becoming well-informed regarding to player values.

I think you have mistaken my intention. I did *not* propose to use rating-only approach and abandoning stats all together. You wouldn't evaluate a player like that, I wouldn't evaluate a player like that, and I really don't think anyone would evaluate a player like that either. If nobody evaluates a player like that, then why should we make computer AI evaluating players like that? No reason at all. I think it would be quite foolish to go with raing-only apporach. In effect, one is blinding himself/herself and barring himself/herself from all possible available information. By the way, I think I kinda implied that ratings/stats should go hand in hand in the second paragraph in post#1.

Raings are the causes (expectation) and stats produced are the effects (reality). One can not look at one side only while overlooking the other.

In this case, the causes do not guarrantee the effects will always happen in a certain and specific way. There are randomness/luck and other possible hidden factors involved. So expectation does not always or necessarily translate into reality. That's is why computer GMs should look see ratings as what could happen and stats and what have happened. Both are valuable as computer GMs must look at stats for reasons why a player is not performing up to his ratings. Actually, Dan has touched on this quite nicely in his post (5th paragraph). Rating and stats should go hand in hand, but neither one should ever be overly relied upon.

On the side note, the thing about being entirely rating based prior to OOTP6 is not really applicable here. In the previous OOTP, computer AI also do not have a good idea on how rating would translate into performance. The computer AI was merely solely relying on ratings in player evaluation - without putting ratings into proper context. So, even if OOTP uses rating only approach (not saying it ever should; it's not the right direction), then it also won't evaluate players well without the ability to do performance projection.

So now, are you seeing why performance projection can help computer AI in evaluting players? Performance projection helps computer AI because it enable comptuer AI to know the exact value of a player.

Anyway, I hope that I finally got my points across this time. I thought the stock broker analogy back in post #4 was good. Ha! It seems I was gravely mistaken. Anyway, thanks for reading. :)

Where is the utility nowadays, and does it work with 6.5?

The Rating Translator can be downloaded via the link in my sig (jazzrack's CLB link). Oh, the details of the program can be found in the manual/readme file included in the package. And, Yes, it does work with OOTP 6.5.

p.s. I haven't had the chance to play OOTP2006 much yet and I am kinda curious about how the player evaluation has progressed in 2006.