please explain normalization

baseballbob · 03-16-2003, 12:44 AM

Well, I've been working on addressing the issue of league and park normalizations for stats that I import into the game via Lahman.

It seems to me that by normalizing the stats to the season those players played in, that the numerical ratings of the players would better reflect their talents. Is this the case?

Well, I'm still going to keep experimenting, simming, and all that until I find something that works for me.

But, amidst all of that, I have to stop and wonder: Can anyone with knowledge of how the game was designed give an in depth explanation of what exactly is going on with the in-game normalization league totals and whether or not the ability scores of players are based on the raw statistical data or on stats that have been normalized to the season?

It seems to me that for something as fundamental to league design as this that a user of the ootp5 software shouldn't really have to go about guessing and calculating and surmising in order to find an understanding of how those numbers are used by the game engine.

I have read the current rule book that i downloaded, i have read the FAQ, and I have read the posts at this site...all in the hopes of coming to a precise understanding of how this game handles normalization. The best I have found so far, after more than a week of efforts, is that no one really is completely sure, there have just been numerical number crunching ad hoc methods and or guesstimate setting methods that have been developed to fit the data ranges to one's expectations.

Does my question, as I have written it so far, make sense?
I would like to know specifically the following:

1) Upon import of player X from Lahman 5.0, are his talent ratings based soley upon raw unadjusted data or are they based on some normalizing calculation based on the era from which he has been imported?

2) What does the button "Recalculate league totals" do exactly?
In the FAQ on historical season creation...for which I've read the normalization section about 5 times in the hopes of squeezing out some deeper mathematical/algorithmic understanding of the nature of the impact of the choices I will make is on the set up of the game....in that FAQ it says that clicking the button "recalculate league totals" only messes things up and that button should not be used. ???
No, let me restate that: ???
???

Actually, I think that if I could get those 2 questions answered that I'd be one happy camper. The game is great. But I'm currently feeling a bit of indignant frustration that there is such an opaque obscurity regarding the normalization factors of a historical league.

I don't think this is a lack of effort or intelligence on my part that i still want to know the answers to these questions directly rather than having to study, analyze and test hypotheses about. Of course, in the absence of available answers I'm sure I'll do just that...but I'm anxious to get my league set up and so its hard to be patient long enough to embark on a numerical study that isn't as much fundamental to the game of baseball as it is quixotic to the machinations of ootp5 player valuance.

I thank you in advance for any help you might be able to provide.

Kevin

baseballbob · 03-16-2003, 09:49 PM

bump....

the essential questions are: (for historical data)

1. unnormalized data...does it produce player talent ratings based on the raw (non-normalized data? or does it normalize somehow before calculating the player talent ratings?)

2. what exactly is happening mathematically in recalculating league totals?

3. are any factors at all other than the raw data taken into account when deriving player talent ratings?

4. just how much do such talent ratings affect the results of the game? Are there instances where such ratings would somehow 'override' the fundamental core stats of the player...or are those ratings there simply for simplified (and more vague which is useful for 'scout' mode) referrence?

A lot of this really started to occur to me when, upon importing the '78 season, the scouting report for Reggie Jackson said something about him not really being likely to amount to much of anything but that he has, though not great, some power.
It makes me wonder if the reports and ratings are calculated based on the non-normalized data...and, of course, HR totals were much lower on average that year than we are used to in these recent years.

5. In general, is there any hope of ever having a posted write-up of how these things such as 'league totals, etc.' were made and what they do and how they impact the game?
Is this simply a trade secret? Or does no one truly know how to give a good explanation of what it is other than: well, use it, tweak it, and trust that it'll work then...

Is this asking too much for such an explanation of something so fundamental to the game itself?

EDIT: Sorry about my tone in the last paragraph that I wrote before. OOTP5 and its 'crew' are awesome and I'm glad to have discovered them...I was simply growing frustrated with my attempts to understand...yet, getting no feedback on earlier posts even though I find it unlikely that this post has been un-read by 'those in the know' (or those who know 'those in the know')
I know that OOTP5 is great with offering support and help and delivering a great product that very much takes into mind what its fan base wants...and, to me, that is a very commendable trait in a game company...
but, I have also grown somewhat perplexed as to what is actually going on with a lack of response to my posed question.

Please, if I have written my post in a way that is not understandable and clarification is necessary, please let me know. Also, if my post is considered to have obvious answers that I should know by now...please let me know that too. Or, if its a case that I HAVE in fact, posted an unreasonable question, please let me know. Or, if the answer is confidential due to 'trade secret' considerations...please press "7"...If, instead, those who have read this post who are 'in the know' don't have time to respond to such an indepth question at this time, please press "8",....If, however, this is a case where the answers need to be first sought out before an answer can be given, please press "9"...or to return to the main menu, please press "0" at any time.

William4192 · 03-17-2003, 04:06 PM

I would really like to see some answers to this. I have many of the same questions.

kq76 · 04-15-2003, 11:11 PM

bump

baseballbob, I love your writing skills.

As for your questions, I'm sorry, but I have no idea. I'm currently in the process of redoing my settings calculator and I've had some of the same questions, but I haven't been able to answer them yet.

Vida Blue · 04-16-2003, 10:39 AM

I have been asking the same questions since OOTP2 and have never gotten or found the answers I am looking for. I usually play historical careers so some statistical accuracy is a must. I am not asking that the game exactly mirror baseball historically, but to come pretty damn close. I love this game but I spend more time backing up and rerunning seasons to get where I want to be historically. I was hoping by version 5 that the historical aspect of the game would be better than it is. I will continue to tweak the league totals and hope that one day I won't have to read "Nancy Drew and the Mystery of the League Totals" anymore.

rogmax11 · 04-16-2003, 11:36 AM

I've gotta chime in on this too. My creds date from OOTP2 and this issue, or seemingly endless clones of this issue, have popped up with great regularity.

And yet, confusion reigneth still.

I gotta acknowledge BBBob for summing up the major themes that have surfaced here, and for doing so in entertaining fashion. It is one thing to cavil, quite another to cavil in style.

So how about it Scott? Or Steve? Or Markus? TigerFan? There had to be some kind of reasoning for the mystery button, some design ethos and intention behind the ostensible function of this "feature". How about shedding a little insight on us?

In honor of Maury Wills' son, I just have to add:

Bump

ctorg · 04-16-2003, 12:27 PM

I can answer question 1 because I've tested it. The game does not normalize data for years. If you import Nap Lajoie in 1901, for instance, when he led the league with 14 home runs, you will not see anything like a great home run rating. I believe he gets a 4, which would be consistent with modern day standards, not 1901.

In other words, it uses the raw, unadjusted data.

In the past, people (including myself) have made normalized versions of the database. The only problem with these is that there are certain players like Babe Ruth who are so far ahead of all the other players in a particular category that the normalization looks quite absurd. I believe that if you normalize Ruth to today's numbers he ends up averaging over 100 homers a year for his career.

I prefer to normalize the database to some degree because I think it does a better job of calculating salaries and rating players.

rogmax11 · 04-16-2003, 12:34 PM

I agree ctorg, but how do you normalize the DB short of editing each player by hand? And what do you use as the norm standard?

ctorg · 04-16-2003, 01:04 PM

Quote:

Originally posted by rogmax11
I agree ctorg, but how do you normalize the DB short of editing each player by hand? And what do you use as the norm standard?

I did it for v4 by doing it year-by-year rather than player-by-player. First I had to divide the batting file into two separate files because Excel can't open the entire thing.

Then I got a listing of the year-by-year league totals for the major leagues (you could do AL and NL separately, but I chose full majors because it took less time and was reasonably as accurate).

I then normalized all players to current year stats by applying a formula that multiplied each of their stats by the percentage difference between the year they played and the current year.

I would create another database and copy the player names, then create formulas relating to their numbers in the original database.

I don't have the stats for yearly totals here with me, but here's the basic idea...

Say you're imorting a guy from Year A, when the league totals averaged a home run every 100 at bats, and you want to normalize him to Year B, where the league averaged a home run every 50 at bats. You simply multiply everyone in the league's home runs by 2 (or 200%).

The problem with normalizing stats to today is that you end up multiplying home runs in some years by as much as 11, which makes a 7-hr hitter a 77-hr hitter. I never did find a good way to get around that.

MTGian · 04-16-2003, 03:30 PM

Greetings,

I am probably asking for trouble here, but I created normalized databases for OOTP4. They are for Lahman v4.5. For more information, you might be able to do a search for normalized databases and look up the information that I posted in the past. Also, you could try searching for my name (MTGian).

I have the following versions.

1) Looks exactly the same as Lahman database, but each entry is normalized for that year. Approx Size: 4 MB.

2) Starting with #1, I then averaged performances over the players entire career. So, you end up with career average data instead of year-by-year data. Approx Size: 1 MB

3) Starting with #2, I culled out players who didn't play very much and delayed players debuts if they played very little in their first season or two. Approx Size: 0.5MB

Send me an e-mail, and I will reply with the version that you request. I do not check this bulletin board regularly, so if you want it you MUST e-mail me. My address is mtg0954@rochester.rr.com. Repeat, I will not send you the files unless you e-mail me first, and I can simply reply, then attach, then send.

Hope this helps.

MTGian

P.S. I did not do the normalization by hand. I used Microsoft Access. You could do it with Excel, but it is much harder.

CubsFan · 04-17-2003, 10:49 AM

Quote:

Originally posted by baseballbob
I have read the current rule book that i downloaded, i have read the FAQ, and I have read the posts at this site...all in the hopes of coming to a precise understanding.... The best I have found so far ... is that no one really is completely sure....

I would like to know specifically the following:
...

2) What does the button "Recalculate league totals" do exactly?
In the FAQ on historical season creation...for which I've read the normalization section about 5 times in the hopes of squeezing out some deeper mathematical/algorithmic understanding of the nature of the impact of the choices I will make is on the set up of the game....in that FAQ it says that clicking the button "recalculate league totals" only messes things up and that button should not be used. ???
No, let me restate that: ???
???
...

Bumping for the answer to this enduring mystery. Stating the question another way (as I have seen someone else do), what did the designers of the game intend for the purpose of the "Recalculate League Totals" button to be?

And don't just answer, "to recalculate the league totals."

(of course, now I'm asking for that...)

ctorg · 04-17-2003, 10:55 AM

Quote:

Originally posted by CubsFan
Bumping for the answer to this enduring mystery. Stating the question another way (as I have seen someone else do), what did the designers of the game intend for the purpose of the "Recalculate League Totals" button to be?

And don't just answer, "to recalculate the league totals."
(of course, now I'm asking for that...)

Why, to recalculate league totals, of course.

Actually, I think - and I could be off on this - but I think the point is to have the game look over the whole league, see all the players' ratings, and set the league totals so that the results of the league will be realistic by modern standards.

I use the new historical button for my leagues and edit the era stats so that I can get results I want, but I never really liked using the regular button.

CubsFan · 04-17-2003, 10:57 AM

Thanks ctorg. I noticed that a thorough answer has been written in another thread. Must have missed it earlier.

03-17-2003, 04:06 PM	#3
William4192 All Star Starter Join Date: Mar 2002 Location: Florida Posts: 1,625	I would really like to see some answers to this. I have many of the same questions. __________________ What is this about Rhode Island? It is neither Road or Island. Come check out the PBA. http://www.pioneerbaseballassociation.com Or email at pbacommish@pioneerbaseballassociation.com

04-15-2003, 11:11 PM	#4
kq76 Global Moderator Join Date: Nov 2002 Posts: 12,035	bump baseballbob, I love your writing skills. As for your questions, I'm sorry, but I have no idea. I'm currently in the process of redoing my settings calculator and I've had some of the same questions, but I haven't been able to answer them yet. __________________ My OOTP Wishlist \| My FAQ List OOTP Wiki \| Your Recommended Team Nicknames, By City (A Crowdsourced Project) For Beta/Devs: Full screen (1920x1080)

04-16-2003, 10:39 AM	#5
Vida Blue Major Leagues Join Date: Dec 2001 Location: Greater Reich of Ontario Posts: 324	I have been asking the same questions since OOTP2 and have never gotten or found the answers I am looking for. I usually play historical careers so some statistical accuracy is a must. I am not asking that the game exactly mirror baseball historically, but to come pretty damn close. I love this game but I spend more time backing up and rerunning seasons to get where I want to be historically. I was hoping by version 5 that the historical aspect of the game would be better than it is. I will continue to tweak the league totals and hope that one day I won't have to read "Nancy Drew and the Mystery of the League Totals" anymore. __________________ "The only way I'm going to get a Gold Glove is with a can of spray paint." - Reggie Jackson

04-16-2003, 12:27 PM	#7
ctorg Global Moderator Join Date: Nov 2002 Location: Queens, NY Posts: 9,848	I can answer question 1 because I've tested it. The game does not normalize data for years. If you import Nap Lajoie in 1901, for instance, when he led the league with 14 home runs, you will not see anything like a great home run rating. I believe he gets a 4, which would be consistent with modern day standards, not 1901. In other words, it uses the raw, unadjusted data. In the past, people (including myself) have made normalized versions of the database. The only problem with these is that there are certain players like Babe Ruth who are so far ahead of all the other players in a particular category that the normalization looks quite absurd. I believe that if you normalize Ruth to today's numbers he ends up averaging over 100 homers a year for his career. I prefer to normalize the database to some degree because I think it does a better job of calculating salaries and rating players. __________________ My music "When the trees blow back and forth, that's what makes the wind." - Steven Wright Fjord emena pancreas thorax fornicate marmalade morpheme proteolysis smaxa cabana offal srue vitriol grope hallelujah lentils

03-16-2003, 12:44 AM	#1
baseballbob Minors (Double A) Join Date: Feb 2003 Posts: 101	please explain normalization Well, I've been working on addressing the issue of league and park normalizations for stats that I import into the game via Lahman. It seems to me that by normalizing the stats to the season those players played in, that the numerical ratings of the players would better reflect their talents. Is this the case? Well, I'm still going to keep experimenting, simming, and all that until I find something that works for me. But, amidst all of that, I have to stop and wonder: Can anyone with knowledge of how the game was designed give an in depth explanation of what exactly is going on with the in-game normalization league totals and whether or not the ability scores of players are based on the raw statistical data or on stats that have been normalized to the season? It seems to me that for something as fundamental to league design as this that a user of the ootp5 software shouldn't really have to go about guessing and calculating and surmising in order to find an understanding of how those numbers are used by the game engine. I have read the current rule book that i downloaded, i have read the FAQ, and I have read the posts at this site...all in the hopes of coming to a precise understanding of how this game handles normalization. The best I have found so far, after more than a week of efforts, is that no one really is completely sure, there have just been numerical number crunching ad hoc methods and or guesstimate setting methods that have been developed to fit the data ranges to one's expectations. Does my question, as I have written it so far, make sense? I would like to know specifically the following: 1) Upon import of player X from Lahman 5.0, are his talent ratings based soley upon raw unadjusted data or are they based on some normalizing calculation based on the era from which he has been imported? 2) What does the button "Recalculate league totals" do exactly? In the FAQ on historical season creation...for which I've read the normalization section about 5 times in the hopes of squeezing out some deeper mathematical/algorithmic understanding of the nature of the impact of the choices I will make is on the set up of the game....in that FAQ it says that clicking the button "recalculate league totals" only messes things up and that button should not be used. ??? No, let me restate that: ??? ??? Actually, I think that if I could get those 2 questions answered that I'd be one happy camper. The game is great. But I'm currently feeling a bit of indignant frustration that there is such an opaque obscurity regarding the normalization factors of a historical league. I don't think this is a lack of effort or intelligence on my part that i still want to know the answers to these questions directly rather than having to study, analyze and test hypotheses about. Of course, in the absence of available answers I'm sure I'll do just that...but I'm anxious to get my league set up and so its hard to be patient long enough to embark on a numerical study that isn't as much fundamental to the game of baseball as it is quixotic to the machinations of ootp5 player valuance. I thank you in advance for any help you might be able to provide. Kevin

03-16-2003, 09:49 PM	#2
baseballbob Minors (Double A) Join Date: Feb 2003 Posts: 101	bump.... the essential questions are: (for historical data) 1. unnormalized data...does it produce player talent ratings based on the raw (non-normalized data? or does it normalize somehow before calculating the player talent ratings?) 2. what exactly is happening mathematically in recalculating league totals? 3. are any factors at all other than the raw data taken into account when deriving player talent ratings? 4. just how much do such talent ratings affect the results of the game? Are there instances where such ratings would somehow 'override' the fundamental core stats of the player...or are those ratings there simply for simplified (and more vague which is useful for 'scout' mode) referrence? A lot of this really started to occur to me when, upon importing the '78 season, the scouting report for Reggie Jackson said something about him not really being likely to amount to much of anything but that he has, though not great, some power. It makes me wonder if the reports and ratings are calculated based on the non-normalized data...and, of course, HR totals were much lower on average that year than we are used to in these recent years. 5. In general, is there any hope of ever having a posted write-up of how these things such as 'league totals, etc.' were made and what they do and how they impact the game? Is this simply a trade secret? Or does no one truly know how to give a good explanation of what it is other than: well, use it, tweak it, and trust that it'll work then... Is this asking too much for such an explanation of something so fundamental to the game itself? EDIT: Sorry about my tone in the last paragraph that I wrote before. OOTP5 and its 'crew' are awesome and I'm glad to have discovered them...I was simply growing frustrated with my attempts to understand...yet, getting no feedback on earlier posts even though I find it unlikely that this post has been un-read by 'those in the know' (or those who know 'those in the know') I know that OOTP5 is great with offering support and help and delivering a great product that very much takes into mind what its fan base wants...and, to me, that is a very commendable trait in a game company... but, I have also grown somewhat perplexed as to what is actually going on with a lack of response to my posed question. Please, if I have written my post in a way that is not understandable and clarification is necessary, please let me know. Also, if my post is considered to have obvious answers that I should know by now...please let me know that too. Or, if its a case that I HAVE in fact, posted an unreasonable question, please let me know. Or, if the answer is confidential due to 'trade secret' considerations...please press "7"...If, instead, those who have read this post who are 'in the know' don't have time to respond to such an indepth question at this time, please press "8",....If, however, this is a case where the answers need to be first sought out before an answer can be given, please press "9"...or to return to the main menu, please press "0" at any time. Last edited by baseballbob; 03-16-2003 at 11:12 PM.

04-16-2003, 11:36 AM	#6
rogmax11 All Star Starter Join Date: Dec 2002 Posts: 1,320	I've gotta chime in on this too. My creds date from OOTP2 and this issue, or seemingly endless clones of this issue, have popped up with great regularity. And yet, confusion reigneth still. I gotta acknowledge BBBob for summing up the major themes that have surfaced here, and for doing so in entertaining fashion. It is one thing to cavil, quite another to cavil in style. So how about it Scott? Or Steve? Or Markus? TigerFan? There had to be some kind of reasoning for the mystery button, some design ethos and intention behind the ostensible function of this "feature". How about shedding a little insight on us? In honor of Maury Wills' son, I just have to add: Bump

04-16-2003, 12:34 PM	#8
rogmax11 All Star Starter Join Date: Dec 2002 Posts: 1,320	I agree ctorg, but how do you normalize the DB short of editing each player by hand? And what do you use as the norm standard?

04-16-2003, 03:30 PM	#10
MTGian Minors (Triple A) Join Date: Jul 2002 Posts: 233	Greetings, I am probably asking for trouble here, but I created normalized databases for OOTP4. They are for Lahman v4.5. For more information, you might be able to do a search for normalized databases and look up the information that I posted in the past. Also, you could try searching for my name (MTGian). I have the following versions. 1) Looks exactly the same as Lahman database, but each entry is normalized for that year. Approx Size: 4 MB. 2) Starting with #1, I then averaged performances over the players entire career. So, you end up with career average data instead of year-by-year data. Approx Size: 1 MB 3) Starting with #2, I culled out players who didn't play very much and delayed players debuts if they played very little in their first season or two. Approx Size: 0.5MB Send me an e-mail, and I will reply with the version that you request. I do not check this bulletin board regularly, so if you want it you MUST e-mail me. My address is mtg0954@rochester.rr.com. Repeat, I will not send you the files unless you e-mail me first, and I can simply reply, then attach, then send. Hope this helps. MTGian P.S. I did not do the normalization by hand. I used Microsoft Access. You could do it with Excel, but it is much harder.

04-17-2003, 10:57 AM	#13
CubsFan Minors (Double A) Join Date: Apr 2002 Posts: 150	Thanks ctorg. I noticed that a thorough answer has been written in another thread. Must have missed it earlier.