My Lahman Database Project

[arod23]

Originally Posted by ctorg

The most difficult part of the whole thing, to me, is doing fielding. Not that it's hard to figure out or anything, but it's really tedious because you have to do each position for each player.

You could shortcut fielding and take what Ankit has done in his career DB. This way a player is available in his rookie year for all positions he played in his career. You may have to just add a few seasons to the beginning of each player's record since his rookie year is different. The difference between your "pure" method and using his data as a shortcut would be negligible...and frankly fielding is a minor issue.

[Carlton]

Seems with your method, players might be eerily underperforming or too consistent

Take Hornsby for example...

Rajah started off STRONG...then slowly tailed down in limited duty in order to get his high BA and low cumlative stats

Ty Cobb did as well...with an average would these two ever hit .400?

Therein lies the problem...and Babe Ruth is the most notable. When he becomes a RF FT he is past the OOTP 'normal' reasoning of improving his skills, and for a number of people...Ruth never becomes the HR King.
Normalizing Ruth would do exactly that...no 60 HRs, no .390 average...no 15 triples.

And using career averages Cobb 'might' get .370 BA potential, very rare would he go above that...and more than likely later in his career, if he has one...he will decline so that his career BA would be lower than .350

Then there are guys that have always been a problem...Koufax, Carlton, Feller, Haines, Grimes...pitchers that hung on too long but had 12 good seasons...they would be footnotes. Or guys like Flick, Bo Jackson, Albert Belle, Addie Joss that never had a decline becoming better than everyone else.

Then there is one of Ankit's big problems....the weird guys who started slow and then had huge upswings...meaning you have to change their ratings when they come in and up their talent...and then hope they live up to them, if they are young enough.

Also fielding is a terrible problem in both DB's Maranville comes in with an 11 in SS range? Pie Traynor a 20? Seems the really great defensive players come in with lousy defensive range in both DB's because technically, if taking just RF into account from BBref...Rabbitt Maranville had 110 range at SS

PLUS, it is important NOT to take a players decline and just their prime into account in regards to defense...because defense really gets hit hard and declines rapidly when a player hits 35

[ctorg]

Originally Posted by Carlton

Seems with your method, players might be eerily underperforming or too consistent

Take Hornsby for example...

Rajah started off STRONG...then slowly tailed down in limited duty in order to get his high BA and low cumlative stats

Ty Cobb did as well...with an average would these two ever hit .400?

Therein lies the problem...and Babe Ruth is the most notable. When he becomes a RF FT he is past the OOTP 'normal' reasoning of improving his skills, and for a number of people...Ruth never becomes the HR King.
Normalizing Ruth would do exactly that...no 60 HRs, no .390 average...no 15 triples.

And using career averages Cobb 'might' get .370 BA potential, very rare would he go above that...and more than likely later in his career, if he has one...he will decline so that his career BA would be lower than .350

Then there are guys that have always been a problem...Koufax, Carlton, Feller, Haines, Grimes...pitchers that hung on too long but had 12 good seasons...they would be footnotes. Or guys like Flick, Bo Jackson, Albert Belle, Addie Joss that never had a decline becoming better than everyone else.

Then there is one of Ankit's big problems....the weird guys who started slow and then had huge upswings...meaning you have to change their ratings when they come in and up their talent...and then hope they live up to them, if they are young enough.

Also fielding is a terrible problem in both DB's Maranville comes in with an 11 in SS range? Pie Traynor a 20? Seems the really great defensive players come in with lousy defensive range in both DB's because technically, if taking just RF into account from BBref...Rabbitt Maranville had 110 range at SS

PLUS, it is important NOT to take a players decline and just their prime into account in regards to defense...because defense really gets hit hard and declines rapidly when a player hits 35

The thing is, those problems are inherent in OOTP as well. If you import from Lahman, his potentials are based on his career averages, not his career peaks. There won't be any difference in talent ratings between mine and Lahman's.

Fielding isn't handled well by OOTP either. It's not a problem with the databases.

[RJolley]

Out of curiousity, what are you using to edit Lahman's? Is there a program out there that handles all the lines of the csv easily? Excel craps out after 64K lines.

[ctorg]

Originally Posted by RJolley

Out of curiousity, what are you using to edit Lahman's? Is there a program out there that handles all the lines of the csv easily? Excel craps out after 64K lines.

I'm editing it using excel, but I use a text editor to split it into three separate files first. I combine them when I'm done. It's kind of a pain, but it works.

[arod23]

Microsoft Access will handle all the csv files...and the updated Lahman 5.2 is available in Access and now Excel (he has split the large files for you). If you do edit them, you will have to combine them into one csv when you are done so OOTP can read it.

[RJolley]

Hmmm, may have to grab the excel files and toy around with them. Thanks for that info.

[ctorg]

Originally Posted by arod23

Microsoft Access will handle all the csv files...and the updated Lahman 5.2 is available in Access and now Excel (he has split the large files for you). If you do edit them, you will have to combine them into one csv when you are done so OOTP can read it.

Yeah, the new Excel format is great. All the rows don't line up exactly, but it's made life much easier. I do the editing in Excel. Others find it easier to do in Access. I'm not much of a fan of Access, and I only have it at work, so I didn't want to use it.

Life keeps throwing me curveballs and sliders. I've been a lot busier than I expected this week. I've made a bit of progress, but I tend to get more of it done at work (when I'm not supposed to) than at home. I should have a chunk of time on Saturday, so I'll probably finish it up then.

[ctorg]

Okay. A bit of an update.

This whole thing is taking me longer than expected. I had a week off from school for Thanksgiving and I thought I could get things done then, but I barely got started. Now I'm back in class, and I'm very busy, so the amount of time I can put into it is limited. Additionally, I have to do it all at my job because my home computer's CD burner is broken and it doesn't have internet access (long story), meaning I can't upload it or otherwise get it off my computer without it being a big pain.

I've decided that rather than stress myself out over doing this, I'm going to just say that it will take as long as it will take and not try to impose a deadline.

I got a lot of work done on it today (because I'm at work on my day off to do some classwork), but it will probably be another few weeks before it's ready. It might end up being next week, and it might end up being next month. I can't say for sure. My ETA at this point is the weekend before Christmas.

All I can say with certainty is that I will get it done. I did it last year and the year before; I just didn't release it. This year, as soon as I announced I would release it, A bunch of things got in my way.

If you're interested, keep an eye on this thread and I'll update.

Right now, I'm about 80% of the way through working on the batters. I still have to do pitchers and fielders, and then I'll be done. I could upload each one separately (batting/pitching/fielding), but the master.csv file is pretty important for everything to work properly.

I'll keep things updated here.

[disposableheros]

take your time. im not going anywhere (at least not purposefully) and id rather have it solid than quick.

thanks for even making it publically available

[ctorg]

The master.csv and batting.csv files are done.

I still have to do pitching and fielding. Pitching is relatively simple to do, but fielding is going to take a while if I want to do it right (and I do).

I took out anyone that had both less than 15 innings pitched and less than 50 at bats for their careers. I think those were pretty conservative deletions, but I didn't want to lose too many players.

For fielding, my plan is to have every player rated for every position he played in every year that he played. It's not nearly as simple as it sounds, so that might take me some time. If you want, I can release an early version without the fielding file, possibly next week, for testing. If you'd like to test it, PM me with your e-mail address.

I'm hoping it will work right the first time through, because I'm not likely to have much of a chance to fix it once I'm done.

[ctorg]

Okay, everything but fielding is done, at least as a first draft. I haven't tested it myself yet, so it might be horribly off in a way I don't realize. I've sent it to a couple of guys for testing. Hopefully it will work okay. Fielding is the most difficult part to do, since guys could play different positions in different years and you obviously can't combine the stats for different positions together. It will take me a while to get that part done, probably a few more weeks.

If only I could put more time into it, I'd have it done in a few days. Alas, I have other responsibilities.

[TotalEnd98]

Any news? I got the urge to start a historical league today, but I'm waiting until this is done to do so.

[Malleus Dei]

Take your time, Ctorg. Some things shouldn't be rushed.

[ctorg]

Originally Posted by TotalEnd98

Any news? I got the urge to start a historical league today, but I'm waiting until this is done to do so.

I'm waiting to hear back from people who are checking it out. I've done everything except the fielding, which is the most difficult part and will take me a few weeks since I'm not doing this anything like full time. PM me if you want to try out the no-fielding-included version.

[GuamYankee]

Hi, just purchased OOTP6. Quick question about the Lahman database. Actually, 2 questions.

1. Is the Lahman database OK for recreating historical seasons 1 at a time? Is it just when you plan to go more than one season that people don't like what they get from the Lahman database?

2. Is version 5.2 compatible with OOTP 6? I'm using it, but for some reason my 1901 season keeps crashing OOTP 6 when it loads up initially. The only way I get it to work is by removing the season file from the OOTP 6 directory, starting OOTP6, and then afterwards pasting the season file back to the OOTP6 directory and loading it from within the game. Not sure what the problem is there!

[ctorg]

Originally Posted by GuamYankee

Hi, just purchased OOTP6. Quick question about the Lahman database. Actually, 2 questions.

1. Is the Lahman database OK for recreating historical seasons 1 at a time? Is it just when you plan to go more than one season that people don't like what they get from the Lahman database?

2. Is version 5.2 compatible with OOTP 6? I'm using it, but for some reason my 1901 season keeps crashing OOTP 6 when it loads up initially. The only way I get it to work is by removing the season file from the OOTP 6 directory, starting OOTP6, and then afterwards pasting the season file back to the OOTP6 directory and loading it from within the game. Not sure what the problem is there!

You should post this as the first post of a new thread, otherwise not many people will see it. I'll give you what I know.

1. The Lahman database is good for importing particular seasons. What sort of gets messed up is basically guys who only had a few at bats or innings pitched in a season but who went on to have a lot more. Their talent levels partially compensate for this, but a lot of things, like stealing ability, are apparently taken solely from single season figures.

2. Yes, it's compatible. I'm not sure what the problem is.

[ctorg]

I haven't had much time to work on the fielding yet, so I don't have an ETA.

I've been informed of a problem with the pitching.csv file. There is an extra column in it, "R". If you delete it, it works fine.

[mberner]

Could this be what ruined all the potential ratings for rookie pitchers? Or is that still a separate problem, related to a lack of career innings pitched?

[ctorg]

Originally Posted by mberner

Could this be what ruined all the potential ratings for rookie pitchers? Or is that still a separate problem, related to a lack of career innings pitched?

Could be. The extra column would have shifted all the numbers over.