Well, let's clear up a couple of things:
1. If you are using recalc, the potentials rating is irrelevant. You might as well ignore it. It is useful only as a kind of magnet that draws player development in a certain direction. But player development is irrelevant if you are using recalc -- except for what might happen during a given season.
2. Recalc fixes some things, but messes up other things. For example, you could have an SS play the whole season at 3B, but he will forget everything he knew about playing 3B at the start of the next season, if he didn't play that position IRL. Turning off recalc fixes some things but messes up other things. As you have discovered. There is no solution. Part of the problem (and we all have it) is that you get focused on one or two guys, and get upset because the game doesn't manage their careers to match what happened in real life. Is the cup half empty or half full?
Edit: I realize what I said could be misleading on one point. Using recalc, the potential ratings aren't irrelevent, because they represent information that the AI uses to value players, in things like trades, etc. The AI doesn't know anything about these players and their careers the way you do, so the AI needs that info. Of course, if you also don't know anything about a specific player's career, you might like have the info as well. But the potential rating will be updated every year by recalc just as everything else is.