There really is no set answer to this; it's a matter of preference. The game adjusts itself for the number of minor league levels, from zero (reserve roster only) on up.
Personally, I prefer four because I like tracking the guys as they move up (or down) through levels; reserve roster only bores me. Four fits nicely into the minor league screen, given my laptop screen resolution. At work, where I am now, I just noticed that six fits into that window.
I'm glad I play the game on my laptop in this regard. Four is good (AAA, AA, A, Rookie) and six, as shown below, would be too much to handle. Five would bother me because it would leave an open gap in the display.
So again, it's a matter of preference.