the biggest problem i run into with panning is making sure everything sounds balanced, cos as has already been mentioned, if you've got chunks of the frequency spectrum missing from one of the stereo channels, things sound a bit off...

Though saying that, i have read on here before about people panning all the sounds from one 1/2 of the spectrum to one side and the other 1/2 to the other, making a sort of 2-way system out of a normal stereo setup. not something i think i'd do myself, but it's an intersting idea...

another thing that annoys me with stereo is that when you take things of centre they seem to lose some their punch & directness, so i often find myself leaving percussive/leading elements in the middle and building up atmospheres around the edges....