You are using samples for your drums? If that's the case, my guess would be that the samples have some noise in them, and the compressor is bringing that noise up to a noticeable level.

Try inserting a gate before the compressor, and set it so it chops the drums to give a cleaner sound. If that doesn't help, take a look at the samples you're using and maybe find some cleaner ones.

Hope that helps.