For what it's worth, I've always drilled my magnet holes by hand. I have a dremel, but it's too easy for something to go wrong with a power tool when you're working with tiny handheld parts- whether it's destroying the part or taking a chunk out of your hand. Pin vices are cheap, readily available, and do the trick just fine. And you have full control over how deep you're cutting. No chance for melting plastic. For larger bits, I have a small 3-jaw chuck designed for hex attachment that I just use by hand.
Same as if you do use the dremel, work yourself up in size slowly if drilling big holes.