I'm a big fan of using cold brewed coffee and adding to the secondary, after some of the alcohol has developed and could find off anything in the coffee.
I do make sure to sanitize the cold coffee brewer, but the ground coffee worries me. However, no problems yet and I've done this a few times.