When I used to use cold brewed coffee, I would dunk my coffee press in sanitizer for a few minutes, drain off the excess and fill it with coffee and boiled (and cooled) water. Covered the top with sanitized foil and never had an issue after two attempts doing that.
Lately, I've taken to simply putting 3-5oz of whole light roasted coffee beans directly in secondary. I've done this three times now without issue even without sanitizing the coffee beans before hand. It gets me a really nice coffee flavor without all the fuss.