Yeah, couple of things. As said above by The Glue, hops are generally an inhospitable host for the bad things that could end up in your beer. Second, usually by the time you are adding your dry hops your beer is fermented and the alcohol will help protect your beer from infection. (This isn't to say that a fermented beer is completely immune to infection...)
That isn't to say some good sanitation isn't in order. Sanitizing both the bag and the secondary carboy are good procedures (and any other equipment that may touch your beer), but beyond that there isn't much reason to worry. People on here claim to just throw handfuls of hops into their fermenter without ill effect.
One thing you should worry about though is oxidation. If you are racking to a secondary make sure you have as little headspace as possible and if you are set up to do so, purge it with CO2 first.