I just leave it where it is because I'm too lazy to move it. Also, not sure if it was mentioned, but if primary fermentation is over (read: you are at your target FG and it is stable) just pitch the hops into primary. No need for secondary.
I am a big proponent of doing whatever you want. When I hear people say that secondary is a source of infection, I get a little defensive (no bigger risk than primary or a bottling bucket, assuming good sanitation processes). However, since you mentioned infection risk, I will say that I believe adding dry hops is less of an infection risk than secondary as hops are a natural preservative. And to your original question, most definitely worth it!