Yep, it's a risk. But it's a matter of how big a risk.
Rinsing sanitized equipment is a risk for just this reason. (On a terminology note, 'sterile' and 'sanitized' are two very different things, and in homebrewing, pretty much nothing is ever 'sterile') This is one of the biggest reasons (aside from convenience) that homebrewers use no-rinse sanitizers such as starsan, iodophor, and many others.
If your tap water is pretty decent, you're probably relatively safe. Of course, it's safer to boil and cool it before using, but MANY homebrewers have used plain tap water to top off with without ever having an infection as a result - myself included. But that doesn't mean it's not risky. Some people use bottled water, which is probably less likely to lead to infection than most tap water, but still not perfect.
IMO, it's one of those things where I go ahead and do it but if I ever had a problem with infected batches, it would be one of the first things I would change about my brewing process to try and stop the infections.