I agree with you about putting the wort at risk, leaving it to sit for that long. I've had a few that ended up with a fair amount of goo in the bottom a few hours after racking, but I always pitch as soon as I can to get the yeast rolling before any nasties can move in.
If you strain, or better yet... double strain (small hole strainer sitting beneath and catching what a bigger holed one doesn't), the hops will filter out a lot of the goo anyway. I have over an inch of trub in the bottom of the primary of my current batch and it doesn't seem to affect anything.
I'd say it ain't worth the risk.