Are you stirring the wort while it is cooling? Before I made my CFC, my ice baths used to take about 20 min tops. Stir it, but ensure that you do not aerate the hot wort by splashing it.
As to your question on the time affecting your wort, I just read somewhere about an Aussie way of letting the wort naturally cool overnight. In fact the guy who tested it said it might be good up to a week or maybe two as he was using the last bit of the wort to whip up a "real starter" as he called it. I am sure that leaving your wort to cool overtime before pitching your yeasties all depends on the big "S". Sanitation. Good luck and try and brewgle that aussie cooling technique.
