You're absolutely right...striving to have as little time between chilling and the beginning of fermentation does not equal fast fermentation times. The former is definitely a good thing in all cases. The latter depends on your preference.
Fermenting ales in cooler temperature ranges will definitely slow things down. Assuming a "normal" gravity ale (like below 1.060), the fermentation should be fine but slower. You will definitely get less esters, fruity flavors and yeast characteristics than you would from a hotter ferment. You'll get fewer fusel alcohols. In general, you'll get a cleaner flavor.
Now, in some cases a hotter ferment is desirable. Maybe you want a more estery beer. Many British ales have a distinct yeasty/estery character. Some Belgians get most of their flavor from esters.
Also, when brewing higher gravity ales, it may be preferrable to ferment at higher temperatures. To some extent, with brews over 1.060, you need to "charge" the ferment. In other words, get the yeast really raging so they can finish the beer. Picture the coyote running well past the edge of the cliff before actually falling down. In high gravity beers, the yeast will kill itself by making the environment so alcoholic. So a quicker ferment is better so that it actually ferments dry. Of course, yeast strain has a huge impact on whether a high gravity beer will finish or not.
Anyway, there's some stuff to think about. We often leave our secondary ales in the shed outside so they have a cooler secondary. Especially for APA or American IPA. Cheers!