for me, it's all just about being ready for the next step and never standing there doing nothing at all during the day.
roughly: 20 minutes to heat up water, 60 minutes to mash, 10 minutes to sparge, 15 minutes to get boiling, one hour of boil, 20 minutes to chill.
That's 3 hours and 5 minutes. half an hour for cleanup at the end, and I'm at roughly 3:30.
I even manage to keg the month old batch while I'm mashing and rack the two week old batch while I'm boiling. I weigh and mill my grain while water is heating at the start. I don't do anything in advance.
Granted, brewing like this isn't super leisurely. I'm a whirlwind of activity for those 3 to 4 hours, but I still enjoy it. If I have friends over while I brew, it ends up taking me a hell of a lot longer because I am standing around bullsh!tting and not cranking through things.