That thing about 1 gallon taking nearly as long as 5 gallon is a myth.
Sure, a 1 hour boil is going to take an hour, no getting around that. But actually bringing a brew from mash temps to a boil, you'll never get faster than a 1 gallon batch. I can hardly get a dunk sparge done by the time the wort is boiling.
Likewise cleaning is a lot simpler and faster. Less surface area, simpler gear, it's very easy. You can bottle a batch in 20 mins.
1 gallon brewing is made up of lots of bite size tasks, which are very easy to fit into your day. No, it's definitely not 5 times faster than a 5 gallon brew, but it's also not "basically the same" amount of time either as a lot of people seem to imply.
At the end of the day you need to find the balance that works for you.
One thing I'm gonna throw out there is partial mash brewing. All the advantages of a smaller brewday but more beer. IMO a well made partial mash brew is no different to an all-grain brew. The ingredient costs are quite a bit higher though.