It probably does add to your lag time, the yeast have to replicate further, this leads to more esters as they form this will reproducing also the yeast may have less numbers and therefore be more strained and strained yeast produce more fusel alcohols (that is the theory). I guess if you manage to have enough nutrients and O2 underpitching only draw back would be the higher level of esters from the increased amount of reproduction?
Until we can interview the yeast we can't be certain of what goes on in our fermenters as amateurs we are, so I look to the pros and beer science says give'm a set pitching rate, I will, until I see good science to say that it is a bad idea.
Since switching to liquid yeast I have had two stuck brews (when I first started) and that was without starters and dodgy yeast from LHBS. Since then I have cultured my own yeast and pitched according to recommended rates and not had a single problem plus to confidence in knowing my yeast is alive healthy and ready to do it job makes me not want to change my process. Most of my brews finish fermenting in 72hrs and then they clear for another week then bottled, my latest a pumpkin ale OG 1.064 when down to 1.016 in 56hrs.
In the end it is what works for you, I think if you have a process that works for you, then stick with it. I like the science side of playing with my yeast, my wife does call me an obsessed science geek, but it makes good beer.
Last minute addition I just saw the OP add another post, to address that issue the reasons I have heard about not doing a starter with dry yeast and my understanding is, it is cheap (cheaper to by another packet than make a starter) and very healthy when dried out. So when rehydrated they are ready to work for you vs liquid yeast can get a bit knocked about and loose viability in transit (time, temp light etc) so a starter increases numbers and confirms viability.
Clem