I can't really address the flavor... yet.
Let me clarify about clarity.

I always get chill haze in my beers for the first couple weeks at serving temp. After that they will clear up completely. However, the beers I have now that I did not secondary have more than just chill haze issues going on; They still have a lot of yeast left in suspension.
I'm sure they will
eventually clear, but since they are in kegs already, I'll be sucking yeast cake into my glass for a while before that happens. Maybe I should I have left them in primary longer than a month, but I am not going to make a change to my process that will require me to wait even longer for my yeast to settle.
So, I can't comment on whether the flavor will
eventually be better or worse, because right now these things just taste like yeast.
So, I guess my opinion is that not using a secondary (for me) seems to mean that I need to keep my beers in fermenters longer than 1 month. Using a secondary gets me where I want in just 1 month, so I am going to keep using them.