I've been thinking about this a bit lately and here's my two cents:
It depends.
If you make a starter like you make your beer, in an airlocked container, you are essentially making a small batch of beer. I used to do this, pitching the entire starter into the unfermented wort. (There are recommendations of doing this when the starter is at high krauesen. Haven't seen any data as to whether this has demonstrated advantages.)
If you make a starter with oxygen exposure, for higher yeast propogation, either by oxygen additions, breathable container closures, stirplate etc., you are making oxidized beer, with less alcohol, but more yeast. In this case, I believe it makes sense to settle the yeast and decant off the "beer", in order to minimize the risk of adding any off flavors.