If the size of the starter is <= 5% of the batch size, I pitch the whole starter, and have never noticed any off flavors as a result
If the size of the starter is >= 10% of the batch size, I chill and decant the starter, adding just the slurry. I do this for lagers. To be honest, I don't know if it is necessary, but I don't want to take a chance.
If the size of the starter is > 5% of the batch size and < 10%, then I don't know what I would do, but this has never occurred for me, so I won't worry about it until it happens.