Short answer: YEP.
Analogy Answer: If a busful of kindergarteners stops at a McDonalds and everyone eats and then everyone except one kid uses the bathroom - if he craps his pants once back on the bus, will the stink change if the others kids are there or not? Sure, extra bodies MIGHT dilute the scent a little...more activity/movement will cause greater circulation, but the stink will still be the same captive stink.
The same works for your yeast, the fewer there are, the more stressed they will be. The fewer nutrients and fermentables left, the more stressed they will be. Piss off your yeast and it'll start crapping things in your drink that you don't want there. ....And it's all captive in your fermentor.