-yeast looked very flocculant already and when I pitched there was yeast left in the bottom of the flask (lots of it)
Did you shake up your flask before pitching? I usually decant off most of the starter wort, swirl up the cake, and pitch. Alternatively you could have swirled the whole flask up and pitched the entire starter. Sounds like you left a good amount of yeast caked to the bottom of your flask.
As things worked out, if fermentation looks like it's kicking off....I'd probably just let it go and allow more time for conditioning down the road in case you get some off-flavors from under-pitching. The beer may end up totally fine though, so just RDWHAHB for the time being.