I've added vanilla beans to the fermenter, after the primary has settled down (about 7-10 days in). You can leave your beer on the yeast cake for up to a month to allow the vanilla to settle in.
I would not suggest using anymore than 3 whole beans, maximum! I made a spiced vanilla porter once with 3 beans, cinnamon, and nutmeg before. It took about a year until it was tasty. The more you throw in, the longer it takes for the flavors to blend together.
Bottled: Udder Nonsense Milk Stout, Off Your Rocker Cider
Keg #1: Udder Nonsense Milk Stout
Keg #2: Explosive IPA
Keg #3: Explosive IPA
Keg #4: Nothing
Next On Deck: Maibock?
Brewed in 2012: 65 gallons
Brewed in 2013: 55 gallons
Brewed in 2014: 25 gallons