I made a vanilla roggenweizen and used 2 vanilla beans soaked in vanilla vodka. They were split, scrapped, cut and scored with paste and all soaking for about a day, then thrown in to the fermentor. Was my first beer ever, and can't recall when I put them in, but I think I put them in right away. I let them sit in there the 2 weeks of fermentation and I barely got any vanilla notes. It may just have been too suble to notice, but I think because I threw them in at the start of fermentation, i blew off a lot of the aroma notes.
This time around, I'm going to put them in near the end of fermentation and make sure they sit about 2 weeks, though I will likely also be tasting it this time around to find that perfect note.
If anything, that's what I would recommend. Whatever you toss in, check on the flavor every few days until you get what you are looking for.