I think it depends on how long you've been in the primary, and how long you want to have the flavoring in. If you are already 3-4 weeks in primary, and want 2 weeks of flavoring, you probably want to go to secondary. If it's 2 weeks in primary, 1 week flavoring, you could keep in primary.
My understanding of the theory is that staying on the yeast too long causes off flavors. But the other theory is that in 5 gallon fermenters, there's not enough pressure on the yeast cake to cause the off flavors, so there's no worry. Who knows...
For flavoring, I suppose there's also the question of once the beans fall to the bottom, will the trub prevent the remaining flavor from escaping the beans? By that time, maybe all the flavor is extracted anyway. Again... who knows...