Any reason I can't just throw my Cacao nibs in the primary?

Thread title says it all.

Currently fermenting about an 8.5% Russian Imp. Stout (1 week in), and I want to put it on some nibs for 2-3 weeks. Any reason I can't just add the nibs straight to the primary instead of racking to my secondary? I don't like using secondary unless absolutely necessary - which is almost never for me.

So I guess the responses I'm looking for aren't about the benefits of secondary fermentation in general. I know people have differing opinions on that.

I just mean specific to the nibs - what's the downside of just using them like I would my dry hops? I plan to keep this beer on the cake about 4-5 weeks total.

Thanks.

Primary Dry nibbing may reduce the aroma of the cacao nibs depending on how long you let them sit and if your primary is in active fermentation.

I would wait to add them for another week at least - basically the way I do my dry hops. Most of the active fermentation is/will be finished.

I currently have a stout a few days in the primary and will be heading out of town for two weeks (it will have been in primary for 1 week).

I would like for this to be a chocolate cherry stout and already have lactose, cocao nibs, and cherry extract.

I wonder if I can add the lactose and cacao (boiled to sanitize) to the primary before going out of town then add another couple oz of cacao nibs extract (vodka) when I get back to up the chocolate flavor.

Has Anybody tried a strategy like this? Any advice? I've never brewed with cacao or extracts before.

I have a chocolate raspberry stout in secondary I added extract and nibs in when I transferred. If I didn't secondary it I would have waited til after the first week or so. I'm not sure how though cuz I really wanted it mixed in . Are you going to stir it all in?
Is messing with the yeast cake by mixing it into the primary the thing to do?
Thanks for asking your questions because it has me asking some mow too.
Hopefully the beer geniuses will be able to shed some light on our questions.

I ended up buying a 6G better bottle and racking to that to add the cacoa and lactose solution. I realized that I could build a pipeline by moving to secondary this time, and that's a great reason to buy another carboy! The stout is currently in bottles with another couple weeks of conditioning ahead. I do wonder if adding the cacao and lactose mixture to primary would cause oxidation??