Either would be fine. I'd probably rack to secondary with some cacao nibs, myself. I think JZ recipe calls for hazelnut in the bottling bucket.
It won't hurt to leave it in the primary if you can't get to it, but with that much gunk in the bottom of the primary, I'd probably use a secondary if for no other reason than to make racking to the bottling bucket a bit easier. This isn't one of those make or break decision, IMO.