I've used both, to varying degrees of success. You get more traditional "chocolate" flavor from cocoa powder, while the cacao nibs can bring some chocolate nose and a more complex flavor (one catch is that there's a fair amount of bitterness, which tends to compound if you use a ton of roasted malts, so cut those back accordingly).
I used 1 oz of cocoa powder in a 1 gallon batch - made a slurry with some water and poured it in with about 5 minutes left in the boil.
For the nibs, I soaked 2 oz (again, for a gallon batch) in some vodka to pull out the alcohol soluble flavors, then tossed the whole mess into secondary. It was kind of a pain to rack out, what with all the nibs getting sucked into the tubing and stuck, but it worked.
In those amounts, it didn't kill my head retention, but it was a little lower than normal.
To sum up: Cocoa powder in the boil works well to get chocolate flavor, but can be kind of one-note. Cacao nibs are much more complex and flavorful, but they don't play nicely with some other flavors. Plan accordingly.