What type of beer are you thinking of making?
Depending on the beer style would/should dictate what type of pepper(s) you use and how much.
As an example I made a smoked habanero imperial porter and I used 3 habanero's soaked in rum (to help sanitize them) and let them soak in the secondary for 6 weeks.
Way too much and way too long!
I'd stick with putting them in the secondary (dry hop as you say) and taste it on a regular basis. Once it gets to the point of pepper taste/flavor/heat rack it off and keg/bottle it.
Also be aware that the flavor/taste may diminish over time but from what I can tell so far the heat does not.
Here is my Smoked Habanero Imperial Porter
Also here are a couple beers that a brewpub in my hometown brew and that info might help you as well.
Amber Chili Ale
Green Chili Beer