No, in fact, your beer will probably be nice and smooth. Steeping grains have virtually all of the diastatic power roasted out of them. This means they just need to be steeped/soaked to extract their remaining sugars and flavors. No mashing required, although a lot of all grain brewers will still mash them. There are some brewers that make a tea from their grains in just this way by letting them soak for 48 hours at two quarts per pound of grain (and doubling the grain amounts) in cold water to make a tea.
The reason for this is that at warm temperatures (170 f and above), roasted grains can leach tannins and astringency compounds into your wort. If you've ever had a bitter stout that tastes like a teabag, this is the astringency. To avoid it, some brewers have found success in making the cold grain tea.
Before you add the tea to your primary, it's a good idea to bring it back to 160 * f for ten minutes or so just to sanitize it, then cool it off and add it to the primary.
