Brewing with toffee (the real stuff, not extract flavoring)

Anybody have any experience using English toffee in a batch of homebrew? I'd like to impart the sweetness and flavor characteristics of Toffee to a banana bread ale I'm doing. With such a high butter fat content in real English toffee, am I going to get myself into trouble adding a pound of this stuff to the wort about 45 minutes into the boil? I feel like I'm in uncharted waters here and don't want to jack this one up! Thanks in advance,

Justin

you can get that flavor without using fat laden toffee. crystal malt and candi sugar come to mind.