I've done it. No off-flavors (that I noticed). Not really sure you should. I use them to ferment my funky beers in, so I don't contaminate my regular brewing stuff.
I haven't died yet, but who knows what kind of crazy chemicals I'm leeching from them. Maybe I'll get lucky and get hit by a car before I get cancer.