your gravity will be lower so your beer might not be as flavorful as you may have wanted. Was it a five gallon batch? I don't think you'll ruin your beer. The yeast should eat away any unwanted bacteria during initial fermentation.
There's a few different ways to add orange flavor to your beer. You can add in some zest in the last 10 minutes or so of the boil, you can add rinds to the secondary and age, or you can soak the rinds in vodka for a while and add that to your secondary. There's lots of information on here about adding orange to beer if you use the search bar you're bound to find something.