It might pick up a little more bitterness from the orange, but I wouldn't worry too much. Without the heat and agitation, it shouldn't be that much more. If you're worried, wait until it hits terminal gravity and transfer to a secondary, leaving the other stuff behind. As for anything else, break material and such, many of us just dump that right in the fermenter anyway. A certain amount of break material is healthy as it feeds the yeast.