It should be fine. Any pumpkin remnants should drop out in the primary, when you move to secondary or the bottling bucket it should stay in the primary. Being that it's been in the wort/primary longer the pumpkin flavor will be stronger than if you removed it with the grain bag.