I've never owned one, but I would guess it is glued in with either sealant or spray foam. You would need to get in there and slice that glue from under the flange and probably from the underside as well.
You are probably better off just painting around it.