Sounds like your bottles weren't clean as you might expect? Do you sanitize your caps?
The day before I plan to bottle, I fill my sink with hot water and PBW then soak my bottles for a couple hours, then rinse well. On bottling day, I sanitize the bottle and the cap, fill and press the cap...