will over carbonation cause off taste?

So I force carbbed a keg of watermelon ale, I then transferred it to a 3 gallon keg after we drank some of it so I could fit another keg in my fridge (it holds 1 3 gallon and 1 5 gallon) over the next couple of days I had a few more beers from the 3 gallon and they tasted fine but my Co2 tank ran out so I used one of those small hand chargers to pressurize the keg, after a few more beers I pressurized the keg to the max the little hand charger would do and disconnected it. After a few days I went to have a beer and it tasted awful, kind of like rubber and I thought it was from the seals on the new keg, I had planned on dumping it but decided to try lowering the pressure and when I did the beer seemed to taste better. Is this possible or am I going nuts?

Thanks

Over carbing can lead to a lot of carbonic acid, which has a kind of 'bite' to it, much like the acrid smell the blanket of CO2 has in yoru primary if you try to stick your face in it and take a whiff.

it will subside.

But rubbery...that's not how I'd describe.

By hand charger do you mean CO2 cartridges or just a pump? Obviously, just a pump will oxidate the beer.

yes co2 charger