Correction: You've never tasted oxidation so the step is unnecessary for you.
Post fermentation oxidation is very well studied and understood. If you aren't purging O2 before packaging then you are oxidizing your beer.
for example
Ethanol reacts via oxidation to form Aldehydes. Aldehydes above 10ppm can be tasted in light beers and above 35ppm in other styles as a green apple flavor.
Linoleic acid oxidizes into dihydroxy and trihdryxyoctadecenoic acids that are precursors to 2-trans-nonenal. The "cardboard" flavor. This reaction will take some time to develop.
Oxidation will darken the wort color.
Pickup of oxygen during packaging should be less than .03ppm (or as close as you can get) there is no way you can get anything close to this without purging the keg!!! Why go through the whole process then skimp out on spending 2-3 minutes to purge the keg?!?!
You are right I have never TASTED oxidation in any of my beers. I have also never seen any darkening from start to finish.
So, I do not have an oxidation PROBLEM. So the step of purging is just a waste of co2 for me. YMMV.