Because it is extra stuff that has a taste that goes into the beer. I know of cases, where mysterious off-flavours were hunted and hunted and not found... until they ditched the yeast nutrient.
Malt is basically really rich in nutrients, and although beer yeasts are a bit of spoiled character, compared to not minding wine yeasts, they still have everything they need within the wort already present. A healthy pitch rate and fitting temperatures is everything that is needed.
One could argue that kveiks are a little exception, as they have eveolved within higher gravity wort, but from my own experiments, I must say that at least lutra performs very well in low gravity worts (1.03) without any extra nutrients. I had the same result with voss in "normal" gravity wort (around 1.05).
So the takeaway is: Do not use stuff in your beer that is unnecessary because it will somehow affect the flavour. if it is in there, it will be also in the flavour, maybe underneath the taste threshhold if you are lucky, but if unnecessary from the beginning, why take the risk?