Yeah there's definitely a difference on fruits, which is why you'll hear people say not to use peaches or to use raspberries. I haven't used kumquats but just from my fruiting experience, I've always found more is better.
And yeah, I'm right there with you, fruiting beer gets $$$$$ quickly.