Myth or not, I've seen reports here on HBT where it allegedly took a long time in a secondary. I was wondering about why that was, as indeed, honey consists mostly of simple sugars, and if used in the primary the yeast goes rampant. As you said, it likely has to do with low yeast count and lack of nutrients after racking and alcohol present plays a role too. Cold crashing before racking reduces the yeast count drastically, but there should be enough left or bottle carbonation would become impossible.
I have a beer or 2 lined up for secondary honey additions and see what happens. I tend to ferment primary, and the few cases I use a secondary, on the cool side to keep fusel alcohol formation minimal.