Don't mix fermentation and carbonation stages. Although the yeasts act on sugars in both cases, the proportions (quantities) of yeasts as well as sugars are different. The wort has much more sugars than the fermented beer. Also, you pitch a lot more yeast at the beginning (prior to fermentation). After the fermentation, the yeasts convert a lot of the sugars to alcohol and CO2. The expended yeast themselves become inactive and settle at the bottom of the fermenter.
When you add the pre-defined measure of priming sugar to the bottling bucket, that's just enough to provide the remaining suspended yeasts the fuel needed to convert it to CO2 along with a little bit of alcohol (probably less than 0.5%).
So, since the number of yeast cells as well as sugars are low, it takes a while to generate the carbonation.
Many times, brewers use fining agents in the secondary to provide the beer more clarity to remove any suspended proteins, hop particles etc. . These fining agents also cause additional yeast to be extracted out of the beer along with other proteins. But all the yeast is not removed. There is still left. It is not uncommon for brewers to pitch a little quantity of yeast while priming to boost the carbonation process in such cases.