Most of the cyanide compounds are bound to the seed - they aren't HCN, they are complex cyanides, which are likely safe to be ingested - however, adding energy and hydrolysis can activate these, cleaving off HCN.
For instance, a friend of mine while in the Israeli army, isolated pure HCN from peach pits as a 'project'. About 1 cup of pits is enough to kill someone, but again, to actually reach a lethal dose you would need to crush the pits, boil the pits, and/or add acid (or base) and boil.
Then drink up... So no, its not a real worry, but stranger things have happened.
And yeah I'm a chemist / plant biologist.
Unreliable source tells me 9 people have died from stone fruit pits in the past half century.