Is it too hard to freeze dry them?
Right on the first try. Most yeast strains cannot survive drying. Manufacturers are finding new ones (dry lager and dry wheat yeasts both hit the market in the last couple years), but the more specialized a yeast is, the more fragile. Think toy poodle vs basic mutt.
Remember one unassailable statistic, as explained by the late, great George Carlin: "Just think of how stupid the average person is, and then realize half of them are even stupider!"
"I would like to die on Mars, just not on impact." Elon Musk