So long as the patented variety wasn't being used for commercial uses, pretty sure that there would be no issue for a lucky "Joe" with Simcoe in his backyard.
The patten owner wouldn't have suffered any monetary damages, so there shouldn't be anything to sue "Joe" for.
The pattern owner might...