The reason they can have the pole there in your backyard is there is an easement and it is guarenteed even upon the sale of the property. Part of the contract, if there is no easement then it is your land to do as you wish. Signs and fences can keep people off the land even if use was was granted in the past. You have the right to deny accss.
You might not have the right to deny access, check out the laws regarding adverse possession in your state. That's why some some large office buildings with a plaza in front will block it off for "maintenance" for a day each year. They retain their property rights that way.