Don't get mad - get even. Let them start building, if they build without the setback - report them, they will make them take down what they started.
BTW - nice guy or not, you better explain what a property line is, building a road on the line is a bad idea, and you letting them get closer and...