Yep, close the door, sit down, and BAM one of the cats is scratching at the door.
Let it in and it will go in and eat (Their food is in there) or rub up against my legs, or just sit there looking around as if they can't remember why they had to be in there.