I would just say “you have no idea” and then give them a big smile.
To the ones you know are married you could say "Funny I was thinking the same thing". You could add "about you" but the other way they don't if you are talking about them or agreeing with them.
If they ask "what do you mean" just reply "You know" and walk away.
There used to be an area here, which shall not be named, but, yeah, that sentiment was well proven.
Shallow men believe in luck. Strong men believe in cause and effect. - Ralph Waldo Emerson