It's difficult to have sterile wort at pitching time. In fact, very difficult. The point of sanitizer is to eliminate as much bacteria/wild yeast as possible, but it doesn't kill them all. What's left over is soon pushed out by a larger, healthier yeast population (provided you pitch correctly) that quickly consumes the available food as fermentation starts up.
If you took the sample without pitching any yeast, whatever bacteria/wild yeast that wasn't killed during the sanitizing process is still present in the wort and will slowly begin to reproduce. Without any competition, over time, a larger population will develop and start to show signs of fermentation.
I really wouldn't worry about your beer unless it didn't ferment. If fermentation took place, the unwanted bugs most likely died in the process.
EDIT: I bet 9 times out of 10 your "contamination sample" will live up to it's name and eventually show signs of contamination. Unless you have some way of completely sterilizing all equipment and the wort itself, you'll likely have some sort of organism living in there. After all, it's wort! Bugs love that stuff. That's the whole reason brewers pitch yeast into it!