I like your thinking, BUT:
I want to believe that we're smarter and wiser in the future - especially someone that would be bright enough to come up with a time-travel device... and they would anticipate the negative effect the mere knowledge of time-travel possibility would have (us researching it more dilligently once learning that it's possible, altering it's development, changing the time traveler's past).
So, even a semi-intelligent traveller wouldn't identify himself as such, and we would never know.
Personally, I love the idea of time travel.
But everyday common sense tells me it's impossible.
But from what little I know of physics & mathmatical theory (I'm an enthusiast, I read books and listen to Hawking's lectures, but I'm pretty dumb when you get down to it)... but from what I know, it's theoretically possible... I mean, time is simply the 4th dimension. It's our own perception of that 4th dimension that makes it different than up/down/left/right/in/out. String theory regards what we call time equally to what we call space.
As soon as we get our heads out of our asses and wrap our minds around what the universe is really about, I think we should be able to accomplish time travel.
EDIT: <thinking> hopefully i didn't get too deep and kill this thread... i hope there's some people out there with some ideas to counter mine...