Here's my best way of judging whether something unethical. If the situation was reversed and you were the person it was being done to, how would you feel?
As for my example about the 3-card monte. Con men are "social engineers". They know exactly what most people would do, and most people would happily take money from an evil con man. It's considered stealing from a thief, and that makes it ok. In fact, the dealer of the con thinks exactly the same thing about you. If you know which card is the right one because that card got marked somehow, then if you bet you're stealing from him. That makes him think it's ok to steal from you. If you don't bet because it seems wrong, he can't (and doesn't want to) take your money. He even ups the stakes by acting racist and mean, so you won't have sympathy for him. He might be less racist than you, but he knows that acting that way makes it easier for you to try to steal from him. (And just to be clear - if he's really a con man, he'll make sure you think you can cheat him, while he's actually cheating you. There have been literally millions of dollars taken from people, 20-100 at a time, people who were sure they had seen the cheat and knew how to beat the game. The only way to win at 3-card monte is not to play - trust me on this one.)
I still don't see how the question about the 3-card monte game is an ethical one. The POINT of the game is for him to use subterfuge to trick you, and the POINT of you playing is to try to discover that subterfuge, and not to be tricked. As long as both sides agree on those two points, there's no reason for it to be an ethical issue at all, since both sides are agreeing to the transaction.
Let me put it this way. Let's say I challenge you to a baseball-throwing contest. I know I've got a great arm, and you know you've got a great arm. Both of us think we can beat the other person, so we each put $10 on the bet. If you throw further, you win my $10, if I beat you I take your $10.
The same thing is happening in 3-card monte. You (the dealer) believe that you can dupe me into losing the card, so much so that you are willing to bet $10 on it. I am confident that I can discover your trick, and am willing to lay $10 on my ability to do so. Neither one of us KNOWS what the outcome will be beforehand--I might be a former grifter myself, and know all of your tricks, plus some; you might be the slickest 3-card monte dealer in the world, and no matter how clever I am, I won't be able to detect your deception. But the point is, by agreeing to the game, both of us agree to the rules, and to be bound by the outcome. If you win, it's not unethical; neither is it if I win. As long as both parties are capable of making a reasonable decision about whether or not to play, there's nothing wrong with it.
As to the "if the situation was reversed" yardstick, it doesn't really apply here. Of course if I win the 3-card monte game I wouldn't be happy if the situation were reversed, but that's always the case in a competitive game, with a winner and a loser. Does that mean that sport is unethical because not everyone comes away happy?
I also disagree with your perception about 3-card monte dealers being "con-men" in the sense that you describe them. If I see a guy spinning cards around on a briefcase in an alley, my first thought is not, "I bet this guy never tries to pull a fast one on people." I ASSUME that his attempts at deceit are part of the game. And for people who choose to play, that's part of the fun! Trying to catch the magician and find his trick is part of the enjoyment. There are two motivations for playing 3-card monte: 1) I think I can beat the guy and 2) I like the thrill of uncertainty, the idea that I "might" win, and the wonder of watching someone do something that I don't understand (like "magically" mixing up the cards so that I guess the wrong one). 3-card monte dealers aren't con-men, so much as they are entertainers--they give people a little excitement and a challenge, and take some money from them in return. It's not like anyone is making that chump who loses the $50 to a streetside card hustler do so, he did it because he wanted to. Not unethical.
You also completely lost me on the racism thing--doesn't seem connected at all to the ethical dilemma, maybe this is an actual situation that happened to you? Anyway, I think it's extraneous to the main point.
Just my .02