I just siphon the wort into the carboy using an autosiphon.. since I use whole hops they don't really make it into the primary very often. As for the break material I don't really bother. If you did have some sort of off taste, and I've never experienced such even when I just dumped the whole lot into the primary, I would think that it would be from the hops. Have you ever done any dry hopping? That "grassy off flavor" sounds to me like the taste that results from dry hopping - leaving spent hops in the primary could have a similar, though less pronounced, effect. The hot and cold break may actually be good for the yeast besides. I think any off flavors that result from the yeast coming in contact with the trub require fairly long contact time - as in longer than 1 week. The yeast don't metabolize the trub until the easy food (sugar) runs out - at this point fermentation dramatically slows and it's time to transfer the wort to the secondary. Off flavors resulting from the trub are typically described as 'soapy' and not grassy. Heck, I don't think you could even call the stuff 'trub' until it has settled on the bottom. You may be actually scooping the krausen out which is probably not a great thing to be doing. If you're concerned about clarity then why not use a whirlfloc tab/irish moss during the boil? You could even use some gelatin or other finings to clarify if you still are satisfied.
As for scooping stuff out of the beer using a colander I wouldn't suggest it. You're exposing that wort to bacteria and wild yeast... in practice it's probably pretty safe but I don't really think you gain anything by doing it.
If you're really concerned with what makes it into the primary, as I a lot of people on here seem to be, then get a funnel with a screen in it; they sell them at my LHBS and I imagine you could get them from one of the internet vendors as well. You could probably just put a nylon screen (like what they sell for steeping grains) over your racking tube and I'll be that would keep a lot of stuff out.