I've found it to be a nuisance as well, my main contention isn't with having particles in the beer when it is done in the primary because most will settle especially if you cold crash. Really because when I retain yeast cakes for future slurries it can be more difficult to layer and seperate the trub from the yeast.
What I do is when I pour off my wort to my primary, I wrap the mesh bag around the bucket (sanitized first of course) and then filter the contents through it. I don't do this with all batches just some that I want to harvest the yeast afterwards.