Yeah don't sweat it especially for a stout. I've also heard (forget where) that it's actually useless or harmful for extract beers. The reasoning being that all the crud has already been removed by the manufacturer when they created the extract. So any proteins you are removing with the moss are the good proteins that create good head and body.
Don't hold me to that

but trevorc's post kinda backs that up.
edit: I think it was in How to Brew by Palmer