I made my first go at all grain last night, and after I was transfering to fermenter from the brew pot, I was noticing a lot of small while specks or flecks in the wort. I noticed some coming to the surface during the boil and forming a little scum, but I skimmed them all off (or so I thought). Is this normal for all grain, or did I do something wrong somewhere?