The occasional little particle is fine. These are going to occasionally work through or lose. I think of it as a big natural filter. The vorlaufing works to compact the grain bed and I sort of picture the first few cloudy pitchers as the grain bed attaining equilibrium, moving the loose grains and small particles and non-dissolved solids from the bottom of the grain bed to the top. Slowly those loose grains and solids get moved to the top until what you have is a nice stable gradient. When the wort that you are draining is consistent and hopefully clear, its time to sparge.
The occasional full piece of grain isn't going to contribute enough tannins to make an astringent beer.
If, no matter how long you vorlauf, your wort remains cloudy or you consistently get tons and tons of grain and solids in your beer, then you will need to troubleshoot your mash process.
I would first take a look at your grain crush. Is it too aggressive? Are you getting tons of flour?
Then I'd look at your mash temps and make sure your thermometers are properly calibrated. If you think you are at 152 but your thermometers are off, you could really be mashing at 148 or lower. Unconverted starches in your wort could be rendering it cloudy (imagine flour in water).
If your crush and temps are fine, then you will have to look at your equipment. It could be that the slots in your manifold are too large. I doubt you would have to remake them, you could probably just cover them with a stainless steel mesh etc.
But truly, I think you are fine. What you are describing doesn't sound like cloudy wort to me, it sounds like clear wort with small particles of finely ground grain and husk in it. I think that is probably pretty common in square coolers, since the grain bed is generally thinner, and I think it is going to settle out and taste fine.