The yeast won't be able to ferment any starches. So without filtration the beer would probably never clear. There's also a risk (however small) that bacteria or wild yeasts could take hold after packaging, since they can metabolize the starches.
If there's a lot left over it could contribute some body/mouthfeel, but it won't taste "starchy". Think of plain white flour - it doesn't really taste like anything.
Edit: You could use amylase enzyme to break them down, but then you run the risk of drying out the beer more than you intended.