Roast barley is indeed very brittle and will mostly turn into powder but, as already mentioned by another poster, the amount will be so small as to have no efffect on sparging. If you don't mill then I expect you will extract very little taste and color from it, in any case much less than any recipe calculator will predict.