I haven't done it but protein rest is actually supposed to be effective in a thick mash. You could start with 1.7-2 liters/kg only and then dilute with boiling water to bring it up to sacch temp and you would still be inside the acceptable thickness range. Use a program such as Beersmith to estimate the volumes (there is a calculator for this type of things). And be prepared that the calculations may not be 100% correct so you may need to adjust slightly when you see how things turn out. Another option is to use decoction mashing where you heat up part of the mash in a kettle and bring it back to increase the heat without major increase in volume.
However, most modern malts do not need the protein rest so you want to be sure that it is necessary before you start experimenting with it.