If you soak it a few times in hot water it should get most of the soluble tannins out. You will definitely taste the tannins in that first soak! Red Oak is ring-porous, meaning the pores running parallel to the grain are open, so it will leach out fairly readily. As opposed to White Oak, where those pores are closed off, so it doesn't leach as fast.
I would think the bigger problem would be nasty stuff growing in the wood between brew sessions. You'll probably want to do a hot-water rinse before each mash, to leach out whatever icky flavors have developed.