The latest teensy boards are indeed nice and have a fast soc, but i am afraid this project is fully based on esp-idf and supporting other platforms then esp would require lots of work which i personally don't have.
Also atm there isn't really any benefit, the esp cores are fast enough, cheap...