I spent 2 hours a day in my junior year of High School building a house. The class was called "Construction Trades" and we built and installed EVERY aspect of the house. We did all of the plumbing, wiring, framing, roofing etc etc. The only things we didn't build were the cabinets. We of course had to have everything certified by electrical inspectors plumbing inspectors building inspectors etc.
How cool was it that a bunch of 16-18 year olds built an entire 4 bedroom home? The co-operative where this took place ran 3 classes a day from different neighbouring schools, and when the project was completed, the auctioned off the house and hopefully made money on the deal.
Could/Would I build one myself? Not a CHANCE! SWMBO and I built the house we are now living in through a local contractor. At the end of the entire process, I can say there are definitely some things I would have handled differently, but hell, live and learn.
SO even with the knowledge of how to build a house, and having the experience, hands on, of building one, i wouldn't even consider trying to manage the process of scheduling dozens of different contractors and dealing with rain etc etc. I would pay someone to do that that does it for a living.
Course it WOULD be a lot of fun eh? Have you looked into Kit Homes? The sell you everything you need, all cut labelled and packaged, and you "just put it together".