I started All Grain, and only have ever done all grain. I read "How to Brew" online, read a lot here, purchased and read the hard copy of "How to Brew", read some more... then spent ~$40 on my 5gallon round Igloo cooler MLT and brewed. I haven't looked back.
In general ingredients for all-grain brewing is cheaper than buying extract, and I enjoyed (still enjoy) the crushing, the mashing, the sparging, etc. and the everything involved with all-grain brewing.
That said, as you can see you can do baby steps with all-grain as well, plus you get lots of other fun stuff you can dive deeper into if you so desire.
I used stock pots I already had, and one bucket fermenter and just did 2.5 gallon batches. Doing all-grain is not rocket science, and does not require a large outlay of money. Prepare ahead of time, have all of your materials in place before you start on brew day, have a written/typed schedule of what needs to happen and when, and brew!
As far as getting started, definitely buy once-- get at least a 60 quart BK, etc. I also recommend trying to maximize how far the dollar goes by having as much of my equipment being multi-purpose as possible (i.e. I ferment and serve from cornie kegs).
Good luck, and enjoy!