I would go all grain.....
That said, maybe you can do both..... (depending on what you have for a pot right now).
What do you have for a brew pot? Get a 10-15 gallon pot and start with Brew in a Bag. Get the pot. Get the brew bag, and that is all you need to get going with all grain, 5 gallon batches. You can add more elements to this along the way if you want to move toward more traditional "sparge" set up. But, there are a lot of people making some great beer with the BIAB setup.
If you already happen to have a big pot...... you essentially need almost nothing to go all grain.
Plus, buying grain for your first few batches will begin to offset cost of extract pretty quickly.
Personally, I would recommend you do not skimp on your pot. Get a big pot, for full boil. Spigot for draining. Maybe a thermometer in it for monitoring mash temps (although you could do this in other ways too). This is one piece that you can use absolutely forever, and, if you ever decide you don't want to brew, it will have solid resale value.