You basically need a bucket with a lid, a racking cane and food grade tubing to be brewing partial boil pre-hopped extract batches: everybody has at least a somewhat big pot at home that can dissolve the extract. A friend of mine started with just that: no hydrometer, no wine thief, no bottling wand, etc. He drilled the lid of the bucket to accept an airlock and bottled in the same bucket. He rented a capper when it came to bottling time.
Now, if you have a second bucket just for bottling (with a spigot if you want), it will make the final product clearer since you won't have to stir up trub to mix the priming solution. Hydrometers are also dirt cheap. A bottling wand is also very inexpensive and will make your life easier.
But if you really, really wanted to get started on the cheap, you could just get a 6 or 7 gallon food grade bucket from Home Depot at about 8$ (or even a frosting bucket from a bakery if it is not scratched, usually free), slap on a lid and get the other stuff from any reasonable LHBS for about 10-20$. Also, there's no reason you would have to trash any of that stuff if you upgrade. Plastic buckets are always useful, wether for storing objects or to contain sanitizer/water during brewing operations. The only things that needs to be thrown out somewhat regularly are the tubing and racking cane since they can scartch easily and the harbour bacteria. But if you aren't brewing 200 gallons a year, I wouldn't expect you to have to trash a lot of equipment due to wear and tear.
The thing that is the most expensive are usually the bottles, unless you happen to have a lot of friends who drink capped brews. Twist-off can work (especially up here in Canada since they seem to be thicker) but I wouldn't count on them anyway.
