For my first batch I went dumpster diving for bottles. I was about half way through disinfecting those when I was sure I would never again clean other peoples fermenting back wash out of beer bottles. $12.95 for 24 empty 12oz bottles is a very reasonable price to me.
Also, nobody knows anyone who went back to bottling after they started kegging...
Starter kit, absolutely.
2 cases of bottles for $26, an extract kit for say $30 and a beginner equipment kit for another 50-75, you are going to be paying a lot of money for the first two cases of beer.
But, once you have it in bottles for about a week you are going to start liking it. If you can leave some of it in bottles for four weeks you will probably be hooked for life.
Once you are hooked $250 for a floor standing bottle capper will sound reasonable.