Go all grain, and get in on group grain buys. Cost per pound gets really low when you do that. I'm just talking ingredients here, not hardware.
I've washed yeast in the past, and am getting ready to try freezing some for long term storage. It's a private collection strain from Wyeast that won't be offered at least through the end of the year. I have some I reserved from the last starter I made, made another starter with the reserved bit, that should be good for freezing (once I mix up the glycerine solution) as well as reserving in the fridge.
Buy hops when on sale, or in bulk too. Price for a full pound bag (online) is typically a lot better than getting the one ounce bag at the LHBS. There are some HBS' that offer 4oz and 1 pound bags too (Farmhouse for one).
Personally, I've been much happier since getting in on bulk grain buys, buying hops either on sale (as much as possible) or in bulk, and having a grain mill/crusher at home. I simply crush the grain on brew day, not even an hour before I'm mashing (also means I'm not tied to a LHBS/HBS for getting good crush, or having what I want to use). It doesn't take much effort to properly store the grain either. You can leave the sacks as they are, when not opened, and go with 5 gallon buckets to store it. Or you can get sealing barrels (make sure you can lift it when full of grain), vittle vaults, or get gamma lids for your buckets. I'm using all of those. I've added gamma lids to my 5 gallon buckets, have two vittle vaults, and four 14 gallon plastic (HDPE) barrels for grain storage. It's important to keep critters/bugs out of the grain, as well as moisture. With that taken care of, you should run out of grain long before it goes south.