need some serious advice/tips so i don't screw up my next batch

ill try to break down my problems, hopefully i can get some advice to make my next batch legit...

i used a good amount of hops (pellets) for a IIPA without a hop bags. i started not using hop bags because i wasnt getting good utilization. when i transferred to primary i just dumped my kettle full of wort (hops and everything) in my glass carboy (*first time using carboy) with a sanitized funnel attached. i didnt use any kind of filter for all the hop sludge and debris. fermented for 11 days and dry hopped w/ 5 ounces without a bag for 10 days in primary (no secondary was used). typically i would dry hop with a bag in a bucket primary. again, the bag seems to mess w/ utilization. before i racked i cold crashed for 3 days to hopefully get everything to drop out.

here are were the problems start-
normally since i use bags for hops in the kettle and for the dry hop ive never had problems racking. well, when i took the carboy out for racking there was a massive layer about 8" high of what looked like a combo of yeast, hops, proteins, etc. ive never seen i layer of crap this big. it looked like i was only going to get 4 gallons of beer instead of 5. i started racking, it clogged halfway through, i unclogged it, and finished. i did get only about 4 gallons and during the racking i seen so much debris go into the keg. well, today i tried pulling a pint and it was clogged. i took out the dip tube now 7 times and it keeps clogging. im pretty much chalking this one as a dumper (unless anyone can chime in to help me save this one).

so, for my next batch i need advice. first, im going to get something like this to hopefully filter out garbage going from kettle to primary---http://www.northernbrewer.com/brewing/brewing-equipment/stirring-straining/double-mesh-stainless-strainer.html

as far as the dry hop, ive heard so many people dry hopping without a bag without any problems. i should be ok with the filter from kettle to carboy and cold crashing, right? or should i go back to dry hopping with a bag? im not sure how to do it with a carboy since ive only done it with a bucket. also, when i was racking from carboy to keg i could barely see the siphon cause the carboy was covered with krausen all the way to the top (there was a blowoff tube used). not really sure how to take care of that problem except maybe measuring the siphon to carboy above the layer or trub.

my goal, to get that filter on my funnel and dump the wort through it to eliminate the garbage getting into my carboy. dry hopping without a bag still and cold crashing, unless you guys think it will clog my keg again? measuring the siphon a few inches above the trub to ensure im not getting excess chunks in the keg and/or clogged siphon. sound good? advice appreciated....

Try whilpooling when you cool to get all the hops sludge and break in the middle of the BK and siphon off the wort from the edge of the kettle into the carboy. You can still use the filter going into the carboy but you can leave most of the trub behind.

When you dry hop, you can put the pellets in a small grain bag. It's easy enough to get out later.

As for the current clogger, you're either going to have to attach some kind of filter to the dip tube or rack the beer through a filter to a fresh keg. I certainly wouldn't dump it out. There must be a way to save it.

I don't think you're getting THAT much garbage from the kettle. You can filter using a normal kitchen-type strainer. The hops will clog it up and create a filter that will catch a lot of the break material and give you less trub.

I think you're getting a lot of that from adding 5 ounces of hops without a way to keep it from clogging your racking cane. Not only are those hops probably soaking up a lot of beer but they are definitely going to clog things up. Bag up the dry hops.

You might want to try using a secondary fermenter to clear it. If you are transferring so much solid material to the keg, it might be worth your while to transfer to a secondary or 'bright' tank for a week to clear it before kegging.

On the rare occasion I have a lot of trub in the fermenter (such as when I dry hop with leaf hops) I take a sanitized 5 gallon paint bag and hold it over the end of my racking cane as I lower it into the fermenter.

in regards to bagging the dry hops, is it even possible to fit 5 ounces in a muslin bag w/ marbles in a carboy? what about getting it out when the hops expand in the bag?