Back when I was using a DIY garden house CFC, we would immediately rinse after transfer and blow dry with an air compressor. After year or more of using, we decided to do a more thorough cleaning by running hot PBW through the chiller, capping it and letting it sit and soak for 30 minutes. The amount of crusty black crap that ran out of there after the soak was not a great sight. It was at that point that I realized merely rinsing before and after using wasn't cutting it.
I suppose if you make hot PBW soaks part of your routine and do it more often than yearly, you could avoid such build up. We were using a hop spider for those batches too, so not a lot of hop trub was going through the chiller. I was more than surprised at how dirty our CFC had become even though we often used it biweekly and rinsed it immediately after chilling. Even then, things will start to build up inside.