If you are looking to increase bittering utilization, nylon or muslin bag will work against you since you are reducing surface area as the hops clump together. Besides an effective whirlpool, using a hop spider is your best option as the hops are allowed more room for wort contact. If you stick...
It's fine to do that, but keep in mind that hop pellets are really absorbant of wort, so if you are dumping a ton of low AA% hops to get your deisred IBU, you are greatly increasing your trub, and decreasing your overall yield. In simpler terms, less beer at the end of the day.