So doing some more research and it seems like I inadvertently used something like a partial no-chill method, which explains a lot. Those 5 and 0 minute additions I did would have been more similar to much earlier additions, so their hop flavor and aroma was pretty much lost and the beer was more bitter than it should have been. Apparently that's the biggest problem with the no-chill method and hoppy beers.
So next brew, probably tomorrow, use some bittering hops with no later additions and then dry-hop. I think the no-chill method is the only real method that works for me unless I had a freezer. It's either that or make some small batches that I can chill fast enough, which I might end up doing later, or wait for winter.