The issue here is the information you get is lopsided when you don’t control for pollen. If you know who both parents are, it allows you to compare apples to apple across various matings of females and males (which allows you to understand the heritability of various traits and the breeding value of various parents).
When you lack pollen control, your estimates of trait heritability are inflated due to confounding factors (maternal inheritance, etc.) and you lack knowledge about the real value of the males being used for breeding.
Controlled crosses are the only way to glean this information from any breeding population, even when it means laborious and time-consuming data collection.
Going about this willy-nilly might lead to a single chance event regarding a new variety, but it is not the way to lead or develop a real breeding program.
That's a level of comparison that's not really relevant outside of research or advanced commercial breeding, though. Many crops are bred by many people having little idea what "trait heritability" is, or otherwise not knowing how heritable their selected traits are. If you start from 20 000 seeds to end up with 2, it doesn't really matter how heritable the traits you are looking for, because as long as it's somewhat heritable, you'll have much more than 2 plants with the trait you want to select from. Sure, making controlled crosses can increase the percentage of seedlings that have the trait you want. But that can be mimicked by just planting more seeds. Takes less time and resources to sow 100x more open pollinated seeds than to switch to controlled pollination. Also pretty easy to just propagate a large number of selected males.
Of course, any breeding program that operates on multi-million dollar budgets is going to go "all in". And also of course, it's possible to do both, to use controlled pollination to lock down on some known heritable traits, to then leave those to mate openly. And sure, time put locking down some traits can save time down the road on future crosses.
But if you run on the assumption that you are just having one go at it, and that you've got at least 1 male, then it's just a numbers game. if you sow 20 000 seeds and there are no other males around, assuming 80% germination, you'll get 16 000 plants of the desired cross. If there's one wild male, you'll get 8000 plants of the desired cross, plus an extra 8000 plants that might not be bad either. If there's 50 other wild males in the area, then you'll still get 314 plants of the desired cross, plus 19 686 plants that might not be bad either. If there's 500 other males, then you'll STILL get 32 plants of the desired cross. But this is starting to be a ridiculous number, and that's assuming all males have the same chance. Practically speaking, if you have 4 male plants that flower at the same time as your females and are close to them, and no known males in the immediate vicinity, the ratio of foreign is not going to be high. And again, might just turn out that some of those foreign males are better than your selected ones anyways.
Of course there's value to controlled pollination and I do intend to so some, but I don't think it needs to be done systematically, nor that amateurs really have to do it at all.
In theory are wild hops (fathers in particular) a smart choice to start a breeding program? My only thought here is maybe I could get my hands on seeds or pollen from someone here relatively easily, but my other thought is if a father is wild, shouldnt they be more disease resistant since they have been living out in nature for potentially hundreds of years?
Many cultivars have wild plants in their pedigree, but usually these aren't their direct parents, but rather grand-parents or further back. I would not equate being wild with being resistant to diseases or pests. Wild populations are a wonderful resource to fetch new beneficial traits, but one should not assume every wild plant is by default fantastic. Being able to survive is not a very high bar of resistance.