FYI- I am pretty knowledgeable about cheese (probably moreso than about beer) if anyone has questions about cheese don't hesitate to ask me.
UP may have been the problem. Did you add calcium chloride?
As far as raw milk goes -- yes, pathogens are always a concern, regardless of whether you are using pasteurized or raw milk.
Raw milk actually has natural protections against pathogens which are destroyed by pasteurization. There are enzymes in the milk from the cow (or goat or sheep) which inhibit pathogens. Raw milk also has natural lactic-acid producing bacteria (and a host of other yeasts, bacterias, and molds) to compete with the pathogens. Truth be told (the FDA will never admit this, although there are hosts of studies to verify it) pathogens like listeria are actually poor competitors with the other non-harmful organisms.
These protective properties of raw milk are all destroyed by pasteurization. Therefore, pasteurized milk you actually have to be more careful with, because any pathogen which gets a foot-hold will have basically free range to do what it wants.
This doesn't mean pathogens aren't a concern with raw milk. If a raw milk is already contaminated when you get it, then you are better off pasteurizing it prior to processing. That is one of the reasons that large processors with milk from dozens of farms pasteurize their milk, since they don't know what is in the milk. However, once the milk is pasteurized, it must be processed immediately, lest something unwanted gets a hold of it.
You should always be very scrupulous with sanitation when handling milk, regardless of weather it is raw or pasteurized. Here's why:
#1 -- Potentially dangerous bacteria are present everywhere. Staphylococcus lives on your skin, and Listeriosis lives in soil, to name but two. They are present in small quantities in ALL dairy products (believe it or not) including raw milk, and quite a number of other food stuffs including both animal and plant products. These bacteria are harmless at the levels they occur normally, but if they find the right growth conditions without competition can reproduce exponentially, to dangerous levels.
Even non-dangerous organisms are an issue of concern. Would you really want to drink milk or yogurt, or eat cheese that smells like vomit, or that has a yellow yeast or black mold growing in it? The organisms that cause these problems are not usually dangerous to your health, but they are also not desirable in most circumstances.
#2 -- Fresh milk (regardless of weather it is pasteurized or raw) is an excellent growth medium for many different organisms. It is rich in carbohydrates (lactose), nitrogen (proteins), moisture (water is the primary component of all milk), and it has a nearly neutral pH (about 6.7).
Beer, on the other hand, is a pretty safe thing to make. The worst you can do to beer if you contaminate it is make it sour, which would actually make it MORE safe because the more acidic the less likely a pathogen could live in it.
In fact, a major reason people made beer historically was because they didn't know if their drinking water was safe, and making beer was a sure way to have clean water that would keep for a while. Boiling the water might steralize it, but it wouldn't KEEP it safe... pathogens could still get back into it and grow. By making small beer (which was really more like a cross between beer and a lacto-fermented beverage like kvass or rejuvelac) people could have a safe source of drinking water that would keep for weeks or months.
Cheese, on the other hand, was a way to set-by milk, because milk is so perishable. Cheese traps the nutritious solids from the highly perishable and watery milk, into a curd which would separate out the liquid into whey, thus making the milk and all its nutrients less perishable, more durable.
Cheese and beer are almost like complimentary opposites -- beer is the process of taking a solid (grain) and turning it into liquid, where cheese is the process of taking a liquid (milk) and turning it into a solid. But at the root, they both come from the same place -- grass. Grass either allowed to go to seed and then turned into malt; or processed by a ruminant animal into milk.
Alright, I'm way off topic here. I'm done.