Hops are indeed the culprit. You can actually notice skunking if you sit out in bright sunlight with a pint. By the end of the glass it will start to appear. Some major brands use hop oil extracts to reduce skunking.
Sunlight can breakdown many different compunds in beer. Without hops you won't get skunking but, I can't imagine it is good for the malt or yeast contributing compounds either.
Still, the question is: Why do you need to brew in direct sunlight? You can't put a cover on the carboy? I use the box it came in. I just poke the airlock out through the flaps.