I'm thinking about doing a taste experiment like this. Would the chance of a type 1 error be low enough if the sample size was just me or should I get my roommate to taste them too?
Type I error is a false positive basically. So in a difference test (you are trying to determine if you can tell the samples apart) it means the chance that the people who got the difference right did so by accident.
A common difference test which works with fairly small sample sizes in sensory analysis is the triangle test. Small in the professional sensory analysis world for this type of test means 20 people or so. In a triangle test you get three coded samples, 2 are the same and 1 is different. You are asked to pick the different sample. If you can't tell them apart, you are correct 1 time out of 3.
If you do this test with 2 people and you were both right, that would happen randomly 1 out of 9 times (1/3 * 1/3). That means that alpha (the probability of a type I error) is 1/9 which is fairly large. Usually alpha is below 1/20 (it should be declared before the test begins, it is the probability of type I error that you will find acceptable, not what you observed). If you have 4 people and all 4 get it right then you have alpha = 1/84, which is pretty good. The problem is, what if not everyone gets it right? Someone has a high threshold for some off flavor or is having a bad day etc. That's why you want a lot of people. 10 people picking the right odd sample out of 20 is much more convincing than 2 out of 4 or 1 out of 2. That is the problem with small sample sizes, 1 or 2 people miss and alpha explodes.
Now, I falsely thought this experiment was a difference test. I still haven't watched the videos (would rather watch them all at once), but it appears that only two samples were provided. So not a difference test but a consumer preference type test. Basically, tell me which qualities these two samples have and which you prefer. If only you and your roomate drink your beer, then that is the perfect sample size for you to say which techniques you prefer. If you are selling beer to thousands of people, or want to claim to the broader home brewing community that a certain ingredient or technique is better, then IMO you want a much larger sample size. Again 20 would be considered small if you approach this from the POV of what is ideal and professional breweries (big ones) and food companies will pay 20 people to participate. As homebrewers we have to get 20 volunteers and then pay maybe $8 each to ship the beer. That gets expensive (the pitching rate write up in the current Zymurgy that Sean Terrill did had a decent sample size, he had people interested enough in the experiment that they paid for their samples to be shipped, that is the way to go IMO).
Only kidding.
