I am trying to create a (roughly) counterbalanced experimental design for a few hundred images that are run through various distortions. Let’s say 4 types of distortions: a,b,c and d.
I want to test a group of observers so that they see each image once and this image should be influenced by one distortion. I want to counterbalance both images and distortions so there is no risk of observers learning a pattern in the distortions. Crucially, I want the combinations of numbers and letters to be equally often presented.
I try to solve a toy example like this:
image <- sample(1:12) # Must be divisible by 4. Will be +100 in my use case
div <- length(a)/4 # How many of each letter
distortion <- sample(c(rep("a",div),rep("b",div),rep("c",div),rep("d",div)))
c <- paste(image, distortion, sep = "")
When I run it, it outputs for example:
"1b" "8d" "4c" "11d" "9a" "6b" "3a" "7c" "2b" "10d" "5c" "12d"
As such, this would be great to show a single observer. My problem that I can’t see a solution to comes when testing multiple observers. Imagine that we want to test 16 people for this toy example. We aim for having about the same number of number + letter combinations. For instance, we want 2b (and 2a,2c,2d) to be present in roughly 25% of cases.
When I run the following code, testing for the proportion of 2b, I get a lot of variance:
shuff <- matrix(NA,16)
for(i in 1:length(shuff)){
image <- sample(1:12)
distortion <- sample(c(rep("a",div),rep("b",div),rep("c",div),rep("d",div)))
c <- paste(image, distortion, sep = "")
shuff[i] <- ("2b" %in% c)*1}
mean(shuff)
There could both be ~ 6% of the cases where 2b was present or +40%. Is there anything obvious to do to achieve a design that is more balanced?
Thank you for reading, my question. I illustrate my problem with R code, but python is also more than fine for solutions.