In my last post, I mentioned that using random blocks of questions can result in repeated questions on an attempt. The reason behind this is that Blackboard does not track which questions have already been selected from the pool for any given test attempt. Questions are always selected from the full pool. The testing system is using sampling with replacement. This means there is a chance of the same question being chosen more than once for the same attempt (student). The likelihood of a student seeing duplicate questions depends on the size of the sample relative to the pool.
So, how large a pool do you need to decrease the likelihood of repeated questions with random blocks? To get a sense of the size of the pool needed I calculated the probability of no replication of questions for a random block of 5 questions drawn from question pools of various sizes.
| Question Pool Size | Probability of No Repeats |
|---|---|
| 10 questions | 30 % |
| 25 questions | 65 % |
| 50 questions | 81 % |
| 100 questions | 90 % |
| 200 questions | 95 % |
OK, these results aren’t especially surprising. As you increase the pool size the you decrease the chances of having questions duplicated. But if you are pulling questions from a single pool you are going to need a lot of questions to reduce the likelihood of duplicate questions.
Is there another way?
The point of random blocks of questions is to present each student with a different set of questions on the test. If you select a single question from any given pool you won’t repeat questions on any given test. Break a larger pool of questions into a number of small pools equal to the number of questions to be drawn and draw a single question from each pool. The pools don’t need to be large, but all the question across pools must be unique. Using this method only a few questions will be needed in each to provide variation across tests.
Using many smaller pools will require a bit more work in creating the test, however there is an additional benefit. You can better control coverage of the test. If you create the questions in each pool on the same topic, selecting a random question from each pool ensures each student receives a question in each topic area. Using a random block of multiple questions from a large pool may result in uneven topical coverage in addition to the problem of question duplication.