The audit of our own question bank

Every practice exam claims to be good. This one publishes the measurement, the method and the script — including the defect that sat here open until it was fixed.

Why this page exists

A practice exam has exactly one job: to tell you whether you are ready. If it measures badly it is worse than nothing, because it hands you false confidence and you book the exam.

The problem is that you cannot check this from the outside. You take the practice exam, you score 85%, and you have no way to know whether that was knowledge or whether the answer key was always option A.

So here is everything you need to check for yourself: what we measure, how we measure it, the script and the data.

What the three tests look for

1. Answer key positionIf the correct answer clusters on one letter, you can pass by always picking that letter. That was the first defect we found in our own bank: 95% of the answers were on A.
2. Length cueQuestion writers tend to qualify the correct option and clip the wrong ones. Someone who knows nothing picks the longest and passes. We measure this at two levels: the cue you can spot by eye, and the tilt only a character-counting script can see.
3. Guessing simulationTwo thousand exams answered at random. The score has to match mathematical chance. Anything above it means a cue slipped past the first two tests.
4. Absolute termsA test-prep heuristic: an option containing 'always', 'never' or 'only' tends to be the wrong one, because writers reach for the exaggeration when manufacturing a distractor. We measure what you score by eliminating those and guessing from the rest.

Why question-by-question review never catches this

Looked at in isolation, each question is flawless. The answer key sitting on A is only a problem when you look at all 600 at once. It is a batch-level defect, and review happens item by item — which is why almost nobody finds it. We only found it because we stopped reviewing and started measuring.

The result, run just now

Script output, unedited · 2026-09-13

====================================================================
  AUDIT — metrica_en.json
  600 questions
====================================================================

[1] ANSWER KEY POSITION
    Question: can you pass by always picking the same letter?

      A   149   24.3%  #######.....................
      B   167   27.2%  ########....................
      C   147   24.0%  #######.....................
      D   150   24.5%  #######.....................

      expected per letter ............... 25.0%
      largest deviation ................. 2.2 pp
      score of always picking B ......... 27.8%
      PASSED  (criterion: deviation <= 5 pp)

[2] LENGTH CUE
    (a) VISIBLE — can you exploit it by eye, sitting the exam?

      the CORRECT one stands out in ..... 1 of 600   <- this is what can be exploited
      a DISTRACTOR stands out in ........ 4 of 600   <- imbalance, not a leak
      PASSED  (criterion: correct standing out in at most 1% of items)

    (b) TILT — can you exploit it by counting characters?

      correct is the longest ............ 25.0%
      expected by chance ................ 25.0%
      length ratio correct/wrong ........ 1.07x  (1.00 = neutral)
      mean difference ................... +2.9 characters
      PASSED  (criterion: deviation <= 8 pp)

[3] GUESSING SIMULATION
    Question: what do you score knowing nothing? (2000 simulations)

      mean guessing score ............... 25.6%  (sd 1.8)
      mathematical chance ............... 25.0%
      PASSED  (criterion: difference <= 2 pp)

[4] ABSOLUTE TERMS
    Question: can you pass by avoiding 'always', 'never', 'only'?

      absolutes in correct options ...... 0.0%
      absolutes in wrong options ........ 0.0%
      score of eliminating then guessing . 25.6%  (chance 25.0%)
      PASSED  (criterion: difference <= 4 pp)

====================================================================
  RESULT: NO CUE EXPLOITABLE BY SOMEONE SITTING THE EXAM
====================================================================

The number that was open, and what was done about it

When this page first went up, a script counting characters scored 49.2% on this bank by always picking the longest option, against 25.0% by chance. We published the defect before we knew how to fix it.

Today that script scores 26.0% — the correct option is the longest in 25.0%, against 25.0% by chance. The fix was not shortening the correct option, which would have degraded the answer: 145 distractors were rewritten, one at a time, until they were as complete as it is. A distractor that used to stop one step short now goes all the way, down the wrong path — the question got harder for the right reason. 5 questions are out of reach of this method (MOD-068, VIZ-002, VIZ-062, GOV-003, GOV-036): the correct option is the name of a function or feature, too short for a longer distractor to fit without becoming absurd.

5 questions out of 600 still have an option that genuinely stands out (1 of them the correct one, 4 a distractor), and using the heuristic by reading scores 25.5% — the same as guessing. Not taking our word for it is the correct response: download the script and the data and run it yourself.

Check it yourself

Download both files and run it. No dependencies: Python 3.8 and the standard library.

python3 auditar.py metrica_en.json

The metrics file contains, for each question, only the length of each option and which one is correct. No stems, no option text. The three tests only look at length and position, so the result is identical to running it against the closed bank — you can audit it without the bank becoming public domain.

If you find a bias our tests miss, write to us. The script is public domain: run it on whatever question bank you are using today. You will probably not enjoy the result.

Download the script (auditar.py) PT data (metrica_pt.json) EN data (metrica_en.json)
Take the free diagnostic
← Back to the main page