The creator of Total Quality Management (when it was cool) came up with an experiment to show why worker ratings were bogus.  He understood this thirty years ago.  It is no longer politically correct, though, for it prevents managers from punishing doctors for expensive healthcare.  He came up with “the red bead game” to explain his ideas.

What is the point of the red bead experiment?

  1. The experiment provides a typical illustration of bad management. There are too many employees involved (e.g., inspectors), and the rigid procedures do not allow workers to offer suggestions for improvement. In addition, during the experiment, management continually blames the workers for defective products that are caused by the system.
  2. System variation (frequently referred to as random variation) is inevitably present in any process, operation or activity.
  3. Knowledge of one source of system variation, such as the proportion of defects (red beads) in the incoming supply, cannot be used to determine the total effect of system variation, such as the proportion of defects in the output. This is because unobservable factors will always affect performance and there is no basis for assuming that the effects of these factors will be equally distributed across workers.
  4. All workers perform within a system that is beyond their control.
  5. There will always be some workers that are above the average and some workers that are below the average.
  6. A worker’s position in the ranking may vary from one period to the next.
  7. Workers should not be ranked because doing so merely represents a ranking of the effect of the system on the workers. In the red bead experiment, 100 percent of the variation in the workers’ performances is determined by the system. Even in this controlled experiment where the workers use the same inputs and tools, they are all victims of the system and cannot be compared in any meaningful way.1
  8. Only management can change the system.
  9. Empirical evidence (i.e., observations of facts, as opposed to secondhand information, or information further removed from fact such as opinion) is never complete. There are always a large number of variables that affect any set of performance results, many of which are unknown and unknowable.
    The red bead experiment is deceptively simple because it provides a powerful message that is difficult for many to grasp. In summary, the misconception that workers can be meaningfully ranked is based on two faulty assumptions. The first assumption is that each worker can control his or her performance. Deming (1986, 315) estimated that 94 percent of the variation in any system is attributable to the system, not to the people working in the system. The second assumption is that any system variation will be equally distributed across workers. Deming (1986, 353) taught that there is no basis for this assumption in real life experiences. The source of the confusion comes from statistical (probability) theory where random numbers are used to obtain samples from a known population. When random numbers are used in an experiment, there is only one source of variation, so the randomness tends to be equally distributed. This is because samples based on random numbers are not influenced by such things as the characteristics of the inputs and tools (e.g., size of the beads and depressions in the paddles) and other real world phenomena. However, in real life experiences, there are many identifiable causes of variation, as well as a great many others that are unknown. The interaction of these forces will produce unbelievably large differences between people (Deming 1986, 110) and there is no logical basis for assuming that these differences will be equally distributed.