Campbell’s Law: Why your metric will be gamed

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

–Donald Campbell, 1979

Campbell, originally an experimental psychologist and trained in experimental method as was customary in his field, soon realized that true experiments could not be done in any of the social sciences because no one would let social scientists treat human beings the way laboratory scientists treated rats and other experimental animals. You couldn’t manipulate people that way because they were free enough to reinterpret the conditions of any experiment and because the institutions where experiments were done were sensitive to the public relations, if not always the moral, issues involved.

[…]

An experimenter might choose a condition for the social program to be tested, but the subjects of the experiments–organizations and the people responsible for them–inevitably and quickly understood how the numbers their actions piled up could be used in ways that might help or hurt their interests. And so, just as routinely, did their best to make sure that the numbers came out the way that gave the best outcome from them, manipulating them in ways their organizational positions and knowledge made available to them. Who knew better how to to that? And that’s been a robust finding. It’s what people organizations do, if they can (and usually they can).

–Howard S. Becker, Writing for Social Scientists (find in a library)

Campbell’s Law: Why your metric will be gamed

Measurement: Validity, Reliability, Accuracy (The Basics)

Validity. Data have validity if they accurately measure the phenomenon they are supposed to represent.

Reliability. Data have reliability if similar results would be produced if the same measurement or procedure were performed multiple times on the same population.

Accuracy. Data are accurate if estimates from the data do not widely deviate from the true population value.

So basic, but so important.

From the National Science Foundation – Science & Engineering Indicators 2018 Methodology.

Measurement: Validity, Reliability, Accuracy (The Basics)

Figures are not always facts

The Woman and the Hen

A Woman had a Hen that laid an egg every day. The Fowl was of superior breed, and the eggs were very fine, and sold for a good price. The Woman thought that by giving the Hen twice as much food as she had been in the habit of giving, the bird might be brought to lay two eggs a day instead of one. So the quantity of food was doubled. The Hen thereupon grew very fat, and stopped laying altogether.

Aesop
Aesop’s Fables (find in a library)

Figures are not always facts

What are you measuring?

The gross national product does not allow for the health of our children, the quality of their education or the joy of their play. It does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials.

It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short, except that which makes life worthwhile.

—Robert F. Kennedy

What are you measuring?