Campbell’s Law: Why your metric will be gamed

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

–Donald Campbell, 1979

Campbell, originally an experimental psychologist and trained in experimental method as was customary in his field, soon realized that true experiments could not be done in any of the social sciences because no one would let social scientists treat human beings the way laboratory scientists treated rats and other experimental animals. You couldn’t manipulate people that way because they were free enough to reinterpret the conditions of any experiment and because the institutions where experiments were done were sensitive to the public relations, if not always the moral, issues involved.

[…]

An experimenter might choose a condition for the social program to be tested, but the subjects of the experiments–organizations and the people responsible for them–inevitably and quickly understood how the numbers their actions piled up could be used in ways that might help or hurt their interests. And so, just as routinely, did their best to make sure that the numbers came out the way that gave the best outcome from them, manipulating them in ways their organizational positions and knowledge made available to them. Who knew better how to to that? And that’s been a robust finding. It’s what people organizations do, if they can (and usually they can).

–Howard S. Becker, Writing for Social Scientists (find in a library)

Campbell’s Law: Why your metric will be gamed

Why Management Science Fails to Perform, according to Peter Drucker

Parts exist in contemplation of the whole.

There is one fundamental insight underlying all management science. It is that the business enterprise is a system of the highest order: a system whose parts are human beings contributing voluntarily of their knowledge, skill, and dedication to a joint venture. And one thing characterizes all genuine systems, whether they be mechanical like the control of a missile, biological like a tree, or social like the business enterprise: it is interdependence. The whole of a system is not necessarily improved if one particular function or part is improved or made more efficient. In fact, the system may well be damaged thereby, or even destroyed. In some cases the best way to strengthen the system may be to weaken a part–to make it less precise or less efficient. For what matters in any system is the performance of the whole; this is the result of growth and of dynamic balance, adjustment, and integration, rather than of mere technical efficiency.

Primary emphasis on the efficiency of parts in management sciences is therefore bound to do damage. It is bound to optimize precision of the tool at the expense of the health and performance of the whole.

Landmarks of Tomorrow
Management: Tasks, Responsibilities, Practices

Why Management Science Fails to Perform, according to Peter Drucker

Evicted: Matthew Desmond’s Pulitzer Prize-Winning Ethnography of Tenants, Landlords, and Eviction in an American City

I’ve always felt that my first duty as an ethnographer was to make sure my work did not harm those who invited me into their lives. But this can be a complicated and delicate matter because it is not always obvious at first what does harm.

With all the talk of data science, big data, and computational modeling, it’s increasingly important to highlight exceptional examples of rigorous research employing different methods, as these methods are no less important in our quest to better understand human social systems.

Perhaps the best book I read in 2018 was Matthew Desmond’s Pulitzer Prize-winning ethnographic study of tenants and landlords, Evicted: Poverty and Profit in the American City (find in a library).

In “About This Project,” Desmond details how his ethnographic study ultimately led to a mixed-methods research inquiry. Desmond describes designing a survey, the Milwaukee Area Renters Study (MARS), to assess the experience of renters in the Milwaukee rental market. He notes that his measurement (i.e., survey) items were greatly influenced by what he learned during his ethnography, which, in my experience, is a critical feature of good survey research – qualitative inquiry driving quantitative measurement (and vice-versa). He noted, for example, that simply asking, “Have you ever been evicted?” is likely to undercount evictions, since “eviction” connotes sheriffs and courts for many of the respondents, and a better measurement item would assess the lose of a rental home due to nonpayment or for other reasons.

The multiple methods and different data sources used in this book informed one another in important ways. I began this project with a set of questions to pursue, but lines of inquiry flexed and waned as my fieldwork progressed. Some would not have sprung to mind had I never set foot in the field. But it was only after analyzing court records and survey data that I was able to see the bigger picture, grasping the magnitude of eviction in poor neighborhoods, identifying disparities, and cataloguing consequences of displacement. My quantitative endeavors also allowed me to assess how representative my observations were. Whenever possible, I subjected my ground-level observations to a kind of statistical check, which determined whether what I was seeing on the ground was also detectable within a larger population.

Desmond also highlights the importance when conducting qualitative research that information be verified whenever possible through alternative sources and, in particular, using official records such as those collected by social services and the courts.

I analyzed two years’ worth of nuisance property citations from the Milwaukee Police Department; obtained records of more than a million 911 calls in Milwaukee; and collected rent rolls, legal transcripts, public property records, school files, and psychological evaluations.

The two surveys that Desmond designed following his fieldwork both achieved very respectable response rates: 84 percent for the MARS survey and 66 percent for the Milwaukee Eviction Court Study.

Desmond was also clear when he noted, in multiple places, his own involvement in the events that he was studying. He describes two instances, in particular, in which he provided funds for the rental of a U-Haul truck and a loan to a mother to purchase a stove and refrigerator in advance of an anticipated visit by Child Protective Services. He also explained that he occasionally provided transportation for individuals looking for housing.

Researchers, particularly those working in field settings–which includes organization scientists–rarely seem to do as good a job as Desmond in examining potential biases introduced by the researcher’s mere presence. In survey methodology training, we’re explicitly taught to understand how the presentation of measurement items can affect response data – whether the survey is incentivized or not and if so, what type of incentive is used (overincentivizing survey participation, for example, will generally reduce the quality of the data); whether surveys are presented electronically, on paper, or by a field interviewer; and even the colors and fonts used when presenting items to respondents.

In light of what we know about survey measurement, it’s a tall order to disentangle and fully understand the bias introduced by a researcher doing ethnographic fieldwork, so I was pleased that Desmond did so in Evicted, and did so in an accessible and highly engaging way (in “About This Project.”)

Quantitative social scientists could learn a great deal from our colleagues with more experience using qualitative methods and inquiry.

Desmond practices open science and promotes re-use of his data:

I have made all survey data publicly available through the Harvard Dataverse Network.

And he suggests that other researchers must attempt to replicate his extensive findings in other geographic areas:

That said, it is ultimately up to future researchers to determine whether what I found in Milwaukee is true in other places. A thousand questions remain unanswered. We need a robust sociology of housing that reaches beyond a narrow focus on policy and public housing. We need a new sociology of displacement that documents the prevalence, causes, and consequences of eviction. And perhaps most important, we need a committed sociology of inequality that includes a serious study of exploitation and extractive markets.

Yet Desmond questions what, in the context of a human socio-economic system like landlords and tenants, we actually mean when we talk of “generalizing” findings or replicating them elsewhere:

Still, I wonder sometimes what we are asking when we ask if findings apply elsewhere. Is it that we really believe that something could happen in Pittsburgh but never in Albuquerque, in Memphis but never in Dubuque? The weight of the evidence is in the other direction, especially when it comes to problems as big and as widespread as urban poverty and unaffordable housing. This study took place in the heart of a major American city, not in an isolated Polish village or a brambly Montana town or on the moon.

Finally, Desmond describes the power of storytelling in conveying research:

Ethnographers shrink themselves in the field but enlarge themselves on the page because first-person accounts convey experience—and experience, authority.

While the product of Matthew Desmond’s extensive ethnographic fieldwork, follow-up research, and synthesis stands on its own and should be read by every social scientist in the U.S., I cannot do better than to close with Desmond’s own words at the end of his methodological documentation, revealing the intense interplay between researcher and subjects in any ethnography:

The harder feat for any fieldworker is not getting in; it’s leaving. And the more difficult ethical dilemma is not how to respond when asked to help but how to respond when you are given so much. I have been blessed by countless acts of generosity from the people I met in Milwaukee. Each one reminds me how gracefully they refuse to be reduced to their hardships. Poverty has not prevailed against their deep humanity.

I highly recommend Matthew Desmond’s Evicted: Poverty and Profit in the American City (find in a library).

Evicted: Matthew Desmond’s Pulitzer Prize-Winning Ethnography of Tenants, Landlords, and Eviction in an American City

Organization Design: “All the elements interact in a system”

Organizations, like individuals, can avoid identity crises by deciding what it is they wish to be and then pursuing it with a healthy obsession.

Some organizations do indeed achieve and maintain an internal consistency. But then they find that it is designed for an environment the organization is no longer in. To have a nice, neat machine bureaucracy in a dynamic industry calling for constant innovation or, alternately, a flexible adhocracy in a stable industry calling for minimum cost makes no sense. Remember that these are configurations of situation as well as structure. Indeed, the very notion of configuration is that all the elements interact in a system. One element does not cause another; instead, all influence each other interactively. Structure is no more designed to fit the situation than situation is selected to fit the structure.

The way to deal with the right structure in the wrong environment may be to change the environment, not the structure. Often, in fact, it is far easier to shift industries or retreat to a suitable niche in an industry than to undo a cohesive structure.

Essentially, the organization has two choices. It can adapt continuously to the environment at the expense of internal consistency—that is, steadily redesign its structure to maintain external fit. Or it can maintain internal consistency at the expense of a gradually worsening fit with its environment, at least until the fit becomes so bad that it must undergo sudden structural redesign to achieve a new internally consistent configuration. In other words, the choice is between evolution and revolution, between perpetual mild adaptation, which favors external fit over time, and infrequent major realignment, which favors internal consistency over time.

–Henry Mintzberg, 1981
Organization Design: Fashion or Fit?

Organization Design: “All the elements interact in a system”

When is a system complex?

“Flocking birds, weather patterns, commercial organisations, swarming robots… Increasingly, many of the systems that we want to engineer or understand are said to be ‘complex’. But what does this mean? How do these so-called ‘complex systems’ differ from the more easily understood systems that we are familiar with?”

Visit: http://complexityprimer.eng.cam.ac.uk for more on complexity and modularity.

When is a system complex?

They’ll have your attention…

In an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information resources that might consume it.

In an information-rich world, most of the cost of information is the cost incurred by the recipient. It is not enough to know how much it costs to produce and transmit information: we much also know now much it costs, in terms of scarce attention, to receive it. I have tried bringing this argument home to my friends by suggesting that they calculate how much the [news] costs them, including the costs of [consuming] it. Making the calculation usually causes them some alarm, but not enough for them to cancel their subscriptions.

–Herbert A. Simon
“Designing Organizations for an Information-Rich World” (1971, PDF)

They’ll have your attention…

Do children and money really bring happiness?

The belief-transmission game is rigged so that we must believe that children and money bring happiness, regardless of whether such beliefs are true.

This doesn’t mean that we should all now quit our jobs and abandon our families. Rather, it means that while we believe we are raising children and earning paychecks to increase our share of happiness, we are actually doing these things for reasons beyond our ken.

We are nodes in a social network that arises and falls by a logic of its own, which is why we continue to toil, continue to mate, and continue to be surprised when we do not experience all the joy we so gullibly anticipated.

Daniel Gilbert
Stumbling on Happiness (find in a library)

Do children and money really bring happiness?

Measurement: Validity, Reliability, Accuracy (The Basics)

Validity. Data have validity if they accurately measure the phenomenon they are supposed to represent.

Reliability. Data have reliability if similar results would be produced if the same measurement or procedure were performed multiple times on the same population.

Accuracy. Data are accurate if estimates from the data do not widely deviate from the true population value.

So basic, but so important.

From the National Science Foundation – Science & Engineering Indicators 2018 Methodology.

Measurement: Validity, Reliability, Accuracy (The Basics)