What do we trust when research and lived experience conflict? (Part 2)
A few edits made, mainly to captions and alt text, on June 15, 2023.
What Research Can’t Do, Plus a Rule of Thumb
When experimental research and lived experience of disabilities come to conflicting conclusions, what should we believe?
The problem becomes personal for autistic people who research autism — or anyone who researches their own disability.
I think both experimental research and lived experience are valuable. However, they are best suited to answer different questions. We should decide which to believe on a case by case basis, depending on what claim is being made and why.
We’ve seen that experimental research excels at providing precise information about large groups of people. However, there are some things it can’t do well. As we’ll see, these are areas where lived experience excels.
What Research Cannot Do
Limitation #1: Research is designed to understand groups, not individual people in depth.
When you watch people take a standardized test or participate in a study, you can learn a lot about how they think. What do they look at, for how long? How much time does it take them to answer? What do they ask questions about? If they talk aloud, how do they explain their answers?
Researchers analyze this data by combining the data from all the individual participants and doing statistical procedures with it. Statistical analysis makes hypotheses falsifiable, which is part of what makes something science.
Statistics are based on the assumption that the people being tested are representative of a larger population. They show how likely it is that differences between groups are real, not just due to chance. [1] Statistics let you conclude what is probably true for the whole population from the behavior of a much smaller group of people. [2]
Limitation #2: Research can only investigate the alternate explanations we choose to test.
However, there are limits to this strength: researchers can’t eliminate alternative hypotheses they haven’t tested.
In research, all we have is a hypothesis until we run an experiment and analyze the results. That’s because we can only perform statistical analysis on data that’s actually been collected. If we haven’t measured something, we can only guess what effect it has.
For example, suppose researchers are investigating how much eye contact autistic children make. The researchers are not interested in studying socioeconomic status or race/ethnicity. However, they happen to test mostly high socioeconomic status, white American children.
The researchers then find that the autistic children they test make less eye contact than their typically developing peers. Does that mean autistic kids make less eye contact than neurotypical ones? Maybe.
But there are potentially infinite alternative explanations to rule out.
For example, would the results be different if the researchers tested families from subcultures where prolonged eye contact is considered rude? Maybe in these cultures, there would be no group difference, or maybe the neurotypical children would make less eye contact.
If the experiment included participants from such cultures, the researchers could look at these children’s results and compare them with American children’s. But when all participants are from a dominant culture where eye contact is polite, we can’t know without doing more research.
Most untested variables are not this obvious. Researchers may not think of them at all. However, we should always keep in mind such variables exist.
Autistic people continually think of alternative hypotheses that researchers hadn’t considered. As a result, they’ve changed the paradigms in autism research. Some examples:
- Autistic people argued that their social difficulties are caused not by “mind blindness,” but by sensory processing and movement differences.
- Milton Damian proposed the Double Empathy Problem explanation of autistic people’s social difficulties: while autistic people have trouble understanding neurotypical people, neurotypical people also have difficulty understanding them.
- Autistic people tend to focus on visual details and not pay attention to the gestalt. That was once attributed to a weakness in integrating information (“weak central coherence”). Michelle Dawson, an autistic researcher, and Laurent Mottron’s team argued that instead, autistic people have a strength in perceiving details. This “enhanced perceptual functioning” is only useful when details are relevant to the task the autistic person is doing.
Autistic people’s lived experience gives them information neurotypical researchers lack. That allows them to consider possible explanations that hadn’t occurred to neurotypical researchers. By proposing new explanations, autistic people make up for one of the great weaknesses of research.
Limitation #3: Research can only answer the questions we think to ask.
When I started following disability research in 2009, the yearly autism research meeting, held by the International Society for Autism Research (INSAR), was dominated by issues that mattered to neurotypical people. These included:
- The genetic mechanisms that cause autism.
- Faster, more accurate diagnosis to allow for earlier intervention.
- Interventions designed to teach autistic people to interact more gracefully or engage in less repetitive behavior.
- Some research on perception and thinking, dominated by weak central coherence theory.
- Research on face processing and social cognition, dominated by mindblindness and similar theories.
On Twitter, autistic people took to the conference chat and called on researchers to change their focus to other, more practically relevant issues:
- Why do autistic people experience anxiety and depression and commit suicide at higher rates than the general population?
- What support do autistic college students need?
- Why are autistic adults underemployed for their education level? Why do they have even lower employment rates than people with other disabilities? What would increase their employment rates and career success?
- How do autistic people process sensory information? How does that affect the way they experience emotions and interact with other people?
- How does autism affect people across the lifespan? What can we learn about adults? Women? Elders? Parents?
- What can we learn about minority populations (race/ethnicity, gender identity, country of origin, native language, etc.)?
- What strengths do autistic people have?
Some of these questions have been taken up by non-autistic researchers.
There’s a reciprocal relationship between research and lived experience. When we make a prediction based on lived experience, we don’t fully know if it’s true — or how well we know — until we do the research. And in order to do the research, we have to come up with the question in the first place.
Limitation #4: Research does not tell us what we should be researching.
In the late 2000s, research on autism and ADHD studied children, not teens or adults. It focused on genetics, prevention, intervention, medication, and the traits listed in the DSM. It explored weaknesses and ignored strengths.
Researchers were just starting to follow autistic people’s advice to ask about life outcomes and sensory processing and motor differences.
In short, researchers asked the questions that parents, teachers, and clinicians wanted answered. They ignored the questions that mattered to autistic people. Why?
It wasn’t because studies had been conducted which showed that autistic people had no strengths.
It wasn’t because any studies showed that autism disappeared by adulthood.
Researchers and the public simply assumed these things so strongly that it never occurred to them to test these assumptions.
Research questions reflected the priorities of those who funded the work and those whose voices were heard.
Autistic people’s concerns weren’t studied until the public started listening to them.
The facts come from science. The values come from us.
When to Trust Research vs. Lived Experience: A Rule of Thumb
If you want to know precise information about people with disabilities, and if you want to eliminate alternative explanations, there’s no substitute for quantitative research.
Research can best address questions like:
- Is autism more common now than it was, say, 20 years ago?
- How similar are autism and ADHD? To what extent do they have the same genetic causes?
- How effective is CBT for treating autistic people’s depression and anxiety? How effective is EMDR for reducing their trauma?
On the other hand, if you want to know about an individual person in depth, you are better off looking to lived experience.
If you want your research to be useful to neurodivergent people, you have to answer questions they care about. To do that, you must consult their firsthand experience.
Loved this story? Hated this story? Got tales of your own to share? Tell me all about it at Mosaic of Minds’ current home on Substack.
Footnotes for Statistics Geeks
This post series was heavily influenced by a class for writers on Reporting Statistics at the University of Chicago.
[1] It is technically possible to make claims about individuals, if one defines the “groups” to be compared differently. You can look at all the data from one individual doing the same thing under different conditions (each repetition is called a “trial”). Those different conditions are the groups to be compared. For example, if you want to learn about reading, you have them read words (or sentences) repeatedly.
However, in order to have enough data to make a statistical claim, you need to have that person do the same task many, many times. That often isn’t practical — it’s time-consuming, repetitive, and boring. Boredom and fatigue actually affect the data you’re collecting by making people slower and less accurate.
A clinician might think it’s worthwhile to spend time and money on experimenting with an individual. A researcher probably wouldn’t think detailed information on one person is worth the money, time, and attention taken away from other experiments. When you generalize to groups, your work is potentially helpful to more people. So, researchers rarely do one-person experiments, even though they’re theoretically possible.
[2] There are different types of statistics that tell us different things about the population of interest. For simplicity, this post focuses on the most commonly used one, classical hypothesis testing. Classical hypothesis testing compares the results of an experiment to the distribution of results that would occur if one’s hypothesis were false. There’s also likelihood estimation, which tells you what statistical model is most likely to be true of the population of interest, given the data collected. Finally, Bayesian statistics measures the strength of your confidence in a hypothesis. It involves framing a precise quantitative hypothesis based on existing data, collecting new data, and adjusting the hypothesis accordingly. (If those words don’t make sense and you’d like to know more about Bayesian statistics, this video and summary might help). Whatever sort of statistics you’re doing, you’re collecting data on a small group in order to learn about a larger population.
