Use science more gracefully, in business and life, via the ghost of Carl Sagan
In his seminal book, The Demon-Haunted World, Carl Sagan advocates for utilizing the Scientific Method as a way to improve our thinking.
- “The method of science, as stodgy and grumpy as it may seem, is far more important than the findings of science.”
That is a powerful statement because the findings have been revelatory. Although it’s the system of thinking that produces the findings, so Carl seeks to teach us this:
- The scientific way of thinking is at once imaginative and disciplined. This is central to its success. Science invites us to let the facts in, even when they don’t conform to our preconceptions. It counsels us to carry alternative hypotheses in our heads and see which best fit the facts. It urges on us a delicate balance between no-holds-barred openness to new ideas, however heretical, and the most rigorous skeptical scrutiny of everything — new ideas and established wisdom.
Today, we are far removed from these core principles. On too many occasions, “Science” has become ideological and used as a weapon in the war of ideas. Sagan’s scientific teeter-totter of balance between being imaginative and disciplined is lopsided towards too much discipline.
His core principles for the scientific way of thinking, “heretical openness to new ideas” and “skeptical scrutiny of established wisdom”, are as difficult to find as rubber gloves during a virus outbreak.
Although, there’s no shortage of Scientific terminology out there. Nowadays, you can find the word “hypothesis” all over the business world. It has been resurrected from your 4th-grade science class, driven by this century’s wave of tech startups.
This hypothesis word has many definitions, so for alignment purposes, let’s define a hypothesis as:
- a data-informed idea, that can be proved or disproved and is a starting point for learning.
Let’s see what Carl thinks:
- But the history of science teaches that the most we can hope for is successive improvement in our understanding, learning from our mistakes, an asymptotic approach to the Universe, but with the proviso that absolute certainty will always elude us.
The keyword here is learning.
In both our personal and business lives, we can begin rephrasing every “idea” as a hypothesis meant for learning. When you do this, you dispel authoritative ideas from a boss, expert, husband, mother, or experienced person and encourage a more democratic approach.
To change your culture from the most senior person's opinion to a culture of learning, I encourage this rephrasing of ideas to hypotheses. Next time you hear an idea in a meeting, respond with, “that’s a great hypothesis, we should test it.”
If you are already a part of this hypothesis-using culture, you may be “Science-d” out by now and seeking to harness your inner Billy Madison. There may be well-intentioned colleagues questioning your heretical ideas, your small sample sizes, and demanding statistical significance before making a decision.
This far-too-disciplined approach has caused unintended consequences. The biggest unintended consequence is the suppression of creativity.
In cultures like this, we begin facing the same imbalance that Sagan said to look out for. When our teeter-totter of balance is lopsided towards discipline, we lose out on imaginative thinking and therefore lose out on creativity.
And, the irony is that our business leaders are yearning for more innovation, so they’ve invested in experimentation, but they are still not satisfied with the results and are therefore still yearning for more innovation.
Experimentation is a great ̶i̶d̶e̶a̶ hypothesis towards increasing innovation as long as Sagan’s teeter-totter is balanced.
The experimentation hypothesis begins to fall apart when we become too disciplined, or when it focuses too much on tools and technology, instead of a way of thinking.
The business world has erupted with scientific approaches in the form of conversion rate optimization (CRO), A/B testing, Personalization, etc., which are well-intentioned but can slowly become the holy grail of knowledge.
If you don’t have data that is 95% statistically significant then you lose credibility. Changes to your website, your email blast, or your ad copy, now need an experiment that has reached 95% statistical significance.
- “Statistical significance is important because it gives you confidence that the changes you make actually have a positive impact on your metrics.” -optimizely.com
This disciplined approach can begin spiraling out of control. Learning can take a back seat and be replaced with increasing a metric to 95% statistical rigor.
To emphasize this point, I’ll tell you a personal story when I consulted a large taxi-hailing company. They came to us with a desire to improve their tag line for getting more drivers to sign up (i.e. Earn $XYZ guaranteed just driving in 6 months). Their current tag line was coveted as “the most successful” one and attempts to dethrone it was futile, so they hired consultants to help.
Instead of continuing down the path they recommended, which was surveying massive sample sizes to get statistical significance, I engaged 10 prospect drivers in a moderated usability study. I introduced them to a scenario that included the whole journey from their coveted ad copy to a landing page, and through to signing up.
The qualitative insights derived from this not-so-science experiment allowed us to understand the humans interpreting it, which inspired a more holistic approach. I learned that the ad copy doesn’t matter as much as the driver’s end-to-end journey from ad copy to a landing page to signing up. Most drivers didn’t even notice a difference in the ad copy’s messaging, which was where the obsessed focus was.
Not only did we get deep insights, we learned why.
On too many occasions, we seek innovation by narrowing our focus and imposing force in our research, but major innovation comes from curiosity-driven research as Sagan advocates:
- Basic research is where scientists are free to pursue their curiosity and interrogate Nature, not with any short-term practical end in view, but to seek knowledge for its own sake. This is how the major discoveries that benefit humanity are largely made. The history of science shows that often you can’t go after the underpinnings of knowledge in a directed way. They may emerge out of the idle musings of some person. Urging major practical inventions while discouraging curiosity-driven research would be spectacularly counterproductive.
Getting back to the Optimizely quote from above, it hails statistical significance as important because it says it gives you confidence but what it fails to mention is the influence of human fallibility.
This is one thing that the medical scientific community has begun to learn the hard way. That — statistical significance is useless if you have poor experiment design.
For example, the commonly known statin drug taken by 40+ million American adults has been shown to reduce the incidence of death from heart disease. This sounds great, on the surface. However, this decrease was almost entirely negated by a corresponding increase in cancer deaths. As a result, overall mortality between the statin and placebo groups over three years was nearly identical.
Then, there’s the medical community’s reliance on surrogate markers as an indication for efficacy. For example, a surrogate marker (e.g. reducing tumor size) is often used as a replacement measure of success for a longer-term measure (e.g. all-cause mortality).
Or, there’s the infinite inability to control for every variable.
And, a quote from Malcolm Kendrick that brilliantly summarizes these points:
- We have found that when patients with cancer are pushed off a high cliff this reduces mortality from cancer deaths by 100%. This represents an unprecedented reduction in cancer mortality, and we suggest this technique might be used to reduce cancer deaths around the world. I would follow this with my seminal study on ‘Removing the human brain to prevent strokes.” — from Doctoring Data: How to sort out medical advice from medical nonsense
Thus, statistical significance can certainly give us confidence, but it might be deluded confidence.
In business, within some over-disciplined CRO and A/B testing circles, there is a similar scenario like the one found in the medical community, and it is illustrated by the image below of Optimizely’s reported experiment win rates based on the type of metric.
As you can see, surrogate markers (clicks, events, page views, engagement) are utilized for measures of “success” in replacement of more long term measures (Revenue, Retention, Loyalty, Lifetime Value).
If we choose a surrogate marker and have yet to show a causal relationship to a longer-term measure like revenue, then we are persisting in deluded confidence.
And, what‘s worse are the stories we tell for why this surrogate marker was increased. These stories replace real learning and are littered with assumptions. The false-positive we sought out to remove by attaining statistical significance ends up being a false positive itself.
- For me, it is far better to grasp the Universe as it really is than to persist in delusion, however satisfying and reassuring. -Carl Sagan
By now, artists may be pumping their fists in agreement, but problems also arise the other way. I purposely focused on the imbalance of discipline, because that is the current problem in the 20th & 21st centuries. If you want to see what happens when we are too imaginative, you can read Sagan’s book, it’s the reason he calls it “The Demon-Haunted World.”
By summoning the ghost of Carl Sagan, we can return to a more balanced approach, and get the confident results we are desiring for.
