Are You Really Sure That Is What The Research Shows?

Written by a human, for humans, always.

Sometimes...

Theory does not work in reality.

Sometimes...

What you hear is not true.

Sometimes...

The wrong motivations can change how the world becomes perceived.

The theory of science is meant to be one of honor...

One that helps us find and reveal the truth.

But...

Not everything done in the name of science does this in truth.

Now...

When I say this, I am not saying that everyone who dedicates themselves to the pursuit of science has ill intentions.

Most people have good intentions...

Even if there are a few snakes and rotten apples among them.

What I am saying...

Is that the wrong motivations can change how we do things...

And that can skew what we perceive as reality.

In scientific research, that motivation is the need to get published.

If your research does not get published...

You usually no longer have a career.

What this means...

Is that those who pursue a career in scientific research...

Have to tailor what they pursue in order to get published.

They need to fit the research they do to whatever the requirements are to get published.

If the research does not fit...

Then it does not get published...

And the individuals no longer have a career.

What makes this more difficult...

Is that research needs funding.

Only certain types of research get funding...

Meaning only certain types of research have the potential to be tested.

Further...

Even decent research takes time, effort, and resources to conduct...

Without knowing what the results will be.

You could spend a month or longer conducting research...

To discover that what you are testing does not fit into what can get published.

This can lead to a career being destroyed.

But...

The thing about research...

Is that it usually involves statistics.

The thing about statistics...

Is that if you take the right numbers and put them in the right places...

You might be able to rescue all the efforts that were done...

By showing "some" result from the research that is just enough to become published.

In research, this is called P-Hacking.

A "P-Value" in scientific research is a bit like a border...

Anything lower than a certain P-Value (often 0.05) means that it is "statistically more likely" that the result was not accidentally obtained...

i.e., there was not a different explanation for why the experiment found those results, or that a different result would be found with a different group of people.

But here is the thing...

That border is not a "real" thing...

It is simply a practically chosen border as a way to attempt to create consistency.

You could have a P-Value of 0.051 and have it turn out to be true...

And you could have a P-Value of 0.049 (or lower), and it turn out to be false.

But that difference can be the difference between being published and not published...

Between keeping a career and losing a career.

This is where P-Hacking comes in...

If you do a test...

And what you test ends up showing a result of "greater than" the P-Value cutoff...

But...

If you could change a few numbers of what was being tested to get "under" the cutoff...

You might be able to rescue the research that was done.

Let me give you an example.

Say you did a sample of people and, within the total sample of people, the result was a P-Value of 0.051...

Your research would then be significantly less likely to be published.

But...

If you look at the numbers and determine that you "could" split the total sample into two subgroups...

And if you "removed" one of those subgroups...

The remaining subgroup would have a P-Value of 0.049...

Suddenly, you have research that could be published.

It is not necessarily lying...

It is skewing the numbers to tell something that is technically true...

Based on a technicality...

That may not be disclosed in the most-read sections of a research paper (The abstract, introduction, and conclusion sections)...

And only is sometimes hidden in the other sections of the research paper (The results, methods, and discussion sections).

Now you may be wondering...

How often does this happen?

As a research paper on the topic says...

"The fact that we find evidence for p-hacking when using p-values from either the Abstracts or the Results sections across all scientific disciplines for which data are available (our overall analysis) supports the conclusion that p-hacking is rife."

What types of fields did they look at and find this type of skewing of data?

Biological sciences...

Chemical sciences...

Medical and health sciences...

Psychology and cognitive sciences...

Information and computing sciences...

Engineering...

And Multidisciplinary research.

Which means...

Most of the research industries that you hear about.

Now...

There is an argument to be made that there is a way to overcome this problem...

Known as Meta-Analysis.

Essentially, this is a type of research paper that combines other research papers to see if what is being tested is consistent.

To this point...

The mentioned research paper looked at this as well...

And they said that it does appear that when Meta-Analyses are done well...

This seems to be valid...

That the weight of the P-Hacking is decreased, and we see a more valid result from the overall research.

However...

Also, as mentioned in this research...

There are two potential flaws.

The first is the question of whether the Meta-Analysis is being done honestly...

Whether it is actually pulling from "all" of the research on a topic, or if it, too, is being selective in the research it is pulling from...

Or whether there are enough unbiased and unskewed research papers within a Meta-Analysis in the first place.

(Which, based on different research mentioned in the above research, it has been "estimated that up to 37% of meta-analyses of clinical trials reporting a significant mean effect size represent false positives", so flawed Meta-Analyses are possible and likely.)

The other problem...

Is whether there is enough research to even conduct a Meta-Analysis.

See...

Very often, research is not re-tested or re-verified...

And if a paper has an interesting enough claim...

People may design entire careers around the research as it enters into the public domain...

Getting turned into books and discussed online without being verified at a deeper scientific level.

This has been something that has plagued the world in what is known as "Pop-Psychology"...

And some of the most referenced psychological research has been re-tested "after" it has become extremely mainstream...

Only to be discovered to be unreplicable...

But it is too late at that point.

The beliefs have gotten so widespread and repeated that they fall into the "Illusory Truth Effect"...

Where we believe false information to be true simply because we have heard it so often.

It also does not account for what is referred to as the "File Drawer Effect" or "File Drawer Problem"...

Where research that does not "fit" for whatever reason...

Such as disagreeing with something that is overwhelmingly held as true...

Or research that could lead to a company or journal losing funding...

Suddenly "disappears" (aka, the "File Drawer")...

Perhaps never becoming seen in the light.

What does all of this mean?

It simply means...

That science is not as cut-and-dry as the theory would like us to believe.

That the reality of science and research is much messier than people like to believe.

That if you hear that "this is what the research shows"...

You have the right to question whether that is really what the research shows.

How do they know?

What research are they using?

What parts of the research are they reading and referring to?

Is there reason to suspect tampering could happen?

Is there research that might claim otherwise?

Has any of that research been included and taken seriously in the claims being made?

Could there be any other potential conclusions drawn from the research?

If we look at other potential conclusions, can any of them be more strongly suggested than what the research decided to focus on for the purposes of getting published?

Before we come to the conclusion that we are certain of what the research shows...

Perhaps we should consider whether what we believe to be true has been looked at closely...

Or whether we are following a narrative that has come about because of skewed science.

I am not saying you should go full "conspiracy theorist" mode...

That there is no good research that exists in the world and that nothing is true...

But there is enough evidence of skewing to suggest...

That much less of the research that we see actually means what we have come to believe it means...

And that we should consider evaluating it all more closely.

Are You Ready to Go Beyond Leadership?

Tired of Broken Algorithms and AI Slop?

Excited to Dive Deeper into Psychophysiological Mastery?

Want to Change The World?

The Seeking Sageship Newsletter is for You!

Click Here to Subscribe for Free!