What Does This Study Actually Tell Us?

The media is filled with reports of the findings of the latest study, which may seem to tell us exactly the opposite of the last study we heard about. There are also plenty of products being marketed that are citing various studies to show how effective they are.

Research literacy is having the knowledge to understand and interpret research findings. A little knowledge can go a long way in making sense of what claims may be legit, and what is complete BS. While some media sources may be well informed, others have quite limited media literacy, so it's useful to have some understanding on your own and not have to rely entirely on the media source.

When it comes to medical research, including both prescription and non-prescription drugs as wells as supplements, the gold standard for quality research design is randomized, double-blinded, placebo-controlled trial. The findings should be published in a high-quality peer-reviewed trial. So, what do all of those terms mean?

Peer Review

For a paper to be published in a peer-reviewed journal, it is first screened by the journal editor. The editor will then find independent scholars (typically three of them) who have expertise in that field or research method. Apart from the editor, the reviewers remain anonymous. The reviewers then give feedback to the editor about the quality of the research and the paper. Typically the author will have to make improvements before the paper is accepted. The peer review process helps to ensure that there are no fundamental flaws in the research design or interpretation of results. Higher quality journals are able to attract more prominent researchers to act as reviewers.

Research Design

Details of the research design will depend on the type of research being conducted and the nature of the research question. There are two broad approaches to research: quantitative and qualitative. Quantitative research produces numerical data, and a lot of medical research falls under this umbrella. Qualitative research produces descriptions, which helps to understand things like what it feels like to have a particular illness. One type is not inherently better than the other; it just depends on what type of question is being asked and what form of answers are being sought.

Randomization is an important element of clinical trials. There can be significant person-to-person variability, and randomization between treatment groups helps to prevent unevenness between the groups. If, for example, 100 people were in a study that tested two possible interventions, 50 people would be randomly assigned to one group and the other 50 would be assigned to the other group.

Blinding serves to prevent differences that might arise from people knowing what intervention they were receiving. Blinding can involve either the patient, the researcher assessing their response, or both. Ideally, both should be unaware of what the patient is receiving, and this is called double blinding.

Having a control group helps the researchers understand how much of an effect is actually due to a drug. The most common is to have a placebo control. This will account for people who would have recovered from a condition anyway, as well as people who respond because of getting the placebo. Comparing the response of the group that received the drug to the response of the placebo group shows how much of the effect is due to the drug itself.

Other Research Concepts

There are two ways of looking at risk: absolute risk and relative risk. Absolutely risk refers to the probability of an event happening, while relative risk compares the probability of one event to the probability of another event. If the risk of a harmful health event is 0.05 percent and there is a 100 percent increase in the relative risk of that same event while taking drug X, the absolute risk is still only 0.1 percent. Media reporting will often go with the dramatic number (in this example 100 percent), which should always make you as what the absolute risk is.

Something that is often confused by those who are not well-versed in research is correlation versus causation. Correlation means that certain things tend to occur together, but it does not at all mean that there is a causative relationship. Some research designs are able to establish causation, but with a lot of research strong inferences can't be made regarding causation. To give a somewhat ridiculous example, we could look at a group of people that died and learn that each one of them was breathing before they died. Does that mean breathing was the cause of death? Similarly, children receive vaccines around the age that they may begin to show signs of autism. That does not mean there's a cause and effect relationship.

In general usage significant connotes importance. The term significant is used differently in research. In the context of statistics, significant results are those that are unlikely to be obtained due to chance. This is determined by statistical calculations. The term effect size refers to how large in scale something is. If drug X is being tested, the results are statistically significant if the benefits of drug X over placebo are determined to be unlikely to be due to chance. Descriptions of effect size would look at whether drug X had a 5 percent reduction in symptoms or a 50 percent reduction compared to placebo.

Systematic reviews look at the existing literature on a topic, and set out clear criteria for inclusion and exclusion of studies to ensure they are high-quality and properly address the question at hand. Meta-analyses are similar, but they pool all of the numerical data to crunch the statistics all together.

For most people, reading a full research paper just isn't realistic, but the most relevant details from a study can be found in the paper's abstract. Google Scholar allows you to search research abstracts. Although you would typically have to pay to access a full article, the abstract is available for free. It will tell you the basic details of the study design, plus the most important findings. Abstracts are meant to be easily readable, so with a little bit of research literacy background you'll be well on your way to figuring out what claims are the real deal, and what is just quackery.