Understanding Research - Statistical Significance
I decided that, during my vacation, you’ll get a series of short posts, labeled “Understanding Research”.
Each post will be about a specific keyword or concept that might not be universally known or often misunderstood.
Let’s do this! The word for today is Statistical Significance.
Scientists have found out that …
We’ve all read that sentence more than once. It can usually be found in some news article describing the latest discoveries of some researchers. Sometimes they even link the paper in their sources, sometimes they don’t. But did you ever click the paper? Did you ever check if the findings have any statistical significance?
I doubt it if you’re not involved in research yourself. But to have an informed opinion about a subject it’s important to be able to understand the research.
Let’s have an example:
There are three people, Mary, John, and Peter. An alien observes them for a year and does experiments, not knowing anything about humans. When the year ends, the alien writes a paper which contains the following statement.
During the 12 months I observed the test subjects I noticed that one of the subjects, Mary gets violently sick every time she consumes any product made from a cow’s milk while Peter and John can drink it without any issues. I thus conclude that female humans are not able to safely drink cow’s milk while male humans are.
See, that doesn’t make sense. The alien took three random humans and by chance, one of them was lactose intolerant and female. A number of three test subjects is clearly not sufficient. But what is?
When collecting results, you always have to assume that a number of those results is totally random. The goal is to collect so many results that those random outliners are visible as just that – random outliners.
So the bigger the reference group, the bigger the chance it isn’t all just a coincidence.
If you’re reading a paper where they only tested about 30 people, it’s often less trustworthy than a paper where they tested 100,000 people. Sure, the second paper can have other faults but it’s an easy way to get a first idea about how trustworthy the results are.
Don’t just believe what some journalist tries to break down for you. Read the source if available.
Read More:
Understanding Statistical Significance
Statistical Significance – Wikipedia
Got a scientific topic which you want to see as a story? Leave me a comment!
You want to support scientists on Steemit? You are a scientist on Steemit? Join the #steemSTEM channel on steemit.chat and connect with us!
STEM is an acronym for Science, Technology, Engineering and Math
I used to think that the mainstream media's coverage of science and new discoveries were as poor as it could get with regards to including references and noting statistical significance, its external validity and whether or not it had been subject to peer review, etc. But then I got familiar with the #science tag on steemit :P
Keep it up! I think we'll need, at one point, to bundle together posts like these in a "steemSTEM guide to critical reading of science-posts on the web". ;)
The science tag on steemit is a joke. A sad joke :(
It'll fall on us to make it better over time then ;)
The beauty, and hazzle, of a decentralized platorm is exactly that it won't improve unless we make it so. But if you need some source of motivation, then it can be useful to look back at the early stages of YouTube whose cotnent was characterized by the dominance of youth 12-17 years old ;)
But Youtube still sucks 😭
Well, we'll take on the challenge to make it better than YouTube then. I'm in ;)
But getting to the point where you can type in a subject you're interested in through the search engine and find something at least remotely close to what you were looking for would be a good start :'D
True 😂
We will change it, bit by bit, post by post!
That is why the "mute" button is very useful :)
True
Good way to explain this, thank you for it.
And secondly, I've seen the problem with journalist trying to explain something based on a study. The problem is though, that other medias only copy what the first journalist wrote and never look at the source material. Then in the end, they might not even provide any information on what study the articles are based on.
You should always check the source study or studies.
I'm also an anti vaxxer, meaning person who is against vacations. Say no to @suesa and her positive approach on vacations!
That is the best sentence I have ever read on anti-vaxxing :)
Thank you, I'm doing my best trying to tell people the dangers of vacations. I haven't had any vacations for a long time myself and I'm feeling better than ever. If everybody knew, nobody would have vacations and we'd all live in a better world!
Vacations are evil corporations plot to keep us off from work!
I am actually more tired due to vacation than due to work ;)
That's something really common. The same happens on weekends: On Friday you are at your best, but on Monday you'll be exhausted and tired.
The same thing happens on vacations. For some reason, you'll get more and more tired during a vacation. Especially if you have special reasons like kids. When I'm getting back to work from a vacation with kids, I'm resting at work.
As crazy as it is.
I actually totally understand this (and this is what I internally had in mind with my first message). Kids are like vampire, emptying the energy pool at huge speed. ^^
I thought you might have kids and understand :D
Kids do empty the energy pool fast. But they make parents so happy so easily too:
Only by sleeping, they make parents extremely happy ;)
When kids sleep, parents... sleep too, trying to recover at a much smaller pace :p
To death with vacations!
Say no to vacations! In the past, people didn't need vacations so they lived healthier and didn't get fat.
Vacations can ruin your natural lifestyle and ruin your life.
Your post makes a lot of sense. The same method is adopted by those who predict election results and unfortunately a lot of people get swayed by it. The result is for all to see in one of the most important countries in the world.
I think every piece of research on any subject must be backed up with the sample size and details of the sample that answers the questions how, what, why and where.
Following for more.
Its quite subjective in nature when it comes to testing. We see this alot in medical field, where a certain drug gets approved and after years of usage they finally ban due to health reasons.
Very often, the outcome can take time to show its effects on physiology and we cannot conclude with short term results (we are all not wired in same manner)
I love the milk example! I will reuse it for my own students :)
Woah! I feel honored. It took me forever to come up with it and I was afraid it's dumb.
It is very simple and funny. In short, something students will remember.
Thank you for this post, @suesa! :) This topic needs a lot of awareness.
I can say from experience, that the problem of "significance" goes a lot deeper than this. If you work in a life science like e.g. biology, "significance" is a tricky thing. I see people applying the standard error when they only have a very small number of samples, same goes with the abuse of the beloved p-value. If you repeat something only 3 times (the standard in biological research in many places), the p-value is pretty much useless. But due to timepressure and money, experiments are often repeated only 3 times, as more would be too costly in either way. A dilemma, which causes the need for even closer observation, as even significant p-values are often misleading. So the whole set of experiments has to be considered together, and to do the right experiments, it takes a very skilled researcher a ton of knowledge. Quite frustrating, especially for young and aspiring scientists!
Nice idea. My friend recently sent me a link to a paper, where there had been tested only 20 people ;p I'm gonna send him a link to your entry ;D
but I think vacation is one of the things that will absorb some new inspiration, and vacation makes the time or moment to give break room for the tired mind of drained brain energy while working continuously, but vacationing in a long time will be a bad thing to our job .. thanks
What
That is, I really agree with the holidays, because I like the holidays but not in a long time ... thanks @suesa ..! hopefully @suesa understand what I mean
This is an uncommon case of quality based on quantity. Have a safe trip mistress :)
#StrengthInNumbers
Exactly!