Although there are some red flags that may tip you off to poor scholarship or propaganda disguised as science, sorting the good from the bad when it comes to educational research can be daunting. This topic is important for practitioners and scholars alike. In fact, the most recent issue of Educational Researcher focuses specifically on the question of what should count as high-quality education research.So, while the conversation about judging research quality continues, how can you tell if the next study that comes across your desk is worthwhile or belongs in the garbage bin? Let’s start with the easy stuff. Remember those red flags I mentioned? Be very cautious when:
It gets trickier when errors and bias in research are subtle or technical. Although not a guarantee of every aspect of quality, research that appears in reputable, peer-reviewed journals is generally more credible than articles that appear in other sources. Peer review means researchers who know the field critique and assess an article before it is published to make sure it is accurate, the methods are rigorous, and the topic is relevant to the journal or the publisher. Many government reports are also reviewed by a commission of experts.Typically, the tone of quality social science is circumspect and judicious. Good research points out its own flaws and limitations rather than obscuring them. Desirable practices including drawing conclusions that are supported by data and avoiding grandiosity and hyperbole. In other words, good research values substance over flash, focusing on the steak rather than the sizzle.I feel pretty confident saying that to date, there has never been a perfect piece of social science. Bias can enter into a study at any point in its development. What research questions are asked and how they are answered depends on how someone thinks about the issue. Good social science does its best to limit and acknowledge those sources of bias rather than pretending they don’t exist.
A number of people have compiled good checklists of questions for judging the quality of research. Some of the questions that frequently appear on these lists include:
So take these things into consideration then next time you come across a piece of research. Also, for a good example of bad science, take a look at The Dangers of Bread. This hilarious send-up is a great way to see how easily the trappings of science and statistics get abused to prove a point. Wendy Vaulton is a researcher with the Lesley University Center for Reading Recovery and Literacy Collaborative. Learn more about the Center's research-based professional development programs.