Back to The Psych Files

Research Methods in Psychology
Dr. Michael Britt

Research On Trial

Purpose

The purpose of this role-play is to learn more about what makes good research by delving in detail into the "case" that the author(s) of an article try to make about the research they have conducted. The authors of any published research article try to convince the reader of the validity of their conclusions, i.e., the significant results they claim to have found. Consumers of research can not read the results of these articles without keeping a critical eye on how the research was conducted. Should you believe what the author claims? Should important decisions concerning people who are in need be based upon this research? As psychologists, you will be called upon to make such decisions in your work. Developing a knowledgeable, insightful, critical eye is important for psychologists, let alone any educated person.

This simulation is intended to teach and develop skills in being a critical thinker and consumer of research. Since it will be necessary to take liberties with the actual way a courtroom trial is run in order to adapt the scenario to our needs, the simulation will not attempt to teach students about the proper way a courtroom is run.

The Proceedings

1. The Research Article We Will Put On Trial. I will give one research article to eight students. I will ask that four of these students to represent the authors as defense lawyers, and four students serve as lawyers for the prosecution. The other students in class will serve as a jury and hear the trial and decide whether they believe the defense that the author's conclusions are valid, or that they side with the prosecution that the author's conclusions are not valid.
2. Presentations by the Defense Lawyers. The defense will present their case first to the court. Each person in the group is expected to have read the article and be thoroughly familiar with each section (Introduction, Method, Results, and Discussion). On the day of the trial I will randomly select which person in the group will present each section. The person selected to present the Introduction section will begin by presenting to the class a summary of the author's introduction section, i.e., the theory that is being investigated, or the problem that is being explored by the author. You must use some form of media to help make your case as clearly and convincingly as possible to the jury (e.g., posters, drawings, handouts, overhead transparencies, etc.). The rest of the defense lawyers will follow by summarizing the other sections of the article. Posters or drawings on the blackboard are particularly encouraged for the Method and Results sections so that the everyone understands what tests were given when, to what group, and what the results indicate.
3. Presentation by the Prosecution lawyers. When the defense lawyers have completed presenting their case, the prosecution lawyers will then come up and present their side of the story. Once again, I will randomly choose which person will present which section to the class. Each person’s task is to go through their section, trying not to repeat what has been said by the defense, but to put a different light on it: what weaknesses are there? What methodological problems do you see? What should have been done differently?
4. The Defense Summation. The defense may be given a final opportunity to respond to the concerns raised by the prosecution. They will only be allowed a few minutes to do so.
5. Jury Deliberations. After the summation I will explain to the class exactly what decisions they will have to return a verdict on. The jury will be allowed to discuss for approximately 10 minutes and then will be asked for their decisions. We will then discuss the verdicts.
6. The Judge (myself) will make sure the trial proceeds fairly. I will instruct the jury as to what they are to decide once the proceedings are over.

The Roles

1. build a convincing case in favor of the article. Try to win the case by presenting the research in its best light. This also means paying attention to the weaknesses as well as the strengths of the article. What argument will you use to defend the author against the attacks of the prosecution if you (and they) see a weakness in the research?
2. defend your article from the questions/criticisms that will be made against it by the prosecution. Know the strengths and weaknesses of the article.

The job of the "prosecution attorneys" is to examine the research carefully and pay particular attention to the weaknesses and possible confounds in every section of the study.

1. build a convincing case against the article. Try to win the case by poking holes in the research. This means paying attention to the weaknesses as well as the strengths of the article. What arguments will you use to place doubt in the jury's mind about the validity of the author's conclusions?
2. expertly pinpoint possible areas of weakness in the research. Prepare your critiques, and be ready to try to convince the jury that the problems you have found are serious enough to place in doubt the conclusions of the authors.

The Jury

The class will listen carefully to the arguments presented by both sides. Your job is to decide whether or not the author has done a good piece of research. No study is ideal, but can you have confidence in the results of this research? Feel free to raise your hand at any point in the trial.

Threats to Validity
What to Look During Your Deliberations
(Campbell and Stanley)

1. History: Did anything unusual happen during the course of the experiment that could have affected the dependent variable?
2. Maturation: Are the results possibly due to natural changes in the subjects over time? For example, are subjects much older at the end of the study? More tired?
3. Testing: If there was a pre-test, could it have affected the post-test in any way? Look for: Subjects who are unfamiliar with taking tests, who may have gotten better at taking them a second time, the same test used both at the pre-test and the post-test.
4. Instrumentation: Examine all the measurement instruments used in the article. Are they good instruments (reliable and valid)? Look for: Reliability and validity information? (i.e., alpha coefficients, test/retest info, etc.). Were they "home grown" instruments? Were people used to measure things? (Inter-judge reliability
5. Regression: If subjects were chosen to participate in the study on the basis of extreme scores, could scores have naturally moved toward the mean upon a re-test? Look for: Subjects chosen because they were poor/very good readers, had low self-esteem, etc.
6. Selection: How were subjects were selected? Was it random? If not, what problems could occur as a result of the selection strategy used? Look for: Information regarding how the subjects were chosen to be in the study.
7. Mortality: Did any subjects drop out? Could this have affected the results? Look for: How many subs took the pre-test and how many took the post-test? Are there unequal n sizes at the data collection points? Small n sizes may violate assumptions of normality of t and f tests. Also: statistical significance using a large N (ex: 1,000 subs) may not be practically significant (ie. Effect sizes).

Back to The Psych Files