Chapter 7
CHAPTER OUTLINE AND OBJECTIVES
I. Overview
II. Why Psychologists Conduct Experiments
· Researchers conduct experiments to test hypotheses about the causes of behavior.
· Experiments allow researchers to decide whether a treatment or program effectively changes behavior.
III. Logic of Experimental Research
· Researchers manipulate an independent variable in an experiment to observe the effect on behavior, as assessed by the dependent variable.
· Experimental control allows researchers to make the causal inference that the independent variable caused the observed changes in the dependent variable.
· Control is the essential ingredient of experiments; experimental control is gained through manipulation, holding conditions constant, and balancing.
· An experiment has internal validity when it fulfills the three conditions required for causal inference: covariation, time-order relationship, and elimination of plausible alternative causes.
· When confounding occurs, a plausible alternative explanation for the observed covariation exists, and therefore, the experiment lacks internal validity. Plausible alternative explanations are ruled out by holding conditions constant and balancing.
IV. Random Groups Design
- In an independent groups design, each group of subjects participates in only one condition of the independent variable.
- Random assignment to conditions is used to form comparable groups by balancing or averaging subject characteristics (individual differences) across the conditions of the independent variable manipulation.
- When random assignment is used to form independent groups for the levels of the independent variable, the experiment is called a random groups design.
A. An Example of a Random Groups Design
B. Block Randomization
· Block randomization balances subject characteristics and potential confoundings that occur during the time in which the experiment is conducted, and it creates groups of equal size.
C. Threats to Internal Validity
· Randomly assigning intact groups to different conditions of the independent variable creates a potential confounding due to preexisting differences among participants in the intact groups.
· Block randomization increases internal validity by balancing extraneous variables across conditions of the independent variable.
· Selective subject loss, but not mechanical subject loss, threatens the internal validity of an experiment.
· Placebo control groups are used to control for the problem of demand characteristics, and double-blind experiments control both demand characteristics and experimenter effects.
V. Analysis and Interpretation of Experimental Findings
A. The Role of Data Analysis in Experiments
· Data analysis and statistics play a critical role in researchers’ ability to make the claim that an independent variable has had an effect on behavior.
· The best way to determine whether the findings of an experiment are reliable is to do a replication of the experiment.
B. Describing the Results
· The two most common descriptive statistics that are used to summarize the results of experiments are the mean and standard deviation.
· Measures of effect size indicate the strength of the relationship between the independent and dependent variables, and they are not affected by sample size.
· One commonly used measure of effect size, d, examines the difference between two group means relative to the average variability in the experiment.
· Meta-analysis uses measures of effect size to summarize the results of many experiments investigating the same independent variable or dependent variable.
C. Confirming What the Results Reveal
· Researchers use inferential statistics to determine whether an independent variable has a reliable effect on a dependent variable.
· Two methods to make inferences based on sample data are null hypothesis testing and confidence intervals.
· Researchers use null hypothesis testing to determine whether mean differences among groups in an experiment are greater than the differences that are expected simply because of error variation.
· A statistically significant outcome is one that has a small likelihood of occurring if the null hypothesis were true.
· Researchers determine whether an independent variable has had an effect on behavior by examining whether the confidence intervals for different samples in an experiment overlap. The degree of overlap provides information as to whether the sample means estimate the same population mean or different population means.
D. What Data Analysis Can’t Tell Us
VI. Establishing the External Validity of Experimental Findings
· The findings of an experiment have external validity when they can be applied to other individuals, settings, and conditions beyond the scope of the specific environment.
· In some investigations (e.g., theory-testing), researchers may choose to emphasize internal validity over external validity; other researchers may choose to increase external validity using sampling or replication.
· Conducting field experiments is one way that researchers can increase the external validity of their research in real-world settings.
· Partial replication is a useful method for establishing the external validity of research findings.
· Researchers often seek to generalize results about conceptual relationships among variables rather than specific conditions, manipulations, settings, and samples.
VII. Matched Groups Design
· A matched group design may be used to create comparable groups when there are too few subjects available for random assignment to work effectively.
· Matching subjects on the dependent variable task is the best approach for creating matched groups, but performance on any matching task must correlate with the dependent variable task.
· After subjects are matched on the matching task they should then be randomly assigned to the conditions of the independent variable.
VIII. Natural Groups Design
· Individual differences variables (or subject variables) are selected rather than manipulated to form natural groups designs.
· The natural groups design represents a type of correlational research in which researchers look for covariations between natural groups variables and dependent variables.
· Causal inferences cannot be made regarding the effects of natural groups variables because plausible alternative explanations for group differences exist.
IX. Summary
REVIEW QUESTIONS AND ANSWERS
These questions appear in the textbook (without answers) at the end of Chapter 7, and can be used for a homework assignment or exam preparation. Answers to these questions appear in italic.
1. Describe two reasons why psychologists conduct experiments.
Psychologists conduct experiments in order test hypotheses about the causes of behavior (i.e., the meet the third goal of research, explanation) and to learn whether a treatment or program effectively changes behavior. (pp. 205–206)
2. Describe how the control techniques of manipulation, holding conditions constant, and balancing contribute to meeting the three conditions necessary for a causal inference.
The covariation condition is met when we manipulate an independent variable with at least two levels and observe a difference between groups on the dependent variable. The condition of establishing a time-order relationship is met when we observe an effect on the dependent variable after the independent variable manipulation. Holding conditions constant and balancing are used to rule out alternative causes for any difference between groups at the conclusion of the experiment. Holding conditions constant means that the independent variable is the only factor allowed to vary between the groups, and balancing subject characteristics across the groups of the experiment allows us to rule out the alternative explanation that differences among the participants in the groups of the experiment caused different outcomes on the dependent variable. (pp. 207)
3. Explain why a research study conducted using the one-group pretest-posttest design has very little internal validity.
Internal validity refers to the ability to make an unambiguous causal inference about the effect of an independent variable on a dependent variable. In this design, however, should there be a difference between the pretest and posttest measures it is impossible to know whether a treatment produced the effect. Some other factor other than the treatment may have caused the posttest measure to change over time. (pp. 207–208)
4. Explain why comparable groups are such an essential feature of the random groups design, and describe how researchers achieve comparable groups.
The logic of the random groups design is to form two (or more) different groups in such a way that the groups are comparable at the start of the experiment. Random assignment is used to balance subject characteristics across the groups. That is, on average, participants who are randomly assigned to conditions of an experiment are equivalent. (p. 212)
5. What preventive steps could you take if you anticipated that selective subject loss could pose a problem in your experiment?
One step is to administer a pretest and screen out participants who are likely to be lost in the study (e.g., because they are not interested in the study procedures). A second alternative is to administer a pretest and when a participant from one group drops out of the study, a comparable participant (based on pretest scores) from another group(s) could be dropped from the study data analysis. (pp. 218–220
Chapter 8 Repeated Measures Designs
CHAPTER OUTLINE AND OBJECTIVES
I. Overview
II. Why Researchers Use Repeated Measures Designs
· Researchers choose to use a repeated measures design in order to (1) conduct an experiment when few participants are available, (2) conduct the experiment more efficiently, (3) increase the sensitivity of the experiment, and (4) study changes in participants’ behavior over time.
III. The Role of Practice Effects in Repeated Measures Designs
- Repeated measures designs cannot be confounded by individual differences variables because the same individuals participate in each condition (level) of the independent variable.
- Participants’ performance in repeated measures designs may change across conditions simply because of repeated testing (not because of the independent variable); these changes are called practice effects.
- Practice effects may threaten the internal validity of a repeated measures experiment when the different conditions of the independent variable are presented in the same order to all participants.
- There are two types of repeated measures designs (complete and incomplete) that differ in the specific ways in which they control for practice effects.
A. Defining Practice Effects
B. Balancing Practice Effects in the Complete Design
· Practice effects are balanced in complete designs within each participant using block randomization or ABBA counterbalancing.
· In block randomization, all of the conditions of the experiment (a block) are randomly ordered each time they are presented.
· In ABBA counterbalancing, a random sequence of all conditions is presented, followed by the opposite of the sequence.
· Block randomization is preferred over ABBA counterbalancing when practice effects are not linear, or when participants’ performance can be affected by anticipation effects.
C. Balancing Practice Effects in the Incomplete Design
· Practice effects are balanced across subjects in the incomplete design rather than for each subject, as in the complete design.
· The rule for balancing practice effects in the incomplete design is that each condition of the experiment must be presented in each ordinal position (first, second, etc.) equally often.
· The best method for balancing practice effects in the incomplete design with four or fewer conditions is to use all possible orders of the conditions.
· Two methods for selecting specific orders to use in an incomplete design are the Latin Square and random starting order with rotation.
· Whether using all possible orders or selected orders, participants should be randomly assigned to the different sequences.
IV. Data Analysis Of Repeated Measures Designs
A. Describing the Results
· Data analysis for a complete design begins with computing a summary score (e.g., mean, median) for each participant.
· Descriptive statistics are used to summarize performance across all participants for each condition of the independent variable.
B. Confirming What the Results Reveal
· The general procedures and logic for null hypothesis testing and for confidence intervals for repeated measures designs are similar to those used for random groups designs.
V. The Problem Of Differential Transfer
· Differential transfer occurs when the effects of one condition persist and influence performance in subsequent conditions.
· Variables that may lead to differential transfer should be tested using a random groups design because differential transfer threatens the internal validity of repeated measures designs.
· Differential transfer can be identified by comparing the results for the same independent variable when tested in a repeated measures design and in a random groups design.
VI. Summary
REVIEW QUESTIONS AND ANSWERS
These review questions appear in the textbook (without answers) at the end of Chapter 8, and can be used for a homework assignment or exam preparation. Answers to these questions appear in italic.
1. Describe what is balanced in a random groups design and what is balanced in a repeated measures design.
In a random groups design random assignment of subjects to conditions is used to balance, or average, individual differences variables across the conditions of the experiment. In a repeated measures design counterbalancing techniques are used to balance (average) practice effects associated with repeated measurement across the conditions of the independent variable. (pp. 245, 248)
2. Briefly describe four reasons why researchers would choose to use a repeated measures design.
Four reasons researchers may choose to use a repeated measures design are: (1) because only a small number of participants is available; (2) for the sake of convenience or efficiency; (3) because repeated measures designs are generally more sensitive; or (4) because the area of research requires its use. (pp. 245–247)
3. Define sensitivity and explain why repeated measures designs are often more sensitive than random groups designs.
Sensitivity refers to the likelihood in an experiment that the effect of an independent variable will be detected when that variable does, indeed, have an effect. Sensitivity is increased to the extent that error variation is reduced. One source of error variation is individual differences among the participants across the conditions of an experiment. Because the same participants experience each condition in a repeated measures design, there is less error variation than in a random groups design that has different participants in each condition. In general, there is more variability between people (random groups design) than there is within people (repeated measures design). (p. 246)
4. Distinguish between a complete design and an incomplete design for repeated measures designs.
In the complete repeated measures design, practice effects are balanced by administering the conditions several times to each subject, using different orders each time, such that the results for each subject are interpretable. In the incomplete repeated measures design, each condition is administered to each subject only once, and the order of administering the conditions is varied across subjects such that by combining the results for all subjects practice effects are balanced and thus the results are interpretable. (p. 250)
5. What options do researchers have in balancing practice effects in a repeated measures experiment using a complete design?
The two techniques for balancing practice effects in the complete repeated measures design are block randomization and ABBA counterbalancing. (pp. 252–254)
6. Under what two circumstances would you recommend against the use of ABBA counterbalancing to balance practice effects in a repeated measures experiment using a complete design?
The use of ABBA counterbalancing would not be recommended in a complete repeated measures design if practice effects are nonlinear or if anticipation effects are likely. Anticipation effects occur when a subject develops expectations for which condition should occur next. (pp. 254–255)
7. State the general rule for balancing practice effects in repeated measures experiments using an incomplete design.
The general rule for balancing practice effects in the incomplete repeated measures design is that each condition of the experiment must appear in each ordinal position equally often. (p. 257)
8. Briefly describe the techniques that researchers can use to balance practice effects in the repeated measures experiments using an incomplete design. Identify which of these techniques is preferred and explain why.
To balance practice effects in the incomplete repeated measures design researchers can use either all possible orders or selected orders. The two techniques using selected orders are the Latin Square and random starting order with rotation. The use of all possible orders is preferred because each condition precedes and follows every other condition at each ordinal position. When the number of conditions prohibits the use of all possible orders, a Latin Square design also generates orders of conditions such that each condition precedes and follows every other condition. (pp. 257–260)
9. Explain why an additional initial step is required to summarize the data for an experiment involving a complete repeated measures design.
Before summarizing and describing the results, researchers must compute a score for each participant’s responses in each condition. Each participant is tested in each condition more than once in a complete design, therefore a summary (e.g., mean, median) score for the each participant’s average performance in each condition must be used in the final analysis. (pp. 260–261)
10. Describe how researchers can determine if differential transfer has occurred in a repeated measures experiment.
Poulton (1982) recommends that the best way to document differential transfer is to study the same independent variable using a random groups design and a repeated measures design. Differential transfer cannot affect the results for the independent variable in a random groups design. If the results for the independent variable differ in the two experiments, differential transfer is likely to be responsible for producing the different outcome. (pp. 264–265)
CHALLENGE QUESTIONS AND ANSWERS
These questions appear in the textbook at the end of Chapter 8, and can be used for a homework assignment, in-class discussion, or exam preparation. Answers to these questions appear in italic below.
[Answer to Challenge Question 1 also appears in the text.]
1. The following problems represent different situations in the repeated measures designs in which practice effects need to be balanced.
A. Consider a repeated measures experiment using a complete design involving one independent variable. The independent variable in the experiment is task difficulty with three levels (Low, Medium, and High). You are to prepare an order for administering the conditions of this experiment so that the independent variable is balanced for practice effects. You are first to use block randomization to balance practice effects and then to use ABBA counterbalancing to balance the practice effects. Each condition should appear twice in the order you prepare. (You can use the first row of the random number table (Table A.1) in the Appendix to determine your two random orders for block randomization.)
Assigning the values 1, 2, 3 to the Low, Medium, and High conditions, respectively, and using the first row of the random number table (Table A.1) in the Appendix beginning with the first number in the row, the block-randomized sequence is: Low-High-Medium-Low-Medium-High. One possible ABBA counterbalanced sequence is Low-Medium-High-High-Medium-Low.
B. Consider a repeated measures experiment using an incomplete design. The independent variable in the experiment is the font size in which a paragraph has been printed, and there are six levels (7, 8, 9, 10, 11, and 12). Present a table showing how you would determine the order of administering the conditions to the first six participants of the experiment. Be sure that practice effects are balanced for these participants.
Because there are six conditions, all possible orders are not feasible. Therefore, either a Latin Square or a random starting order with rotation is needed to balance practice effects. A possible set of sequences using rotation is
Position | |||||||
Participant | 1st | 2nd | 3rd | 4th | 5th | 6th | |
1 | 8 | 10 | 11 | 9 | 7 | 12 | |
2 | 10 | 11 | 9 | 7 | 12 | 8 | |
3 | 11 | 9 | 7 | 12 | 8 | 10 | |
4 | 9 | 7 | 12 | 8 | 10 | 11 | |
5 | 7 | 12 | 8 | 10 | 11 | 9 | |
6 | 12 | 8 | 10 | 11 | 9 | 7 | |
2. The pursuit rotor is a test of perceptual-motor coordination. It involves a turntable with a disk about the size of a dime embedded in it. The participant is given a pointer and is asked to keep the pointer on the disk while the turntable is rotating. The dependent variable is the percentage of time on each trial that the participant keeps the pointer on the disk. Learning on this task is linearly related to trials over many periods of practice, and the task generally takes a long time to master. A researcher wants to study the influence of time of day on the performance on this task with four different times (10 A.M., 2 P.M., 6 P.M., and 10 P.M.). The participants will receive a constant number of trials under each of the four conditions, and participants will be tested in one condition per day over four consecutive days.
A. What design is being used for the time-of-day variable in this experiment?
The incomplete repeated measures design is being used for the time-of-day variable (10 A.M., 2 P.M., 6 P.M., 10 P.M.) because each person participates in each time of day condition only once.
B. Prepare a Latin Square to balance practice effects across the conditions of the experiment.
Assign the following: A = 10 A.M., B = 2 P.M., C = 6 P.M., and D = 10 P.M.
Using the Latin Square displayed on p. 259, the conditions would be presented in the following orders:
1st | 2nd | 3rd | 4th | 1st | 2nd | 3rd | 4th | |
B | A | C | D | 2 P.M. | 10 A.M. | 6 P.M. | 10 P.M. | |
A | D | B | C | 10 A.M. | 10 P.M. | 2 P.M. | 6 P.M. | |
D | C | A | B | 10 P.M. | 6 P.M. | 10 A.M. | 2 P.M. | |
C | B | D | A | 6 P.M. | 2 P.M. | 10 P.M. | 10 A.M. |
C. The researcher decides to use all possible orders to balance practice effects. The researchers assigns each participant to one of the 24 possible orders of the conditions. Which experimental design is included when you look only at the first condition to which each participant was assigned?
The random groups design is included at the first ordinal position, provided that participants are randomly assigned to orders. With 24 possible orders and 24 participants, 6 participants would be assigned to each of the 4 conditions, with 48 participants there would 12 participants in each of the 4 conditions, etc.
D. How could the researcher test whether differential transfer occurred when all possible orders were used to balance practice effects?
The researcher first could analyze the results for the four conditions at the first ordinal position; this represents a random groups design rather than a repeated measures design because at the first ordinal position participants have only experienced one condition. Then the researcher could analyze the overall results for the four conditions in the repeated measures design. If the results four the four conditions differ for the repeated measures design compared to the random groups design, differential transfer may have taken place
