Experimental Research

Experimental research is a systematic and scientific approach to research in which the researcher manipulates one or more variables, and controls and measures any change in other variables.

Experimental Research is often used where:
1. There is time priority in a causal relationship (cause precedes effect)
2. There is consistency in a causal relationship (a cause will always lead to the same effect)
3. The magnitude of the correlation is great.

Aims of Experimental Research

Experiments are conducted to be able to predict phenomenons. Typically, an experiment is constructed to be able to explain some kind of causation. Experimental research is important to society - it helps us to improve our everyday lives.

One of the main strengths of experimental research is that it can determine a cause and effect relationship between two variables. However, because of its reliance on a controlled, artificial environment, this kind of research is often difficult to generalize to real world situations.
Independent and Dependent Variables
In an experimental study, the independent variable is the factor that the experimenter controls and manipulates throughout the experiment. The dependent variable, on the other hand, is the variable that is influenced and affected by the independent variable. In a simple experiment, a researcher might hypothesize that cookies will make individuals work harder. In one condition, participants will be offered a cookie if they complete a task, while in another they are not. In this case the presence of a reward (the cookie) is the independent variable and completing the task is the dependent variable.
An experiment can have more than one independent variable. A researcher might decide to see if a cookie will only make individuals work harder if the task is easy. In this case both the presence of a reward and task difficulty would be independent variables.
Experimental Design
The purpose of an experiment is to investigate the relationship between two variables to test a hypothesis. By using the scientific method, a psychologist can plan and design an experiment that will answer the research question. The basic steps of experimental design are:
Identifying a question and performing preliminary research to determine what is already known
Creating a hypothesis
Identifying and defining the independent and dependent variables
Determining how the independent variable will be manipulated and how the dependent variable will be measured

The Scientific Method
The scientific method is the process by which new scientific knowledge is gained and verified. First you must identify a question and, after some preliminary research, form a hypothesis. After designing an experiment to test the hypothesis and collecting data, a scientist will use this information to draw a conclusion. The conclusion will either support the hypothesis or refute it. Based on this information, the scientist will then either reformulate the hypothesis or build upon the original hypothesis. The scientific method cannot prove a hypothesis, only support or refute it.

MODULE R13
EXPERIMENTAL RESEARCH AND DESIGN Experimental Research - An attempt by the researcher to maintain control over all factors that may affect the result of an experiment. In doing this, the researcher attempts to determine or predict what may occur.
Experimental Design - A blueprint of the procedure that enables the researcher to test his hypothesis by reaching valid conclusions about relationships between independent and dependent variables. It refers to the conceptual framework within which the experiment is conducted.
Steps involved in conducting an experimental study

Identify and define the problem.

Formulate hypotheses and deduce their consequences.

Construct an experimental design that represents all the elements, conditions, and relations of the consequences.
1. Select sample of subjects.
2. Group or pair subjects.
3. Identify and control non experimental factors.
4. Select or construct, and validate instruments to measure outcomes.
5. Conduct pilot study.
6. Determine place, time, and duration of the experiment.

Conduct the experiment.

Compile raw data and reduce to usable form.

Apply an appropriate test of significance.

Essentials of Experimental Research

Manipulation of an independent variable.

An attempt is made to hold all other variables except the dependent variable constant - control.

Effect is observed of the manipulation of the independent variable on the dependent variable - observation.
Experimental control attempts to predict events that will occur in the experimental setting by neutralizing the effects of other factors.
Methods of Experimental Control

Physical Control

Gives all subjects equal exposure to the independent variable.

Controls non experimental variables that affect the dependent variable.

Selective Control - Manipulate indirectly by selecting in or out variables that cannot be controlled.

Statistical Control - Variables not conducive to physical or selective manipulation may be controlled by statistical techniques (example: covariance). Validity of Experimental Design

Internal Validity asks did the experimental treatment make the difference in this specific instance rather than other extraneous variables?

External Validity asks to what populations, settings, treatment variables, and measurement variables can this observed effect be generalized?
Factors Jeopardizing Internal Validity

History - The events occurring between the first and second measurements in addition to the experimental variable which might affect the measurement.
Example: Researcher collects gross sales data before and after a 5 day 50% off sale. During the sale a hurricane occurs and results of the study may be affected because of the hurricane, not the sale.

Maturation - The process of maturing which takes place in the individual during the duration of the experiment which is not a result of specific events but of simply growing older, growing more tired, or similar changes.
Example: Subjects become tired after completing a training session, and their responses on the Posttest are affected.

Pre-testing - The effect created on the second measurement by having a measurement before the experiment.
Example: Subjects take a Pretest and think about some of the items. On the Posttest they change to answers they feel are more acceptable. Experimental group learns from the pretest.

Measuring Instruments - Changes in instruments, calibration of instruments, observers, or scorers may cause changes in the measurements.
Example: Interviewers are very careful with their first two or three interviews but on the 4th, 5th, 6th become fatigued and are less careful and make errors.

Statistical Regression - Groups are chosen because of extreme scores of measurements; those scores or measurements tend to move toward the mean with repeated measurements even without an experimental variable.
Example: Managers who are performing poorly are selected for training. Their average Posttest scores will be higher than their Pretest scores because of statistical regression, even if no training were given.

Differential Selection - Different individuals or groups would have different previous knowledge or ability which would affect the final measurement if not taken into account.
Example: A group of subjects who have viewed a TV program is compared with a group which has not. There is no way of knowing that the groups would have been equivalent since they were not randomly assigned to view the TV program.

Experimental Mortality - The loss of subjects from comparison groups could greatly affect the comparisons because of unique characteristics of those subjects. Groups to be compared need to be the same after as before the experiment.
Example: Over a 6 month experiment aimed to change accounting practices, 12 accountants drop out of the experimental group and none drop out of the control group. Not only is there differential loss in the two groups, but the 12 dropouts may be very different from those who remained in the experimental group.

Interaction of Factors, such as Selection Maturation, etc. - Combinations of these factors may interact especially in multiple group comparisons to produce erroneous measurements.
Factors Jeopardizing External Validity or Generalizability
Pre-Testing -Individuals who were pretested might be less or more sensitive to the experimental variable or might have "learned" from the pre-test making them unrepresentative of the population who had not been pre-tested.
Example: Prior to viewing a film on Environmental Effects of Chemical, a group of subjects is given a 60 item antichemical test. Taking the Pretest may increase the effect of the film. The film may not be effective for a nonpretested group.
Differential Selection - The selection of the subjects determines how the findings can be generalized. Subjects selected from a small group or one with particular characteristics would limit generalizability. Randomly chosen subjects from the entire population could be generalized to the entire population.
Example: Researcher, requesting permission to conduct experiment, is turned down by 11 corporations, but the 12th corporation grant permission. The 12th corporation is obviously different then the others because they accepted. Thus subjects in the 12th corporation may be more accepting or sensitive to the treatment.
Experimental Procedures - The experimental procedures and arrangements have a certain amount of effect on the subjects in the experimental settings. Generalization to persons not in the experimental setting may be precluded.
Example: Department heads realize they are being studied, try to guess what the experimenter wants and respond accordingly rather than respond to the treatment.
Multiple Treatment Interference - If the subjects are exposed to more than one treatment then the findings could only be generalized to individuals exposed to the same treatments in the same order of presentation.
Example: A group of CPA’s is given training in working with managers followed by training in working with comptrollers. Since training effects cannot be deleted, the first training will affect the second.
Tools of Experimental Design Used to Control Factors Jeopardizing Validity

Pre-Test - The pre-test, or measurement before the experiment begins, can aid control for differential selection by determining the presence or knowledge of the experimental variable before the experiment begins. It can aid control of experimental mortality because the subjects can be removed from the entire comparison by removing their pre-tests.
However, pre-tests cause problems by their effect on the second measurement and by causing generalizability problems to a population not pre-tested and those with no experimental arrangements.

Control Group -The use of a matched or similar group which is not exposed to the experimental variable can help reduce the effect of History, Maturation, Instrumentation, and Interaction of Factors. The control group is exposed to all conditions of the experiment except the experimental variable.

Randomization - Use of random selection procedures for subjects can aid in control of Statistical Regression, Differential Selection, and the Interaction of Factors. It greatly increases generalizability by helping make the groups representative of the populations.

Additional Groups - The effects of Pre-tests and Experimental Procedures can be partially controlled through the use of groups which were not pre-tested or exposed to experimental arrangements. They would have to be used in conjunction with other pre-tested groups or other factors jeopardizing validity would be present.

The method by which treatments are applied to subjects using these tools to control factors jeopardizing validity is the essence of experimental design.
Tools of Control Internal Sources
Pre-Test/
Post Test
Control Group
Randomization
Additional
Groups
History X

Maturation X

Pre-Testing

X
Measuring Instrument X

Statistical Regression X
X

Differential Selection
X

X Experimental Mortality
X

Interaction of Factors X
X

External Sources

Pre-Testing

X
Differential Selection
X

X Procedures

X
Multiple Treatment

Experimental Designs
Pre-Experimental Design - loose in structure, could be biased
Aim of the Research
Name of the Design
Notation Paradigm
Comments
To attempt to explain a consequent by an antecedent
One-shot experimental case study
X » O
An approach that prematurely links antecedents and consequences. The least reliable of all experimental approaches.
To evaluate the influence of a variable
One group pretest-posttest
O » X » O
An approach that provides a measure of change but can provide no conclusive results.
To determine the influence of a variable on one group and not on another
Static group comparison
Group 1: X » O
Group 2: - » O
Weakness lies in no examination of pre-experimental equivalence of groups. Conclusion is reached by comparing the performance of each group to determine the effect of a variable on one of them. True Experimental Design - greater control and refinement, greater control of validity
Aim of the Research
Name of the Design
Notation Paradigm
Comments
To study the effect of an influence on a carefully controlled sample
Pretest-posttest control group
R - - [ O » X » O [ O » - » O
This design has been called "the old workhorse of traditional experimentation." If effectively carried out, this design controls for eight threats of internal validity. Data are analyzed by analysis of covariance on posttest scores with the pretest the covariate.
To minimize the effect of pretesting
Solomon four-group design
R - - [ O » X » O [ O » - » O [- » X » O [ - » - » O
This is an extension of the pretest-posttest control group design and probably the most powerful experimental approach. Data are analyzed by analysis of variance on posttest scores.
To evaluate a situation that cannot be pretested
Posttest only control group
R - - [ X » O [ - » O
An adaptation of the last two groups in the Solomon four-group design. Randomness is critical. Probably, the simplest and best test for significance in this design is the t-test.
Quasi-Experimental Design - not randomly selected
Aim of the Research
Name of the Design
Notation Paradigm
Comments
To investigate a situation in which random selection and assignment are not possible
Nonrandomized control group pretest-posttest
O » X » O
O » - » O
One of the strongest and most widely used quasi-experimental designs. Differs from experimental designs because test and control groups are not equivalent. Comparing pretest results will indicate degree of equivalency between experimental and control groups.
To determine the influence of a variable introduced only after a series of initial observations and only where one group is available
Time series experiment
O » O » X » O » O
If substantial change follows introduction of the variable, then the variable can be suspect as to the cause of the change. To increase external validity, repeat the experiment in different places under different conditions.
To bolster the validity of the above design with the addition of a control group
Control group time series
O » O » X » O » O
O » O » - » O » O
A variant of the above design by accompanying it with a parallel set of observations without the introduction of the experimental variable.
To control history in time designs with a variant of the above design
Equivalent time-samples
[X1 » O1] »[X0 » O2] » [x1 » O3]
An on-again, off-again design in which the experimental variable is sometimes present, sometimes absent.
Correlational and Ex Post Facto Design
Aim of the Research
Name of the Design
Notation Paradigm
Comments
To seek for cause-effect relationships between two sets of data
Causal-comparative correlational studies
-»
Oa ¥ Ob
«-
A very deceptive procedure that requires much insight for its use. Causality cannot be inferred merely because a positive and close correlation ratio exists.
To search backward from consequent data for antecedent causes
Ex post facto studies This approach is experimentation in reverse. Seldom is proof through data substantiation possible. Logic and inference are the principal tools of this design
Leedy, P.D. (1997). Practical research: Planning and design (6th ed.). Upper Saddle River, NJ: Prentice-Hall, Inc., p. 232-233. SELF ASSESSMENT
1. Define experimental research.
Define experimental design.
2. List six steps involved in conducting an experimental study.
3. Describe the basis of an experiment.
4. Name three characteristics of experimental research.
5. State the purpose of experimental control.
6. State three broad methods of experimental control.
7. Name two type of validity of experimental design.
8. Define eight factors jeopardizing internal validity of a research design.
9. Define four factors jeopardizing external validity.
10. Describe the tools of experimental design used to control the factors jeopardizing validity of a research design.
11. Define the essence of experimental design.
12. Name and describe the four types of experimental designs.
BENEFITS AND LIMITATIONS OF EXPERIMENTAL RESEARCH
Experimental research is generally recognized as the most appropriate method for drawing causal conclusions about instructional interventions, for example, which instructional method is most effective for which type of student under which conditions. In a careful analysis of educational research methods, Richard Shavelson and Lisa Towne concluded that “from a scientific perspective, randomized trials (we also use the term experiment to refer to causal studies that feature random assignment) are the ideal for establishing whether one or more factors caused change in an outcome because of their strong ability to enable fair comparisons” (2002, p. 110). Similarly, Richard Mayer notes: “experimental methods— which involve random assignment to treatments and control of extraneous variables—have been the gold standard for educational psychology since the field evolved in the early 1900s” (2005, p. 74). Mayer states, “when properly implemented, they allow for drawing causal conclusions, such as the conclusion that a particular instructional method causes better learning outcomes” (p. 75). Overall, if one wants to determine whether a particular instructional intervention causes an improvement in student learning, then one should use experimental research methodology.

Although experiments are widely recognized as the method of choice for determining the effects of an instructional intervention, they are subject to limitations involving method and theory. First, concerning method, the requirements for random assignment, experiment control, and appropriate measures can impose artificiality on the situation. Perfectly controlled conditions are generally not possible in authentic educational environments such as schools. Thus, there may be a trade-off between experimental rigor and practical authenticity, in which highly controlled experiments may be too far removed from real classroom contexts. Experimental researchers should be sensitive to this limitation, by incorporating mitigating features in their experiments that maintain ecological validity.

Second, concerning theory, experimental research may be able to tell that one method of instruction is better than conventional practice, but may not be able to specify why; it may not be able to pinpoint the mechanisms that create the improvement. In these cases, it is useful to derive clear predictions from competing theories so experimental research can be used to test the specific predictions of competing theories. In addition, more focused research methods—such as naturalistic observation or in-depth interviews—may provide richer data that allows for the development of a detailed explanation for why an intervention might have a new effect. Experimental researchers should be sensitive to this limitation, by using complementary methods in addition to experiments that provide new kinds of evidence.