Friday, 3 October 2014


True experimental designs compare people who have received an intervention ("treatment group") to an equivalent group who did not receive the intervention ("control group"). Subjects are randomly assigned to either the treatment or control groups; indeed, random assignment is the hallmark of the true experimental designs (a.k.a., randomized trials). In many circles, the randomized trial is the "gold standard" of quantitative research, reflecting its degree of methodological rigor.
"Rigor" refers to the degree to which evaluators can rule out alternate explanations for their findings. The true experiment, with its random assignment, allows evaluators to state with increased confidence that the intervention, and not some other factor, is responsible for the given results. What random assignment does (at least hypothetically), is to create equivalent groups, thereby controlling for the effects of other factors. Still, it will be necessary to monitor that equivalence to see if it is indeed true. This is the only design that allows researchers to specify cause-and-effect relationships.
The true experiment, however, is not the only way to get at "truth." An example of this is in the relationship between cancer and cigarette smoking. For many years, cigarette companies hid behind the excuse that health officials could not "scientifically prove" that smoking causes cancer. In the framework of the true experiment, individuals would have to be assigned to "smoking" and "non-smoking" groups; obviously, this would have been quite unethical! Because researchers could not do this, it took multiple studies to account for all the other possible explanations (e.g., smokers may have a variety of other poor health practices that could account for higher cancer rates).

A.    Definition of True Experiment
True experimental design is regarded as the most accurate form of experimental research, in that it tries to prove or disprove a hypothesis mathematically, with statistical analysis. For some of the physical sciences, such as physics, chemistry and geology, they are standard and commonly used. For social sciences, psychology and biology, they can be a little more difficult to set up.
For an experiment to be classed as a true experimental design, it must fit all of the following criteria.
·         The sample groups must be assignedrandomly.
·         There must be a viable control group.
·         Only one variable can be manipulated and tested. It is possible to test more than one, but such experiments and their statistical analysis tend to be cumbersome and difficult.
·         The tested subjects must be randomly assigned to either control or experimental groups.


B.     Types of True Experiment
1.    Pre-test/Post-test control group design 
This is also called the classic controlled experimental design, and the randomized pre-test/post-test design because it:
a)    Controls the assignment of subjects to experimental (treatment) and control groups through the use of a table of random numbers.
b)     Controls the timing of the independent variable (treatment) and which group is exposed to it.
c)    Controls all other conditions under which the experiment takes place.

2.    Post-Test Only Control Group Design

 This design follows all the same steps as the classic pre-test/post-test design except that it omits the pre-test.  There are many situations where a pre-test is impossible because the participants have already been exposed to the treatment, or it would be too expensive or too time-consuming.  For large enough groups, this design can control for most of the same threats to internal and external validity as the classic controlled experimental design.  For example, it eliminates the threat to internal validity of pre-testing by eliminating the pre-test.  It may also decrease the problem of experimental mortality by shortening the length of the study (no pre-test).
For small groups, however, a pre-test is necessary.  Also, a pre-test is necessary if the researcher wants to determine the exact amount of change attributable to the independent variable alone. Public administrators would like to be able to use experimental designs for policy and program evaluation. 
3.    Solomon Four Group Design
The Solomon four group design is a way of avoiding some of the difficulties associated with the pretest-posttest design. This design contains two extra control groups, which serve to reduce the influence of confounding variables and allow the researcher to test whether the pretest itself has an effect on the subjects.
Whilst much more complex to set up and analyze, this design type combats many of the internal validity issues that can plagueresearch. It allows the researcher to exert complete control over the variables and allows the researcher to check that the pretest did not influence the results.
The Solomon four group test is a standard pretest-posttest two-group design and the posttest only control design. The various combinations of tested and untested groups with treatment and control groups allows the researcher to ensure that confounding variables and extraneous factors have not influenced the results.

C.    Advantages and Disadvantages of True Experiment

1.      Advantages
The results of a true experimental design can be statistically analyzed and so there can be little argument about the results. It is also much easier for other researchers to replicate the experiment and validate the results. For physical sciences working with mainly numerical data, it is much easier tomanipulate one variable, so true experimental design usually gives a yes or no answer.

2.      Disadvantages

Whilst perfect in principle, there are a number of problems with this type of design. Firstly, they can be almost too perfect, with the conditions being under complete controland not being representative of real world conditions.
For psychologists and behavioral biologists, for example, there can never be any guarantee that a human or living organism will exhibit ‘normal’ behavior under experimental conditions.
True experiments can be too accurate and it is very difficult to obtain a complete rejection or acceptance of a hypothesis because the standards of proof required are so difficult to reach.
True experiments are also difficult and expensive to set up. They can also be very impractical. While for some fields, like physics, there are not as many variables so the design is easy, for social sciences and biological sciences, where variations are not so clearly defined it is much more difficult to exclude other factors that may be affecting the manipulated variable.


D.    Controlling for Threats to Internal Validity

1)      History:  did some other current event effect the change in the dependent variable?  No, because both groups experienced the same current events.
2)      Maturation:  were changes in the dependent variable due to normal developmental processes?  No, because both groups experienced the same developmental processes.
3)      Statistical Regression:  did subjects come from low or high performing groups?  Differences between the two groups that could influence the dependent variable would be controlled for as subjects were generally equivalent at the beginning of the research.
4)      Selection:  were the subjects self-selected into experimental and control groups, which could affect the dependent variable?  No, the subjects were assigned by strict random selection and all had equal chance of getting the treatment or control condition.
5)      Experimental Mortality:  did some subjects drop out?  did this affect the results?  About the same number of students made it through the entire study in both the experimental and control groups, so there appears to be no bias.
6)      Testing:  Did the pre-test affect the scores on the post-test?  Both groups got a pre-test; but a pre-test may have made the experimental group more sensitive to the treatment.
7)      Instrumentation:  Did the measurement method change during the research?  The measurement method and instruments did not change.
8)      Design contamination:  did the control group find out about the experimental treatment?  did either group have a reason to want to make the research succeed or fail?  The researcher must do some qualitative investigation to find out if there was design contamination. 
There are several common threats to internal validity in experimental research. They are described in our text.  I have review each below:
  • Subject Characteristics (Selection Bias/Differential Selection) -- The groups may have been different from the start. If you were testing instructional strategies to improve reading and one group enjoyed reading more than the other group, they may improve more in their reading because they enjoy it, rather than the instructional strategy you used.
  • Loss of Subjects (Mortality) -- All of the high or low scoring subject may have dropped out or were missing from one of the groups. If we collected posttest data on a day when the honor society was on field trip at the treatment school, the mean for the treatment group would probably be much lower than it really should have been.
  • Location -- Perhaps one group was at a disadvantage because of their location.  The city may have been demolishing a building next to one of the schools in our study and there are constant distractions which interferes with our treatment.
  • Instrumentation Instrument Decay -- The testing instruments may not be scores similarly. Perhaps the person grading the posttest is fatigued and pays less attention to the last set of papers reviewed. It may be that those papers are from one of our groups and will received different scores than the earlier group's papers
  • Data Collector Characteristics -- The subjects of one group may react differently to the data collector than the other group. A male interviewing males and females about their attitudes toward a type of math instruction may not receive the same responses from females as a female interviewing females would.
  • Data Collector Bias -- The person collecting data my favors one group, or some characteristic some subject possess, over another. A principal who favors strict classroom management may rate students' attention under different teaching conditions with a bias toward one of the teaching conditions.
  • Testing -- The act of taking a pretest or posttest may influence the results of the experiment. Suppose we were conducting a unit to increase student sensitivity to prejudice. As a pretest we have the control and treatment groups watch Shindler's List and write a reaction essay. The pretest may have actually increased both groups' sensitivity and we find that our treatment groups didn't score any higher on a posttest given later than the control group did. If we hadn't given the pretest, we might have seen differences in the groups at the end of the study.
  • History -- Something may happen at one site during our study that influences the results. Perhaps a classmate dies in a car accident at the control site for a study teaching children bike safety. The control group may actually demonstrate more concern about bike safety than the treatment group.
  • Maturation --There may be natural changes in the subjects that can account for the changes found in a study. A critical thinking unit may appear more effective if it taught during a time when children are developing abstract reasoning.
  • Hawthorne Effect -- The subjects may respond differently just because they are being studied. The name comes from a classic study in which researchers were studying the effect of lighting on worker productivity. As the intensity of the factor lights increased, so did the work productivity. One researcher suggested that they reverse the treatment and lower the lights. The productivity of the workers continued to increase. It appears that being observed by the researchers was increasing productivity, not the intensity of the lights.
  • John Henry Effect -- One group may view that it is competition with the other group and may work harder than than they would under normal circumstances. This generally is applied to the control group "taking on" the treatment group. The terms refers to the classic story of John Henry laying railroad track.
  • Resentful Demoralization of the Control Group -- The control group may become discouraged because it is not receiving the special attention that is given to the treatment group. They may perform lower than usual because of this.
  • Regression (Statistical Regression) -- A class that scores particularly low can be expected to score slightly higher just by chance. Likewise, a class that scores particularly high, will have a tendency to score slightly lower by chance. The change in these scores may have nothing to do with the treatment.
  • Implementation --The treatment may not be implemented as intended. A study where teachers are asked to use student modeling techniques may not show positive results, not because modeling techniques don't work, but because the teacher didn't implement them or didn't implement them as they were designed.
  • Compensatory Equalization of Treatment -- Someone may feel sorry for the control group because they are not receiving much attention and give them special treatment. For example, a researcher could be studying the effect of laptop computers on students' attitudes toward math. The teacher feels sorry for the class that doesn't have computers and sponsors a popcorn party during math class. The control group begins to develop a more positive attitude about mathematics.
  • Experimental Treatment Diffusion -- Sometimes the control group actually implements the treatment. If two different techniques are being tested in two different third grades in the same building, the teachers may share what they are doing. Unconsciously, the control may use of the techniques she or he learned from the treatment teacher.

E.     Controlling for Threats to External Validity
1)      Unique program features:  There may have been an unusually motivated set of facilitators for the small group discussions.
2)      Effects of Selection:  Probably applicable to other medical students.
3)      Effects of Setting:  Medical schools have their own cultures; doubtful if this would be applicable to other types of students.
4)      Effects of History
5)      Effects of Testing
6)      Reactive effects of experimental arrangements:  It would be best to replicate the results in other medical schools.  


True experimental design is an integral part of science, usually acting as a final test of a hypothesis. Whilst they can be cumbersome and expensive to set up, literature reviews, qualitative research and descriptive research can serve as a good precursor to generate a testable hypothesis, saving time and money. Whilst they can be a little artificial and restrictive, they are the only type of research that is accepted by all disciplines as statistically provable. In true experimental research, the researcher not only manipulates the independent variable, he or she also randomly assigned individuals to the various treatment categories (i.e., control and treatment).



Post a Comment