Posted By TDOM on August 20, 2010
Prats and pitfalls of quoting scientific research
Recently I was involved in a discussion on another site concerning conflicting research findings. I cited several pieces of research that supported a position and discussed findings that conflicted with this research with the owner of that site. I don’t know whether the owner had actually read any of the research he cited to support his arguments, the website he referenced contained only the abstracts and a broken link to a full report of one of the studies that it cited. Reading abstracts is not the best way to determine whether a study is well done, reliable, or valid. An abstract typically contains a summary of the research, including its purpose, methodology, and conclusion. It is not detailed and usually doesn’t contain enough information to determine its quality. But abstracts were all I was able to obtain for the references he cited, so it was what I used.
I was attempting to suggest possible reasons for the differing results based on sample description and methodology contained in the abstracts. I was very careful to characterize my statements as “speculation” since I had not read the actual studies themselves. I do not wish to rehash that discussion here. I only use it now to point out that frequently, those who report on the findings of a particular study will rely solely upon the abstract and/or the conclusion for their information. This can have disastrous results. It is the body of the report that will contain information vital to the evaluation of the report itself. It is the body, not the abstract or conclusion, that will contain information as to whether the research can be trusted, or whether it may be pseudo-science or bad science.
Several months ago, I reviewed The Global Gender Gap Report, 2009 released by the World Economic Forum in an article titled When does “Inequality = Equality”? The authors of this report claimed that women had not yet achieved equality with men in any country in the world. This was what was widely reported in the media. However, just by reading the first few pages, I was able to determine that the article was heavily biased towards demonstrating this inequality. The authors even admitted to their bias. In fact, the way the study was structured, it would have been impossible to find a country where women have achieved equality with men since any subject area where women were considered to have an advantage compared to men was “equality;” meaning that if only one area existed where women were at a disadvantage, they would be considered to be at an overall disadvantage in that country, even if they held the advantage in all other areas. Further, the subject areas that were studied tended to be those where it is well known that women hold a disadvantage and other areas where men are disadvantaged were left out entirely. This is an example of very bad science that was used by the World Economic Forum for the stated purpose of influencing public policy. If the WEF were actually “committed to improving the state of the world” as its logo indicates, would it actually perpetrate such fraud in order to advance its political agenda?
Another illustration of the misuse of science can be found in the first paragraph of this press release and news reporting on the report.
Press release:
“Twenty-seven percent of public school students ages 12 to 17 say that their school is both gang- and drug-infected (drugs are used, kept or sold on school grounds), according to the National Survey of American Attitudes on Substance Abuse XV: Teens and Parents, the 15th annual back-to-school survey conducted by The National Center on Addiction and Substance Abuse (CASA*) at Columbia University. This means that 5.7 million public school students attend schools which are both gang- and drug-infected.”
Note that in the first sentence it is students who say their school is both gang and drug infested. Note that the last sentence states that this is fact. Simply because students say it, does not make it true. That is a problem inherent in survey research. People lie, exaggerate, answer with what they think the researcher wants to hear, etc.
The press release also does not state the criteria for determining whether a school is gang or drug infested. It simply tells states that it is both gang-and drug-infected if drugs are used, kept, or sold on school grounds. How many students had to report such conditions before a school is declared to meet this criterion? One student? Twenty? The press release reports that two samples of about a thousand teens each were conducted for this study. What it doesn’t say is whether several students from each of a few schools were surveyed or whether only one student from two thousand schools was surveyed. It is quite likely that there is at least one student in every public school across the country that uses drugs and takes them to school. But would this mean that every school is drug infected? What people need to know is that there may be huge differences between what students say about a school and what is actually happening. For instance, what does the principal or local law enforcement say? Have there been any arrests? Any fights or incidents of violence? Without reading the entire study, these things cannot be known.
So why is this a problem? Because the contents of the press release are being reported as fact. No questions asked. I have found three news articles based on the press release. Not one of them contains any information that is not found in the press release. Not one of them adds anything that isn’t there. Not one of them quotes one of the researchers from an interview. Not one of them presents a different viewpoint. In other words, not one of the reporters has actually done the job of being a reporter, instead they have each been parrots, repeating only what they have been told and this could be misleading the public.
Of course, none of this means that there is anything wrong with the study. It could be that the researchers built in several controls to protect the integrity of the data they were collecting. But this is not indicated in a press release. Journalists who simply parrot press releases are lazy and irresponsible. Questions should have been asked and other research consulted. This is a misleading and potentially damaging misuse of science.
Let’s return momentarily to the discussion I had on that other blog. One of the studies referenced as support for the other guy’s position was conducted by Chandy, Blum, & Resnick (1996). I was not able to obtain a copy of that study, but I did find a similar study conducted by the same authors in 1997. The first study compared the effects of sexual abuse of boys with that of girls. The second only examined the boys. It appears that in both studies the authors used the exact same data. The sample size for the boys was exactly the same (n=370). The data came from the Adolescent Health Survey conducted in Minnesota in 1986-7, a study that appears to have been conducted by Resnick, Blum and others. The reason sample of 370 is significant and links both studies to the same data is that 370 is the exact number of boys indicating sexual abuse in the survey. Therefore, while I do not have the data source from the referenced article, I am confident it is the same. This link allows me to comment on the referenced study although I have not read it.
The central theme of the discussion on that other site was that one of the studies I referenced in my article indicated that sexually abused boys view their experiences as more positive than sexually abused girls. I drew the conclusion that this would mean they suffered less trauma. The studies referenced on the other site indicate that boys may suffer more trauma than girls as a result of sexual abuse. My position was that sampling and methodological differences between the studies may account for the different results. This appears to have been correct, though not necessarily in the way I initially thought.
The Chandy study defined sexual abuse in an odd manner. According to Chandy et al, “sexual abuse is when someone in your family, or someone else, touches you in a place you did not want to be touched, or does something to you sexually which they shouldn’t have done” (p. 5). This definition is vague and could be somewhat misleading. For instance, touching someplace you did not want to be touched could include a tap on the shoulder if the respondent did not wish to be touched there, but it would not necessarily be sexual abuse. Second, it would rule out being touched on the genitals by a significantly older person if the respondent wanted to be touched there. Thus the results may include instances where no abuse took place or exclude instances that might be considered statutory rape. The latter is probably more likely. This would certainly account for a difference in results.
Boys who viewed their experiences positively and may not be as traumatized by those experiences would likely not respond to the survey by indicating they were abused (they might have believed they “wanted” it to happen or not believe that it was something they “shouldn’t have done.” Thus they would not be part of the sample used by Chandy et al. But since the other study inquired about specific actions independent of desire or interpretation, they would show up in that sample. This could certainly influence the results and create a difference large enough for the researchers to reach opposing conclusions.
Thus, as I stated on the other site, my hypothesis that “…it could be the case that boys who do suffer trauma, may suffer more than girls who suffer trauma, but that a lower percentage of boys actually suffer trauma as a result of being abused” is supported by the examination of the differences between the two studies. Chandy found that boys suffer more trauma, but likely excluded a large number of cases that were less traumatic. Again, the use of abstracts, without evaluating the entire article, to support a position can lead to misleading conclusions or assumptions.
The reference to reports and studies and statistics is something that I see quite often in the mainstream media, and also on blogs. It is much more forgivable on a blog, especially if written by a nonprofessional. I am guilty of it myself. I have occasionally used statistics without going to the source. This is one reason why bad statistics never die. One person quotes the stat, then another person quotes the person who quoted the stat, and someone else quotes that person until no one knows where it originated, by everyone assumes it is true, so they keep using it. This is bad science and misuse of scientific data. It turns myth into fact and in the wrong hands can be used to influence public policy.
Nonprofessionals and bloggers should be obligated to get their facts straight, but are not obligated to cite every source. Know where your information comes from and don’t misuse science.
References
Chandy, J. M., Blum, R. W., & Resnick, M. D. (1997). Sexually abuse male adolescents: How vulnerable are they? Journal of Child Sexual Abuse , 6 (2), 1-16.
TDOM
Category: Psychology, Social concerns |
No Comments »
Tags: Center on Addiction and Substance Abuse, Columbia University, National Survey of American Attitudes, Research, Sexual abuse, Substance abuse, United States, World Economic Forum