Illuminations: Evaluating Research Articles

In this installment of Illuminations, I would like to educate readers on how to evaluate research articles. I will use Lisa Littman’s (first) article on Rapid Onset Gender Dysphoria (ROGD) as an example because so many anti-trans groups cite the
Littman paper on ROGD as gospel without acknowledging that it is problematic on many levels. Next time I will directly
address the concept of ROGD.

Each part of a research article has a standard format and goal and contains valuable information which speaks to the issue of its professionalism. The publisher, title and author(s) are the first bit of information to evaluate.
Most scholarly journals contain articles describing high quality research that has been peer reviewed by experts in the field prior to publication. Professional journals are given impact ratings which can be found online. Information about the primary author can also be found. Consider the impact of the journal, the bias of the publisher and facts about the author.

The first ROGD paper by Lisa Littman was published in PLOSone in August 2018. The impact factor for PlosOne is currently 2.74 and has been on a steady decline. As a point of reference, the JAMA impact factor is 45.54.
PLOSone is a pay to publish journal which prints about 50% of all submissions received. It publishes articles with the aim of having the scientific community ascertain the importance, post publication, through debate and comment.

Lisa Littman is not a family therapist or psychologist. She is a Public Health expert that is interested in detransition. Many immediately criticized the Littman study to a degree never seen before. Due to the magnitude of criticism, PLOSone reevaluated the article just two weeks after publication and asked Littman to publish a revised and corrected version. The PLOS One Editor said, “we have reached the conclusion that the study and resultant data reported in the article represent a valid contribution to the scientific literature. However, we have also determined that the study, including its goals, methodology, and conclusions, were not adequately framed in the published version, and that these needed to be corrected.”

On the same day that PLOS One announced its unfavorable review, Brown University (Littman’s employer), retracted its press release promoting the study. They stated, “As a research institution, we feel we must ensure that work that is featured on the
University website conforms to the highest academic standards. Given the concerns raised about research design and methods, the most responsible course of action was to stop publicizing the work published in this particular instance.”

Before understanding the problems with the Littman article, one must understand what an article should be:

Research articles tend to have 6 parts, and each part is clearly labeled. As noted by PlosOne, the Littman article had significant faults in 3 areas: Goals, Methodology, and Conclusions.

The Abstract is the first part of the article, normally at the top and set apart from the rest of the article. The abstract is a very brief summary which describes what the article is about and summarizes each of the article sections.

In the Introduction and Literature Review, the researchers explain why they selected the topic to study and why it is important. They then summarize prior research that is relevant to their study. All prior research that is cited is listed in the last section of the article, the References.

In the Introduction The stated Goals of the Littman study were to describe the presentation of ROGD and generate hypotheses based on the premise that “social and peer contagion” is a key determinant of ROGD. This is not the typical format of scientific research which more commonly presents a hypothesis and then tests it.

The specific problem with this goal is that ROGD is not recognized by any major professional association and is not a diagnosis. ROGD is a made-up subtype of gender dysphoria that is based on un accepting parents’ accounts of their teenage children suddenly manifesting symptoms of gender dysphoria and self-identifying as transgender. Littman proposed a phenomena ROGD and presented it as if it were an agreed upon, valid, and reliable ‘thing’. She then attempted to find evidence of this made-up phenomena.

Finally, the “literature” that Littman cites to describe ROGD is from 4thwavenow, Transgender Trend, and Youth Trans Critical Professionals (all very anti-trans groups). This should be a red flag.

The Methods or Data Analysis portion of the article explains how the researchers conducted the research. It includes information on the participants and data collection methods used explains how the data was analyzed. Typically, the prediction, expected outcome or hypothesis is that there is nothing significant or no relationship between two variables and the data collected is analyzed statistically to disprove that. Research methodology requires that findings be statistically significant, rather than assumed. There were a few problems with the Methodology in the Littman study. Even without understanding statistics, questions to ask about this section include was the sample big enough and representative of the general population and were the measures used reliable and meaningful? “Data” in the Littman study was obtained from a survey placed on three websites for concerned parents of children with gender dysphoria, asking for responses from parents whose children had experienced “sudden or rapid development of gender dysphoria beginning between the ages of 10 and 21.” Littman surveyed around 250 parents recruited from three websites where she had seen parents describe sudden gender transitions in their adolescents, practically guaranteeing that she would get the “data” she was looking for. This qualifies as a type of sampling bias that researchers know as self-selection. It makes the validity and reliability of findings dubious. Furthermore, the study was criticized for lacking any clinical data or responses from the adolescents themselves. She did not ask trans youth about their experience of “rapid onset” gender dysphoria – only their disbelieving parents.

In the Results, the authors tell you what they found, basically explaining how the data translates into significant findings or not. The horrible sampling method used by Littman makes statistical analysis of the data meaningless. No conclusions can be made. When an argument’s premises assume the truth of the conclusion, instead of supporting it, one is engaging in a type of circular logic commonly known as “begging the question.” It neither proves nor disproves anything.

In the Discussion, the authors discuss how their findings (results) tie back into the other research done in the field and why what they found is meaningful. They typically do some self-critique, discuss limitations, and may also give ideas for further research.

The Littman paper was republished with an updated Title, Abstract, Introduction, Methodology, Discussion, and Conclusion sections. Only the Results section was mostly unchanged. Let’s briefly look at how the revised article addressed these problems. In the revision, Littman did admit in writing that, “Rapid-onset gender dysphoria (ROGD) is not a formal mental health diagnosis at this time.” Accordingly, in her correction, Littman re-worded the Goal of the study to be “a study of parental observations which serves to develop hypotheses”.

Littman could not revise the Methodology and the correction did not address the issue of selection. Littman softened her Conclusions by adding that, “Additional research that includes adolescents and young adults, along with consensus among experts in the field, will be needed to determine if what is described here as rapid-onset gender dysphoria (ROGD) will become a formal diagnosis.”

Next time I will address why ROGD will never become a formal diagnosis.

*       Littman, L. L. (2018). Rapid-onset gender dysphoria in adolescents and young adults: A study of parental reports. PLoS ONE,13(8), e0202330. https://doi.org/10.1371/journal.pone.0202330
*       Littman, L.L. (2019). Correction: Parent reports of adolescents and young adults perceived to show signs of a rapid onset of gender dysphoria. PLoS ONE 14(3): e0214157. https://doi.org/10.1371/journal.pone.0214157