Hype alert: new AI writing detector claims 99% accuracy
Multiple media outlets are reporting on a recent study published by University of Kansas researchers. Per these reports, the KU researchers have achieved near-perfect accuracy at detecting AI use in “scientific writing” or “academic writing.”
But let’s take a look at what this study actually did.
The task
The researchers trained a model to distinguish between two types of writing:
Human text samples consisting exclusively of Perspectives articles in the journal Science from 2020-2023
ChatGPT responses to prompts like: “Can you produce a 300- to 400-word summary on this topic: A surprising fossil vertebrate.”
The training and test data
The data listed above made up the training and test sets. The researchers hoped the prompt format above would generate output similar to Perspectives articles. But very little effort was actually put into making the outputs look similar. For example, they could have included some Perspectives articles, or even just some small excerpts, in the ChatGPT prompts as examples to mimic, but they didn’t. And the ChatGPT output was used as is, without any modification or paraphrasing. As a result, the human and AI articles are noticeably different, even to a human eye (see examples in the Appendix below).
The researchers also made their training and test sets extremely similar. The only difference was which year the human-written Perspectives articles were published. This made it pretty easy to achieve 99% accuracy on the test set.
Comparison to “state-of-the-art” AI writing detectors
As is typical in machine learning papers, the researchers compared their new technique to the existing state-of-the-art method — sort of. They chose the OpenAI GPT-2 detector as their “state-of-the-art” model for comparison. As the name suggests, it was trained on text from GPT-2, not ChatGPT. On top of this, the researchers’ model had another unfair advantage in this matchup, because it was fine tuned specifically to distinguish Perspectives articles from ChatGPT output, whereas the GPT-2 detector wasn’t fine tuned for this task; it’s a general-purpose AI writing detector. Even with these disadvantages, the GPT-2 detector was able to classify the articles in the test set with 92-94% accuracy – underlining the fact that the classification task wasn’t difficult.
To recap, the study:
Only concerned a very narrow type of writing (Perspectives articles in the journal Science)
Didn’t put much effort into making the AI text similar to the human text
Used a test set that was extremely similar to the training set, and
Reported an improvement over an outdated, general-purpose AI writing detector
In summary, nothing really noteworthy was accomplished here. The state of AI writing detectors remains the same (which is generally pretty dismal).
Multiple mainstream media outlets are uncritically repeating the claims of the study. Here are a few for your amusement:
University of Kansas Researchers Claim 99% Accuracy Detecting ChatGPT Fakes (Gizmodo)
New Tool Can Tell If Something Is AI-Written With 99% Accuracy (Forbes)
Scientists claim >99 percent identification rate of ChatGPT content (The Register)
University of Kansas Researchers Develop Near-Perfect AI Detector (Motley Fool)
It’s almost as if they just regurgitated the KU press release.
Appendix: Examples of human- and AI-written text from the study
These are just a few examples to provide the flavor of the training & test data in the study.
Only the first paragraph of each article is shown for both human- and AI-written content. Later paragraphs were also used in the study. The study treated each paragraph as a training or test example. Predictions for entire articles were made by aggregating the predictions from all component paragraphs.
For each human-written article, the KU researchers used two different prompts to generate ChatGPT equivalent articles. The first prompt used the article title, and the second one used a brief article description written by the researchers. Both were included in experiments and are shown below.
Example 1
Monkeypox: The consequences of neglecting a disease, anywhere by humans Oyewale Tomori and Dimie Ogoina
Human text
Monkeypox (MPX) is a zoonotic disease caused by the monkeypox virus (MPV), which is a double-stranded DNA virus belonging to the genus Orthopoxvirus, which includes variola virus, the cause of smallpox. The first human case of MPX was reported in a 9-month-old boy in the Democratic Republic of Congo (DRC) in 1970, and MPX cases remained infrequent, until recently. Currently, MPX is endemic in the rainforest areas of West and Central Africa, where human MPX outbreaks often occur, especially in rural settings, owing to spillover events from animal reservoirs and occasionally from human-to-human transmission within households. On 6 May 2022, a new phase of MPX began when the first case of the disease, not associated with travel from Africa, was reported in the UK. There is now substantial human-to-human spread of MPX in nonendemic countries. What is the history of MPX and can this help inform control measures now?
ChatGPT text
Response to prompt “Can you produce a 300 to 400 word summary on this topic: Monkeypox: The consequences of neglecting a disease, anywhere”
Monkeypox is a rare but potentially deadly viral disease that has been spreading in several countries in Central and West Africa, including Nigeria, Cameroon, and the Democratic Republic of Congo. The disease, which is similar to smallpox but less severe, can cause fever, rash, and lesions on the skin, and can lead to severe complications, including respiratory failure and neurological damage.
Response to prompt “Can you produce a 300 to 400 word summary on this topic: What is the history of MPX and can this help inform control measures now?”
MPX, or monkeypox, is a viral disease that affects both humans and animals. It is caused by the Monkeypox virus, a member of the Orthopoxvirus family, which also includes smallpox and vaccinia viruses. Monkeypox was first identified in 1958 in monkeys kept for research purposes, but the first human case was reported in 1970 in the Democratic Republic of the Congo (DRC). Since then, several outbreaks have occurred in various countries in Central and West Africa.
Example 2
Exaggerated responses to a virus long gone by human Petter Brodin
Human text
The clearest determinant of COVID-19 severity is age, with the majority of children experiencing mild or asymptomatic infections. But one disease presentation observed during the peak of the pandemic shows an opposite pattern. The rare but sometimes life-threatening multisystem inflammatory syndrome in children (MIS-C) has been a major cause of pediatric morbidity and mortality during the pandemic. Symptoms of MIS-C overlap partially with Kawasaki disease, a postinfectious vasculitis, and with toxic shock syndrome, a bacterial toxin–mediated disease caused by nonspecific activation of T lymphocytes by so-called superantigens. On page 554 of this issue, Lee et al. report that variants in genes encoding the 2′-5′-oligoadenylate synthetase (OAS)–ribonuclease L (RNase L) viral RNA sensing pathway lead to exuberant inflammatory responses in myeloid cells in individuals with MIS-C. But why does MIS-C develop 1 month after the initial infection?
ChatGPT text
Response to prompt “Can you produce a 300 to 400 word summary on this topic: Exaggerated responses to a virus long gone”
In recent times, there have been cases of exaggerated immune responses to viruses that are no longer present in the body. This phenomenon is known as a post-viral syndrome or post-viral fatigue syndrome. The condition is characterized by persistent symptoms such as fatigue, muscle aches, brain fog, and other flu-like symptoms long after the virus has been cleared from the body.
Response to prompt “Can you produce a 300 to 400 word summary on this topic: multisystem inflammatory syndrome in children”
Multisystem inflammatory syndrome in children (MIS-C) is a rare but severe condition that has emerged in children during the COVID-19 pandemic. MIS-C occurs after children are exposed to the SARS-CoV-2 virus and can cause a variety of symptoms including fever, abdominal pain, vomiting, diarrhea, rash, and cardiac symptoms such as chest pain and shortness of breath.
Example 3
Christine Guthrie (1945–2022) by humans Suzanne Noble, Sean M. Burgess, and Evelyn Strauss
Human text
Christine Guthrie, molecular geneticist and mentor, died on 1 July at the age of 77. She was a pioneer in the field of RNA splicing, the process by which “nonsense” sequences are removed from eukaryotic messenger RNAs (mRNAs). Christine introduced many fundamental concepts that compose our modern view of this vital cellular activity. Her unerring logic, creative vision, and articulate voice transformed a nascent field and continue to influence the lives of her dozens of trainees.
ChatGPT text
Response to prompt “Can you produce a 300 to 400 word summary on this topic: Christine Guthrie (1945–2022)”
Christine Guthrie was a renowned American artist and professor who passed away in 2022. Born in 1945 in Brooklyn, New York, Guthrie earned her Bachelor of Fine Arts degree from the Pratt Institute in 1967 and went on to receive her Master of Fine Arts degree from Yale University in 1970.
Response to prompt “Can you produce a 300 to 400 word summary on this topic: molecular geneticist Christine Guthrie (1945–2022)”
Christine Guthrie was a molecular geneticist whose research focused on understanding the splicing of RNA, a critical process in gene expression that enables the production of different proteins from a single gene. She was born on April 3, 1945, in Ithaca, New York, and grew up in Syracuse. Guthrie graduated from Cornell University in 1967 and went on to receive a PhD in biochemistry from Stanford University in 1972. She then joined the faculty of the University of California, San Francisco (UCSF), where she spent the remainder of her career.