More and more we want to know: is it human or AI? Is this image real? This song? This video? Publishing platform Substack is trying to add some clarity by partnering with AI detector Pangram to help readers understand how much AI is present in all the content they consume on the publishing platform.
The tool is available to all Substack subscribers and works on posts, notes, comments, and replies published on or after Tuesday that are more than 100 words long. Creators will also be able to analyze their drafts with Pangram to see how much AI is detected and will also be able to provide a statement to explain their writing process, Substack said.
Web and iOS users can try Pangram’s features now, and Android users will have access later.
In a blog post announcing the new feature, Chris Best, co-founder and CEO of Substack, said his platform aims to avoid “Claudefishing,” which is when a reader consumes content without realizing it was generated by AI.
“When readers are left to question whether what they are reading is real, it undermines trust in authorship and threatens the livelihoods of writers, including those who use AI tools thoughtfully to produce work they believe in,” Best wrote.
From CNET: AI Slop is destroying the Internet. These are the people who are fighting to save him
Pangram estimates that about 40% of the text on some social media platforms comes from the AI gene.noted, with a superb two-thirds of it on the networking site LinkedIn. And AI growth agency Graphite recently published a white paper claiming that AI now writes as many articles online as humans. A Gartner survey found that 68% of people wonder if the content they see is real and that half of consumers prefer brands that don’t use AI in their marketing.
I test it by myself
I only recently joined Substack, but I wanted to check out the new Pangram AI detection tool. The tool works on posts, ratings, comments, and replies, so I decided to use posts as a makeshift testing lab. First, I logged into my Substack account and published a new post. I copied and pasted the text of an article I had written – by myself, without AI – into the Substack publication and published it.
After a few minutes, I clicked on the live post. In the three-dot menu at the top right of the page, I saw an option to “Find AI Text”. I clicked on it and a small window appeared accurately stating that my article was “written entirely by humans.” A line appears below that indicates the percentage of the text that is AI, AI-assisted, or human.
I then took the same message but added a long, fully AI-generated paragraph that I pasted from ChatGPT. When I ran the “AI Text Search,” the tool reported that 10% of the article was written by AI, a roughly accurate estimate.
I then published a completely AI-generated article, and the analysis correctly identified it as 100% AI-generated.
For another test, I added a few AI-generated sentences to a post that was otherwise purely human content, and the analysis reported it as 100% human-generated. Close, but not quite true. So the Pangram tool worked pretty well for my rough experimentation.
Although I was quite happy with how Substack’s AI detection system worked on my own content, I was disappointed to learn that one of my favorite podcaster-bloggers apparently went full AI in his latest post. However, other writers I looked at used entirely human-generated text.
What about false positives?
In his announcement, Best admitted that the AI detection tool is “not perfect,” but cited independent research that found Pangram’s software was 97.5 percent accurate in detecting entirely AI-generated text. The University of Maryland found that Pangram was 99.3% accurate in identifying “AI-generated humanized text,” which essentially means a human rewrote or paraphrased the AI content.
Substack writers who do not use AI are understandably concerned about being mislabeled as AI-assisted authors. As a result, the risk of false positives is a significant problem. Substack said creators can analyze their content to see if it will be labeled as containing AI while it is still in draft form. If Pangram incorrectly identifies AI text, the creator can report the error to Substack and have the false analysis removed.
To promote greater transparency, publishers can now include a “How I do this” statement if they want readers to know to what extent, if any, they have used AI in their content. Better said, Pangram cannot detect whether the AI was used to gather information or as a source of writings.
“We are not against using AI to make their jobs easier, and we believe people should be free to choose the tools they use to express themselves,” Best said. “But people should know what they’re getting.”
Since becoming fascinated by the Cambridge coffee shop webcam in the 1990s, I’ve written about VPNs, the NFL, smartphones, decent salaries, plus/minus and everything in between. See full bio
