Fact Checking Myself and My Podcast

Background

I started adding fact checks to my podcast show notes when it became very cheap and easy to get transcripts.

Until today, I typed up a prompt each time with my transcript into Chat GPT.

I usually feel good about the results. Almost all the inaccurate and false information is more hyperbole or GPT missing the context, like humor.

Today, for whatever reason, I wondered if GPT has been pandering to me. That might explain how and why my fact checks come back so positive. Here is a fact check of episode 60. This is an example of a simple quick fact check response.

Introduction

After some trial and error, I've created a thorough prompting for future fact checks. This is mostly a post for me. The following is a prompt that I can/will use in the future to do my fact checks.

This will be a work in progress project for me.

Fact Checking Prompts

I'm going to use multiple phases. The first phase is an extraction of factual claims. The second phase will be the fact checking of the claims and links to further information. I read that the LLMs will do better if tasks are separated. After following that advice, it's obvious.

I've modified what GPT recommended to fit my preferences. Then I fed my revised copy back to GPT and then revised that again. I'll take any and all feedback!

First Prompt: (This is what I will copy and paste with the transcript in the future)

Extract factual claims from the attached transcript. Do not evaluate whether the claims are true yet.

Do not summarize the transcript. Do not assume the speakers are correct. Treat the transcript as a conversation that may contain errors, exaggerations, missing context, misleading phrasing, inaccurate information, and false information.

1. Create a claim inventory with these columns:Claim ID
2. Timestamp / transcript location
3. Speaker, if identifiable
4. Exact claim or close paraphrase
5. Claim type:
• External factual claim: about real-world history, biography, science, law, publication history, adaptation history, statistics, quotations, terminology, etc.
• In-universe plot claim: about what happens inside the book, film, game, or fictional world
• Interpretation / analysis: thematic, symbolic, moral, literary, psychological, or speculative reading
• Personal anecdote: speaker’s own experience or opinion
1. Source claim: claim that a source, interview, article, book, or scholar says something
6. Category — history, science, law, economics, biography, statistics, quotation, chronology, terminology, literature, adaptation history, psychology, medicine, etc.
7. Verification priority:
• High: names, dates, “first/only/oldest/most/never/always,” direct quotations, scientific claims, legal claims, medical claims, statistics, publication history, biography, claims about author intent, or claims likely to damage credibility if wrong
• Medium: claims that are factual but low-stakes or easily corrected
• Low: plot details, loose interpretations, personal impressions, or minor details
8. Why it needs verification, if applicable
9. Suggested source type:
• Primary text
• Scholarly source
• Official record
• Peer-reviewed science
• Government or university source
• Major reference work
• Reputable journalism
• Needs specialist source

Important rules:

• Do not fact-check yet.
• Do not collapse distinct factual claims into one row if they require different sources.
• Do not over-extract every minor plot beat unless it affects interpretation or could be misstated publicly.
• Separate “the speaker says X” from “X is true.”
• Mark speculative language clearly: “speaker speculates,” “speaker infers,” “speaker interprets.”
• Preserve claims using words like “first,” “only,” “oldest,” “most,” “never,” “always,” “everyone,” and “none” exactly, because these are high-risk.

Second Prompt: (Copy & Paste)

Using the extracted claim inventory, fact-check the High-priority claims first, then Medium-priority claims if space allows. Do not spend equal time on low-risk plot details unless they affect a larger factual or interpretive claim.

1. Create a table with these columns:Claim ID
2. Timestamp / transcript location
3. Claim — exact claim or close paraphrase
4. Claim type — external factual claim, in-universe plot claim, interpretation, personal anecdote, or source claim
5. Category — history, science, law, economics, biography, statistics, quotation, chronology, terminology, literature, adaptation history, psychology, medicine, etc.
6. Fact-check status — accurate, mostly accurate, misleading, unsupported, disputed, false, needs more evidence, or speculative
7. Most accurate version — give the most objective corrected version of the claim
8. Source(s) — provide links or citations
9. Source relevance note — explain exactly what the source proves and what it does not prove
10. Expert consensus — high, moderate, low, no consensus, or not applicable
11. Type of evidence — primary text, peer-reviewed scholarship, systematic review, government source, court record, historical record, university press, major reference work, reputable journalism, anecdotal, etc.
12. Risk level — high, medium, or low, based on how damaging the error would be if published
13. Notes / caveats / opposing evidence
14. Recommended edit — keep, revise, qualify, remove, or move to speculation

Status rules:

• Accurate: directly supported by strong evidence.
• Mostly accurate: basically right but missing nuance or minor correction.
• Misleading: partly true but phrased in a way likely to mislead listeners.
• Unsupported: no reliable source found after checking, or the claim is asserted without sufficient evidence.
• Disputed: credible sources disagree.
• False: contradicted by reliable evidence.
• Needs more evidence: plausible, but not enough reliable evidence was found to classify confidently.
• Speculative: an inference, interpretation, or hypothesis rather than a verifiable fact.

Source rules:

• Do not merely provide a link; explain exactly what each source proves, and mark any part of the claim that the source does not prove.
• Do not cite a source unless it directly supports the statement.
• Prefer primary sources and scholarly sources over blogs, Wikipedia, Reddit, YouTube, or unsourced articles.
• For books, films, and literary claims, prefer the primary text, scholarly criticism, author interviews, university presses, and major reference works.
• For science claims, prefer peer-reviewed papers, systematic reviews, government science agencies, medical institutions, or university sources.
• For history claims, prefer academic historians, primary documents, university presses, archives, or major reference works.
• For law claims, prefer statutes, court opinions, court filings, official government pages, and reputable legal analysis.
• For current events, use the most recent reliable sources and include publication dates.
• For people, verify names, dates, job titles, affiliations, direct quotations, and whether the person actually said the quoted material.
• For claims using numbers, rankings, “first,” “only,” “oldest,” “never,” “always,” “everyone,” or “most,” check especially care
• fully.
• If a claim is about author intent, do not infer intent from the text alone. Use interviews, letters, essays, biographies, or scholarship. If none exist, mark it as interpretation or needs more evidence.
• If a claim is about what happens in a fictional work, verify against the primary text or film where possible.

Third Prompt: (Copy & Paste)

From the fact-check table, create a short “Top Corrections and Accuracy Assessment” section for the beginning of my report.

Include:

1. A two-sentence overall accuracy assessment of the transcript.
2. A ranked list of the most important corrections, sorted in this order:
• False
• Misleading
• Disputed
• Unsupported
• Needs more evidence
3. For each correction, include:
• Timestamp / transcript location
• Original claim
• Fact-check status
• Corrected claim
• Risk level
• One sentence explaining why the correction matters
4. Exclude minor details unless they affect credibility, interpretation, or factual accuracy.
5. Keep this section concise enough to paste at the top of a written report. 

Conclusion

Separating the fact check into extract and then evaluate creates a more detailed fact check. See an example here, Ep 64.

As of now, I'm not 100% satisfied, but until next time...

Life According to Morla the Aged One

I'm rereading The Neverending Story with my daughter. It's a very fun story. It's one of the books I've been looking forward to reading when my daughter was old enough.

The quote below almost reminds me of my friend Brendan. We recently discussed purpose here, among other topics.​ It's been part of a continuous discussion since we first discussed my favorite book Siddhartha.

The Neverending Story is a story within a story. In the novel, the book The Neverending Story has two snakes eating each other by the tail. Reading the description reminds me of a yin yang like symbol.

In chapter three, the story wanders to the Swamps of Sadness-where the oldest being in Fantastica lives. Morla the Aged One is a tortoise. She speaks to herself calling herself "old woman" and uses the first person plural "we." Possibly a form depressive disorder from her environment. She resist helping and insist:

“Sakes alive!” Morla gurgled. “We’re old, son, much too old. Lived long enough. Seen too much. When you know as much as we do, nothing matters. Things just repeat. Day and night, summer and winter. The world is empty and aimless. Everything circles around. Whatever starts up must pass away, whatever is born must die. It all cancels out, good and bad, beautiful and ugly. Everything’s empty. Nothing is real. Nothing matters.”

Morla reminds me of Benjamin from Animal Farm. But, unlike Benjamin, Morla lives in a fantasy land. It's hard to say how much of Morla's apathy is due to her age versus living in the Swamps of Sadness. 

Jimbo's Podcasting Progression

I recently discussed why I podcast with my friend Brendan, listen here.

I mentioned how I progressed due to listening to myself and also being the editor. My errors costs me more time and work editing. They also cost me psychologically stress hearing myself make the errors. The story I tell myself is that I got better.

It's difficult to say because I edited those podcasts. My memory says the editing got better. But I could have just gotten better at editing.

Either way, here is little experiment. Below is the first podcast I published and one of the better podcasts we produced.

Ep 2 My Name Is Earl “Pilot”
Ep #67 Extended Discussion: Russian Doll “Nothing in this World Is Easy”

Both episodes are on the big podcast apps. 

For anyone who gives them both a listen, I'd love to hear what you think.

Jimbo out!