Fact Checking Myself and My Podcast

Background

I started adding fact checks to my podcast show notes when it became very cheap and easy to get transcripts.

Until today, I typed up a prompt each time with my transcript into Chat GPT.

I usually feel good about the results. Almost all the inaccurate and false information is more hyperbole or GPT missing the context, like humor.

Today, for whatever reason, I wondered if GPT has been pandering to me. That might explain how and why my fact checks come back so positive. Here is a fact check of episode 60. This is an example of a simple quick fact check response.

Introduction

After some trial and error, I've created a thorough prompting for future fact checks. This is mostly a post for me. The following is a prompt that I can/will use in the future to do my fact checks.

This will be a work in progress project for me.

Fact Checking Prompts

I'm going to use multiple phases. The first phase is an extraction of factual claims. The second phase will be the fact checking of the claims and links to further information. I read that the LLMs will do better if tasks are separated. After following that advice, it's obvious.

I've modified what GPT recommended to fit my preferences. Then I fed my revised copy back to GPT and then revised that again. I'll take any and all feedback!

First Prompt: (This is what I will copy and paste with the transcript in the future)

Extract factual claims from the attached transcript. Do not evaluate whether the claims are true yet.

Do not summarize the transcript. Do not assume the speakers are correct. Treat the transcript as a conversation that may contain errors, exaggerations, missing context, misleading phrasing, inaccurate information, and false information.

1. Create a claim inventory with these columns:Claim ID
2. Timestamp / transcript location
3. Speaker, if identifiable
4. Exact claim or close paraphrase
5. Claim type:
• External factual claim: about real-world history, biography, science, law, publication history, adaptation history, statistics, quotations, terminology, etc.
• In-universe plot claim: about what happens inside the book, film, game, or fictional world
• Interpretation / analysis: thematic, symbolic, moral, literary, psychological, or speculative reading
• Personal anecdote: speaker’s own experience or opinion
1. Source claim: claim that a source, interview, article, book, or scholar says something
6. Category — history, science, law, economics, biography, statistics, quotation, chronology, terminology, literature, adaptation history, psychology, medicine, etc.
7. Verification priority:
• High: names, dates, “first/only/oldest/most/never/always,” direct quotations, scientific claims, legal claims, medical claims, statistics, publication history, biography, claims about author intent, or claims likely to damage credibility if wrong
• Medium: claims that are factual but low-stakes or easily corrected
• Low: plot details, loose interpretations, personal impressions, or minor details
8. Why it needs verification, if applicable
9. Suggested source type:
• Primary text
• Scholarly source
• Official record
• Peer-reviewed science
• Government or university source
• Major reference work
• Reputable journalism
• Needs specialist source

Important rules:

• Do not fact-check yet.
• Do not collapse distinct factual claims into one row if they require different sources.
• Do not over-extract every minor plot beat unless it affects interpretation or could be misstated publicly.
• Separate “the speaker says X” from “X is true.”
• Mark speculative language clearly: “speaker speculates,” “speaker infers,” “speaker interprets.”
• Preserve claims using words like “first,” “only,” “oldest,” “most,” “never,” “always,” “everyone,” and “none” exactly, because these are high-risk.

Second Prompt: (Copy & Paste)

Using the extracted claim inventory, fact-check the High-priority claims first, then Medium-priority claims if space allows. Do not spend equal time on low-risk plot details unless they affect a larger factual or interpretive claim.

1. Create a table with these columns:Claim ID
2. Timestamp / transcript location
3. Claim — exact claim or close paraphrase
4. Claim type — external factual claim, in-universe plot claim, interpretation, personal anecdote, or source claim
5. Category — history, science, law, economics, biography, statistics, quotation, chronology, terminology, literature, adaptation history, psychology, medicine, etc.
6. Fact-check status — accurate, mostly accurate, misleading, unsupported, disputed, false, needs more evidence, or speculative
7. Most accurate version — give the most objective corrected version of the claim
8. Source(s) — provide links or citations
9. Source relevance note — explain exactly what the source proves and what it does not prove
10. Expert consensus — high, moderate, low, no consensus, or not applicable
11. Type of evidence — primary text, peer-reviewed scholarship, systematic review, government source, court record, historical record, university press, major reference work, reputable journalism, anecdotal, etc.
12. Risk level — high, medium, or low, based on how damaging the error would be if published
13. Notes / caveats / opposing evidence
14. Recommended edit — keep, revise, qualify, remove, or move to speculation

Status rules:

• Accurate: directly supported by strong evidence.
• Mostly accurate: basically right but missing nuance or minor correction.
• Misleading: partly true but phrased in a way likely to mislead listeners.
• Unsupported: no reliable source found after checking, or the claim is asserted without sufficient evidence.
• Disputed: credible sources disagree.
• False: contradicted by reliable evidence.
• Needs more evidence: plausible, but not enough reliable evidence was found to classify confidently.
• Speculative: an inference, interpretation, or hypothesis rather than a verifiable fact.

Source rules:

• Do not merely provide a link; explain exactly what each source proves, and mark any part of the claim that the source does not prove.
• Do not cite a source unless it directly supports the statement.
• Prefer primary sources and scholarly sources over blogs, Wikipedia, Reddit, YouTube, or unsourced articles.
• For books, films, and literary claims, prefer the primary text, scholarly criticism, author interviews, university presses, and major reference works.
• For science claims, prefer peer-reviewed papers, systematic reviews, government science agencies, medical institutions, or university sources.
• For history claims, prefer academic historians, primary documents, university presses, archives, or major reference works.
• For law claims, prefer statutes, court opinions, court filings, official government pages, and reputable legal analysis.
• For current events, use the most recent reliable sources and include publication dates.
• For people, verify names, dates, job titles, affiliations, direct quotations, and whether the person actually said the quoted material.
• For claims using numbers, rankings, “first,” “only,” “oldest,” “never,” “always,” “everyone,” or “most,” check especially care
• fully.
• If a claim is about author intent, do not infer intent from the text alone. Use interviews, letters, essays, biographies, or scholarship. If none exist, mark it as interpretation or needs more evidence.
• If a claim is about what happens in a fictional work, verify against the primary text or film where possible.

Third Prompt: (Copy & Paste)

From the fact-check table, create a short “Top Corrections and Accuracy Assessment” section for the beginning of my report.

Include:

1. A two-sentence overall accuracy assessment of the transcript.
2. A ranked list of the most important corrections, sorted in this order:
• False
• Misleading
• Disputed
• Unsupported
• Needs more evidence
3. For each correction, include:
• Timestamp / transcript location
• Original claim
• Fact-check status
• Corrected claim
• Risk level
• One sentence explaining why the correction matters
4. Exclude minor details unless they affect credibility, interpretation, or factual accuracy.
5. Keep this section concise enough to paste at the top of a written report. 

Conclusion

Separating the fact check into extract and then evaluate creates a more detailed fact check. See an example here, Ep 64.

As of now, I'm not 100% satisfied, but until next time...