https://github.com/jagilley/fact-checker

Fact-checking LLM outputs with self-ask
https://github.com/jagilley/fact-checker

llm python

Last synced: about 1 month ago
JSON representation

Fact-checking LLM outputs with self-ask

• Host: GitHub
• URL: https://github.com/jagilley/fact-checker
• Owner: jagilley
• Created: 2022-12-06T23:58:18.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-10-23T21:08:13.000Z (8 months ago)
• Last Synced: 2024-02-08T23:34:00.064Z (5 months ago)
• Topics: llm, python
• Language: Jupyter Notebook
• Homepage:
• Size: 6.84 KB
• Stars: 252
• Watchers: 5
• Forks: 40
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Lists

• awesome-langchain - Fact Checker - checking LLM outputs with langchain  (Open Source Projects / Other / Chatbots)
• awesome-langchain-zh - Fact Checker - checker?style=social): 使用 langchain 核实 LLM 输出的事实 (开源项目 / 其他聊天机器人)

README

        
# fact checking with prompt chaining

This repo is a simple demonstration of doing fact-checking with prompt chaining. How it works:

- you ask your desired LLM a question

- the LLM generates an initial answer to the question

- the LLM self-interrogates what the assumptions were that went into that answer

- the LLM sequentially determines if each of these assumptions are true

- the LLM generates a new answer to the question, incorporating the new information

## to run

Run

`python3 fact_checker.py 'insert question here'`

*Be sure to wrap your question in quotes if you're passing it as a command line argument.*

Alternatively, you can use the provided `fact_checker.ipynb` notebook.

## example

Question: *"What type of mammal lays the biggest eggs?"*

**Initial answer:**

The biggest eggs laid by any mammal belong to the elephant.

**Assumptions made:**

- The elephant is a mammal 

- Mammals lay eggs 

- Eggs come in different sizes 

- Elephants lay bigger eggs than other mammals

**Verification of assumptions:**

- The elephant is a mammal: TRUE 

- Mammals lay eggs: FALSE - Most mammals give birth to live young.

- Eggs come in different sizes: TRUE 

- Elephants lay bigger eggs than other mammals: FALSE - Elephants do not lay eggs.

**New answer:**

This question cannot be answered because elephants do not lay eggs and most mammals give birth to live young.

## credits

Proof of concept by [Jasper](https://twitter.com/0xjasper)