Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/codingforentrepreneurs/reddit-content-research-agent

Build a Reddit Content Research Agent with LLMs, LangChain, SERP, Jupyter, Django, Bright Data, and much more.
https://github.com/codingforentrepreneurs/reddit-content-research-agent

brightdata django jupyter langchain langgraph reddit scraping serp

Last synced: about 2 months ago
JSON representation

Build a Reddit Content Research Agent with LLMs, LangChain, SERP, Jupyter, Django, Bright Data, and much more.

Awesome Lists containing this project

README 

          
[![Video Thumbnail](https://img.youtube.com/vi/XI-iP-qk_Vk/maxresdefault.jpg)](https://www.youtube.com/watch?v=XI-iP-qk_Vk)



# Reddit Content Research Agent



Using LangChain, Jupyter, Django, [Bright Data](https://kirr.co/9i3tja), and other tools, let's build an app that can find near real-time information on keywords and topics you care about all trending on Reddit and various sub-Reddits.



## Video Chapters

**🎥 [Watch Full Video: Building an AI-Powered Reddit Scraping System](https://www.youtube.com/watch?v=XI-iP-qk_Vk)**



- [00:00:00](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=0s) Welcome

- [00:03:46](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=226s) Demo ([final code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/18635852dbd04ba4d5b03dfe673b19071c616b0a))

- [00:12:03](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=723s) Using Search Engine Results

- [00:14:16](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=856s) Setup your Python Project ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/d669a4dd13613e52d15efbdfd11b898a9994dceb))

- [00:20:36](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=1236s) Load API Keys with Dotenv Files ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/d2c82fdabdf536db40423083d9f0c3b80c585efc))

- [00:24:26](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=1466s) Intro to LangChain

- [00:26:19](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=1579s) Bright Data Serp API with Python & LangChain ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/642a7019ed32b97723396e454416a7eb21aa58af))

- [00:38:01](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=2281s) Strip Notebook Outputs for Security with pre-commit ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/79ce005f1e4b73151292f6c43cb10d27b52121c6))

- [00:42:56](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=2576s) Setup Google Gemini Models with LangChain ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/9eb770cab51eaa96cdcaf5ca28129aadd1583fa9))

- [00:52:43](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=3163s) LLM with Structured Output ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/319300402466e573c5e69c278cc6a38d9efcb350))

- [00:59:58](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=3598s) LLM Tool Calling The Hard Way ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/7d4d7ef68b760d2946181053fc47112b9a8a4892))

- [01:08:19](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=4099s) Tool Calling with LangGraph ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/471c7dbd04113e86272df81f075e646e51a92380))

- [01:23:41](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=5021s) Search & Format Reddit Communities via LLM and Bright Data ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/d4f54053dc4068a659b9edaf21bd92f3c987e800))

- [01:29:38](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=5378s) Scrape Reddit with the Bright Data Crawl API ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/01119492ce657de9d4f0e6dedcaace563bf3caa2))

- [01:41:58](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=6118s) Get Crawl API Snapshot Progress ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/4faf585e231b12890347e41b3451fc3dc0363ea9))

- [01:47:00](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=6420s) Download Data from the Crawl API ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/f2f38af08e6f61e62668c1e9f8b39fa55d971a8f))

- [01:54:53](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=6893s) Automating Data Pulls for Users

- [01:58:39](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=7119s) Install & Start the Django Project ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/f392920e99bb2580874da73c55cf4121721c294f))

- [02:02:31](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=7351s) Combine Django with Jupyter ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/7057029eba690ac3e956b6d32aba671bd7f8a03a))

- [02:05:23](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=7523s) Implement Postgres Database with Django ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/6ce67a9a42c2128a7f17516d49146e27d8430f0a))

- [02:15:19](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=8119s) Setup Redis for Django & Caching ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/bf031d02d0abe94663428db538388421f1f7c7ba))

- [02:22:36](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=8556s) Getting Started with Celery & Django ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/d50f6bed0695f57c69738282022257e75eea1350))

- [02:33:51](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=9231s) Webhooks & Cloudflare Tunnels

- [02:36:47](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=9407s) Setup Cloudflare Tunnel with a Custom Domain ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/fb1f847441c5a55fb8c51542bc58ba3477a4f56a))

- [02:45:24](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=9924s) Django Qstash for Webhook-based Background Tasks ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/5b8f673b79ebd18f3ff646efe62c46f1bb7aa7df))

- [02:52:55](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=10375s) Bright Data to Django Model Part 1 ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/b748e2570854e0ec609c7387668b571e9dbfa3b0))

- [03:02:16](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=10936s) Bright Data to Django Model Part 2 ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/b748e2570854e0ec609c7387668b571e9dbfa3b0))

- [03:09:38](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=11378s) Store Bright Data Snapshots ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/bf045d6804247adfd0d3ee7e9ee16b7cc46900f3))

- [03:17:38](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=11858s) Helper Functions for Scraping Events Part 1 ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/3b76912e3e7eeec8fc1a43e96a896399bd3310f6))

- [03:24:56](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=12296s) Helper Functions for Scraping Events Part 2 ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/3b76912e3e7eeec8fc1a43e96a896399bd3310f6))

- [03:32:52](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=12772s) Saving Snapshot Scraping Results ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/1a7b5b5330fbd5171ca29dd0999150795e6de806))

- [03:38:29](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=13109s) Configure Scraping as Background Tasks ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/435218753e1a1ef2684f1235cc6fecff93a1e23a))

- [03:49:48](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=13788s) Run Background Scraping Tasks ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/48ded25b2de7b40b36cbd9b9bd1d0bc913865765))

- [03:53:48](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=14028s) Poll Scrape Status as Background Task ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/80b0eb4db19b0c758d0813b6da266e5965750e14))

- [04:02:04](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=14524s) Tracking Scrape Event Finished At Time ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/5199c3ed4a30accd0d97ce327aab3bb75a80337a))

- [04:08:36](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=14916s) A Webhook Handler View in Django ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/5c1a8f386ea3a3c4f42d081e66007d355f881483))

- [04:16:11](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=15371s) Tracking Scraping Snapshots through Webhooks with Django ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/3ee59c0c7164e615a0206cb63af0e7acd4780a20))

- [04:25:48](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=15948s) Improved Auth Key for Webhooks ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/5bf8e146450df3d5ac420bac0b97c99243ad02aa))

- [04:30:42](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=16242s) Webhook Handler for Reddit Posts ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/a5a065757a3c7b1a9746a96458ad16f8b99d4acc))

- [04:38:32](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=16712s) Adjust Data to Scrape ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/eb5fc4732c3d9e3bf6d4c10e79db5ed2207143b1))

- [04:50:47](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=17447s) Background Sync Snapshot Reddit Results ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/9ad4599646e2ec48409d61485a8d2344dc61fbcb))

- [05:04:39](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=18279s) Storing Reddit Communities in Django ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/7534eccf9779104190c772ae3e8773571a75c8fb))

- [05:16:53](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=18893s) Reddit AI Agent into Django Project ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/878cea21e0e439b772a412d0f3aaa4541116f080))

- [05:26:04](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=19564s) Topic Extraction Agent ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/99a81bb03a73a17bdc001d5662f7a85d5e81da1e))

- [05:32:50](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=19970s) Fuzzy Query to Scraping ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/a1179cd74ec26ec01f0cdb59ffbe7c948b531dfa))

- [05:40:45](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=20445s) Auto Scrape Reddit Communities on Save ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/3ed252380c9030286a8f6b0a55a1eef97a9e47b2))

- [05:52:37](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=21157s) Scraping Workflow as a Service Function ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/94fbe18e76fff91f6664f930b2fc6a12a0f21ed0))

- [06:00:17](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=21617s) Store Queries & Topics ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/06464b722be0d517ea8892d0ef755d215c7c4287))

- [06:09:24](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=22164s) Topics to Reddit Communities ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/483966a1388bff5ead548d3186698f6119c6286f))

- [06:16:41](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=22601s) Full Query Automation ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/c024d814cebd0c885cd3fd922b5b06309782cda1))

- [06:23:09](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=22989s) Reddit Community Trackablity ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/2bf0e6170c6ae9db34b38e55f056f7e95f73e079))

- [06:28:19](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=23299s) Scheduled Background Task to Trigger Reddit Scraping ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/8503189ae9404dce83c841e1f35b4ebeb7bcbae7))

- [06:33:27](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=23607s) Django Management Command to Trigger Scraping ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/1dcaf36637e2a2ac9d7f282f948dff59ab23e664))

- [06:36:17](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=23777s) Final Query Commands ([section code](https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent/tree/a89c1162137364ad21af3bff9d8532e0cc9b2d6c))

- [06:38:22](https://www.youtube.com/watch?v=XI-iP-qk_Vk&t=23902s) Thank you and next steps