{"id":24159419,"url":"https://github.com/pakagronglb/reddit-scraper","last_synced_at":"2025-10-20T04:13:47.555Z","repository":{"id":295197953,"uuid":"947589211","full_name":"pakagronglb/reddit-scraper","owner":"pakagronglb","description":"A powerful Reddit data scraping tool with a user-friendly Streamlit interface. Extract posts and comments from subreddits or specific posts with ease.","archived":false,"fork":false,"pushed_at":"2025-03-13T03:42:43.000Z","size":22,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-24T07:08:57.410Z","etag":null,"topics":["python","reddit","reddit-api","web-scraping"],"latest_commit_sha":null,"homepage":"https://reddit-scraper-123.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pakagronglb.png","metadata":{},"created_at":"2025-03-12T23:44:04.000Z","updated_at":"2025-04-18T09:00:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"24ea6341-870c-4e53-b40b-c9009550725c","html_url":"https://github.com/pakagronglb/reddit-scraper","commit_stats":null,"previous_names":["pakagronglb/reddit-scraper"],"tags_count":null,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pakagronglb%2Freddit-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pakagronglb%2Freddit-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pakagronglb%2Freddit-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pakagronglb%2Freddit-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pakagronglb","download_url":"https://codeload.github.com/pakagronglb/reddit-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pakagronglb%2Freddit-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259273954,"owners_count":22832437,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","reddit","reddit-api","web-scraping"],"created_at":"2025-01-12T15:17:22.345Z","updated_at":"2025-10-20T04:13:42.532Z","avatar_url":"https://github.com/pakagronglb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reddit Data Scraper 📊\n\n![Screenshot 2024-12-10 162954](https://github.com/user-attachments/assets/65a9c7be-359b-48aa-9104-55b43fe0d351)\n\n[![Python](https://img.shields.io/badge/Python-3.9+-blue.svg)](https://www.python.org)\n[![Streamlit](https://img.shields.io/badge/Streamlit-1.29.0-FF4B4B.svg)](https://streamlit.io)\n[![PRAW](https://img.shields.io/badge/PRAW-7.7.1-orange.svg)](https://praw.readthedocs.io)\n[![Pandas](https://img.shields.io/badge/Pandas-2.1.1-150458.svg)](https://pandas.pydata.org)\n\nA powerful Reddit data scraping tool with a user-friendly Streamlit interface. Extract posts and comments from subreddits or specific posts with ease.\n\n## 🚀 Features\n\n- 📱 User-friendly web interface\n- 🔍 Scrape posts from any subreddit\n- 💬 Extract comments from specific posts\n- 📊 Export data to CSV\n- ⏱️ Time-based filtering\n- 🔄 Caching for better performance\n\n## 🛠️ Tech Stack\n\n- **Python** - Core programming language\n- **Streamlit** - Web interface framework\n- **PRAW** - Reddit API wrapper\n- **Pandas** - Data manipulation and analysis\n- **python-dotenv** - Environment variable management\n\n## 📋 Prerequisites\n\n- Python 3.9 or higher\n- Reddit API credentials ([Get them here](https://www.reddit.com/prefs/apps))\n\n## ⚙️ Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/pakagronglb/reddit-scraper.git\ncd reddit-scraper\n```\n\n2. Create a virtual environment:\n```bash\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n```\n\n3. Install dependencies:\n```bash\npip install -r requirements.txt\n```\n\n4. Set up environment variables:\nCreate a `.env` file in the project root:\n```env\nREDDIT_CLIENT_ID=your_client_id\nREDDIT_CLIENT_SECRET=your_client_secret\nREDDIT_USER_AGENT=your_user_agent\n```\n\n## 🚀 Usage\n\n1. Start the application:\n```bash\nstreamlit run main.py\n```\n\n2. Access the web interface at `http://localhost:8501`\n\n3. Choose your scraping option:\n   - **Subreddit Posts**: Enter subreddit name, post limit, and time filter\n   - **Specific Post**: Enter the Reddit post URL\n\n4. Click \"Scrape\" and download the results as CSV\n\n## 🌐 Deployment\n\n### Streamlit Cloud\n\n1. Push your code to GitHub\n2. Visit [share.streamlit.io](https://share.streamlit.io)\n3. Connect your repository\n4. Add your Reddit API credentials in Streamlit secrets\n\n### Heroku\n\n1. Create a Heroku app:\n```bash\nheroku create your-app-name\n```\n\n2. Set environment variables:\n```bash\nheroku config:set REDDIT_CLIENT_ID=your_client_id\nheroku config:set REDDIT_CLIENT_SECRET=your_client_secret\nheroku config:set REDDIT_USER_AGENT=your_user_agent\n```\n\n3. Deploy:\n```bash\ngit push heroku main\n```\n\n## 📝 Configuration\n\n- `requirements.txt` - Project dependencies\n- `.env` - Local environment variables\n- `Procfile` - Heroku deployment configuration\n- `runtime.txt` - Python runtime specification\n\n## 🔒 Security\n\n- Never commit your `.env` file or `.streamlit/secrets.toml`\n- Use environment variables for sensitive data\n- Keep your Reddit API credentials secure\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Commit your changes\n4. Push to the branch\n5. Open a Pull Request\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 👏 Acknowledgments\n\n- [PRAW Documentation](https://praw.readthedocs.io/)\n- [Streamlit Documentation](https://docs.streamlit.io/)\n- [Reddit API Documentation](https://www.reddit.com/dev/api/)\n\n## 📧 Contact\n\nYour Name - [@pakagronglb](https://twitter.com/pakagronglb)\n\nProject Link: [https://github.com/pakagronglb/reddit-scraper](https://github.com/pakagronglb/reddit-scraper)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpakagronglb%2Freddit-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpakagronglb%2Freddit-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpakagronglb%2Freddit-scraper/lists"}