{"id":51601001,"url":"https://github.com/othonhugo/praw-cli","last_synced_at":"2026-07-11T22:01:48.027Z","repository":{"id":368716465,"uuid":"1278724935","full_name":"othonhugo/praw-cli","owner":"othonhugo","description":"Composable Reddit crawler CLI with lazy pipelines, powerful filtering, reproducible datasets, and flexible output formats.","archived":false,"fork":false,"pushed_at":"2026-07-09T19:30:12.000Z","size":244,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-09T21:06:58.846Z","etag":null,"topics":["cli","command-line","crawler","data-collection","data-pipeline","data-science","data-scraping","etl","natural-language-processing","nlp","praw","praw-reddit","python","reddit","reddit-api","research-tools","scraper","text-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/othonhugo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-24T03:44:01.000Z","updated_at":"2026-07-09T19:29:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/othonhugo/praw-cli","commit_stats":null,"previous_names":["othonhugo/prawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/othonhugo/praw-cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/othonhugo%2Fpraw-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/othonhugo%2Fpraw-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/othonhugo%2Fpraw-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/othonhugo%2Fpraw-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/othonhugo","download_url":"https://codeload.github.com/othonhugo/praw-cli/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/othonhugo%2Fpraw-cli/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35376136,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","command-line","crawler","data-collection","data-pipeline","data-science","data-scraping","etl","natural-language-processing","nlp","praw","praw-reddit","python","reddit","reddit-api","research-tools","scraper","text-analysis"],"created_at":"2026-07-11T22:01:46.361Z","updated_at":"2026-07-11T22:01:48.018Z","avatar_url":"https://github.com/othonhugo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# praw-cli\n\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-purple.svg)](https://github.com/astral-sh/ruff)\n\nA composable Reddit crawler for the command line. Fetch posts and comments from any source, shape the data through a lazy filter pipeline, and emit it in any format — all without writing a single line of Python.\n\n```bash\n# stream the top 500 posts from r/MachineLearning as JSONL\npraw-cli posts r/MachineLearning --sort top --time year --limit 500 --format jsonl\n\n# search across r/programming, keep only high-signal posts, project to four fields\npraw-cli search \"New web framework\" --sub r/programming \\\n  --filter \"score\u003e=100\" --filter \"num_comments\u003e=10\" \\\n  --fields id,title,score,url --format csv --output results.csv\n\n# full comment tree of a submission, depth-first, minimum score 5\npraw-cli comments https://reddit.com/r/Python/comments/xyz/ \\\n  --depth 10 --min-score 5 --format jsonl\n```\n\n## Key Features \u0026 How They Solve Your Problems\n\n### Smart Memory \u0026 Reliability\n\n- **Constant Memory Footprint (Lazy Pipelines)**: Typical scrapers buffer huge arrays in memory, causing out-of-memory crashes on large crawls. praw-cli processes data lazily item-by-item (`Iterator[Record]`). Streaming 100,000 posts uses the same constant memory as streaming 10.\n- **Resumable Extractions (Checkpointing)**: Long-running scrapes often fail halfway due to network drops or API limits. praw-cli checkpoints progress, letting you `--resume` interrupted sessions without refetching from scratch.\n\n### Zero-Code Data Preparation\n\n- **Data Science-Focused DSL**: Stop writing custom Python scripts just to filter text. Chain `--filter` conditions directly in the CLI:\n  - _NLP Cleaning_: Keep long-form content using `selftext len\u003e= 500`.\n  - _Temporal Sorting_: Restrict dates natively using ISO-8601 strings, like `created_utc \u003e= 2024-01-01`.\n  - _Targeted Mining_: Target specific topics with keyword groupings (`has`, `has_all`) or regular expressions (`title ~= \\bbot\\b`).\n- **Field Projections**: Keep output datasets lightweight and clean by extracting only the schema columns you need (e.g., `--fields id,title,score,author`).\n\n### Instant Pandas \u0026 R Integrations\n\n- **Diverse Output Formats**: Stream outputs directly to `jsonl` (preferred for streaming/Pandas), standard `json`, `csv` (for R/Excel), or rich console tables and Markdown reports.\n- **Multi-Sink Pipeline**: Write the raw data to a `.jsonl` database while simultaneously writing a preview to a `.csv` summary in a single pass.\n\n### Built-in Scientific Rigor\n\n- **Crawl Manifests**: Every execution automatically generates a `.manifest.json` detailing exact parameters, versioning, records filtered, and a config fingerprint, preventing configuration drift in research environments.\n- **Deterministic Sampling**: Extract reproducible subsets of huge subreddits using systematic or randomized sampling (e.g., Bernoulli trial at `rate = 0.1` with a fixed seed).\n\n## Installation\n\nRequires Python 3.12 or later and a [Reddit API application](https://www.reddit.com/prefs/apps) (free, read-only access is sufficient for most use cases).\n\n### Via PyPI (Recommended)\n\nInstall the package directly using `pip` or `pipx`:\n\n```bash\npip install praw-cli\n```\n\n### From Source (Using `uv`)\n\nIf you want to run it locally or contribute to development:\n\n```bash\ngit clone https://github.com/othonhugo/praw-cli\ncd praw-cli\nuv sync\nsource .venv/bin/activate\n```\n\n## Documentation\n\nFull documentation is available in the `docs/` directory.\n\n- **[Getting Started](docs/getting-started.md)**: Installation and Authentication\n- **Usage Guide**:\n  - [Commands](docs/usage/commands.md)\n  - [Pipelines \u0026 Filters](docs/usage/pipelines-filters.md)\n  - [Formatters \u0026 Outputs](docs/usage/outputs.md)\n  - [Scientific Use \u0026 Reproducibility](docs/usage/reproducibility.md)\n- **Development**:\n  - [System Architecture](docs/development/architecture.md)\n  - [Decision Log (ADR)](docs/development/decisions.md)\n  - [Contributing Guide](docs/development/contributing.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fothonhugo%2Fpraw-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fothonhugo%2Fpraw-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fothonhugo%2Fpraw-cli/lists"}