{"id":20490053,"url":"https://github.com/vitorfs/woid","last_synced_at":"2025-04-09T16:10:33.144Z","repository":{"id":49689851,"uuid":"40613706","full_name":"vitorfs/woid","owner":"vitorfs","description":"Simple news aggregator displaying top stories in real time","archived":false,"fork":false,"pushed_at":"2021-06-10T21:31:11.000Z","size":578,"stargazers_count":247,"open_issues_count":5,"forks_count":119,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-02T10:09:49.530Z","etag":null,"topics":["crawler","django","news"],"latest_commit_sha":null,"homepage":"https://woid.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vitorfs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-12T17:10:40.000Z","updated_at":"2025-02-03T01:17:02.000Z","dependencies_parsed_at":"2022-09-14T12:41:56.755Z","dependency_job_id":null,"html_url":"https://github.com/vitorfs/woid","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorfs%2Fwoid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorfs%2Fwoid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorfs%2Fwoid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitorfs%2Fwoid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vitorfs","download_url":"https://codeload.github.com/vitorfs/woid/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248065283,"owners_count":21041871,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","django","news"],"created_at":"2024-11-15T17:15:23.182Z","updated_at":"2025-04-09T16:10:33.127Z","avatar_url":"https://github.com/vitorfs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Woid\n\n[![Python Version](https://img.shields.io/badge/python-3.6-brightgreen.svg)](https://python.org)\n[![Django Version](https://img.shields.io/badge/django-2.1-brightgreen.svg)](https://djangoproject.com)\n\n**Table of Contents**\n\n* [Running Locally](#running-locally)\n* [Supported Services](#supported-services)\n* [Crawlers](#crawlers)\n  * [API Key Required: The New York Times](#the-new-york-times)\n  * [API Key Required: Product Hunt](#product-hunt)\n  * [Cron Jobs](#cron-jobs)\n* [License](#license)\n\n## Running Locally\n\nFirst, clone the repository to your local machine:\n\n```\ngit clone https://github.com/vitorfs/woid.git\n```\n\nInstall the requirements:\n\n```bash\npip install -r requirements/dev.txt\n```\n\nApply the migrations:\n\n```bash\npython manage.py migrate\n```\n\nLoad the initial data:\n\n```bash\npython manage.py loaddata services.json\n```\n\nFinally, run the development server:\n\n```bash\npython manage.py runserver\n```\n\nThe site will be available at **127.0.0.1:8000**.\n\n## Supported Services\n\nCurrently Woid crawl the following services to collect top stories: \n\n* Hacker News `hn`\n* Reddit `reddit`\n* GitHub `github`\n* The New York Times `nytimes`\n* Product Hunt `producthunt`\n\n\n## Crawlers\n\nYou can run the crawlers manually to collect the top stories using the following command:\n\n```bash\npython manage.py crawl reddit\n```\n\nYou can pass multiple services at once:\n\n```bash\npython manage.py crawl reddit hn nytimes\n```\n\nValid values: `hn`, `reddit`, `github`, `nytimes`, `producthunt`.\n\n### The New York Times\n\nTo crawl The New York Times you will need an API key.\n\nYou can register one application at [developer.nytimes.com](https://developer.nytimes.com).\n\n### Product Hunt\n\nProduct Hunt require an API key to consume their API. \n\nYou can register one application at [api.producthunt.com/v1/docs](https://api.producthunt.com/v1/docs)\n\n### Cron Jobs\n\nYou can set up cron jobs to execute the crawlers periodically. Here is what my crontab looks like:\n\n```\n*/5 * * * * /home/woid/venv/bin/python /home/woid/woid/manage.py crawl reddit hn producthunt \u003e\u003e /home/woid/logs/cron.log 2\u003e\u00261\n*/30 * * * * /home/woid/venv/bin/python /home/woid/woid/manage.py crawl nytimes github \u003e\u003e /home/woid/logs/cron.log 2\u003e\u00261\n```\n\n## License\n\nThe source code is released under the [Apache 2.0 license](https://github.com/vitorfs/woid/blob/master/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitorfs%2Fwoid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitorfs%2Fwoid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitorfs%2Fwoid/lists"}