{"id":15336004,"url":"https://github.com/kinow/nihongo-scraper","last_synced_at":"2026-01-06T09:36:44.357Z","repository":{"id":137722145,"uuid":"97596692","full_name":"kinow/nihongo-scraper","owner":"kinow","description":"Scraper to download Japanese news, quizzes, and other resources for use offline. Data is used for personal study only, and NLP is applied to isolate Kanji for reading cards, for example.","archived":false,"fork":false,"pushed_at":"2017-07-18T12:52:28.000Z","size":2,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-01T21:35:59.914Z","etag":null,"topics":["education","japanese","language","nihongo","nlp","python","scraper","scrapy"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kinow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-18T12:42:26.000Z","updated_at":"2019-04-03T10:44:42.000Z","dependencies_parsed_at":"2023-07-16T07:49:39.713Z","dependency_job_id":null,"html_url":"https://github.com/kinow/nihongo-scraper","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"92b981b5987f8eaca9fb45d7d28374a58fb3d182"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fnihongo-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fnihongo-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fnihongo-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinow%2Fnihongo-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kinow","download_url":"https://codeload.github.com/kinow/nihongo-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245901138,"owners_count":20690976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["education","japanese","language","nihongo","nlp","python","scraper","scrapy"],"created_at":"2024-10-01T10:14:41.584Z","updated_at":"2026-01-06T09:36:44.307Z","avatar_url":"https://github.com/kinow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nihongo Scraper\n\nScraper to download Japanese news, quizzes, and other resources for use offline.\nData is used for personal study only, and NLP is applied to isolate Kanji for\nreading cards, for example.\n\n* nihongo-spider simply scrapes a known site with quizzes and saves the response as JSON/CSV\n\nURL's used are hidden, to prevent a mass of requests to all the sites, or bots following\nlinks from GitHub.\n\n## Build\n\n```\ngit clone https://github.com/kinow/nihongo-scraper.git\ncd nihongo-scraper\npip install -r requirements\n```\n\n## Execute nihongo-spider\n\n```\ncat \u003e .env \u003c\u003cEOF\nSTART_URL=http://\u003curl\u003e/context/path/\nEOF\nscrapy runspider nihongo-spider.py -o questions.json\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinow%2Fnihongo-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkinow%2Fnihongo-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinow%2Fnihongo-scraper/lists"}