{"id":13414792,"url":"https://github.com/Gerapy/Gerapy","last_synced_at":"2025-03-14T22:32:12.511Z","repository":{"id":22291423,"uuid":"95846805","full_name":"Gerapy/Gerapy","owner":"Gerapy","description":"Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js","archived":false,"fork":false,"pushed_at":"2024-10-29T22:08:29.000Z","size":38367,"stargazers_count":3350,"open_issues_count":71,"forks_count":644,"subscribers_count":125,"default_branch":"master","last_synced_at":"2024-11-08T15:53:42.124Z","etag":null,"topics":["dashboard","distributed","django","docker","gerapy","scrapy","scrapyd","spider","vue","vuejs","webspider"],"latest_commit_sha":null,"homepage":"https://docs.gerapy.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Gerapy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-30T03:58:27.000Z","updated_at":"2024-11-07T10:48:27.000Z","dependencies_parsed_at":"2023-01-14T08:00:41.642Z","dependency_job_id":"e63f6330-b00e-442f-a992-ecdcfb1888bb","html_url":"https://github.com/Gerapy/Gerapy","commit_stats":{"total_commits":547,"total_committers":16,"mean_commits":34.1875,"dds":"0.34552102376599636","last_synced_commit":"a575dcc9a30b41535a44e35a311348c2d89216b8"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gerapy%2FGerapy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gerapy%2FGerapy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gerapy%2FGerapy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gerapy%2FGerapy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Gerapy","download_url":"https://codeload.github.com/Gerapy/Gerapy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243658055,"owners_count":20326459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dashboard","distributed","django","docker","gerapy","scrapy","scrapyd","spider","vue","vuejs","webspider"],"created_at":"2024-07-30T21:00:36.866Z","updated_at":"2025-03-14T22:32:12.503Z","avatar_url":"https://github.com/Gerapy.png","language":"Python","readme":"# Gerapy\n\n![Build](https://github.com/Gerapy/Gerapy/workflows/build/badge.svg)\n![Read the Docs](https://img.shields.io/readthedocs/gerapy)\n![PyPI - Python Version](https://img.shields.io/badge/python-3.6%2B-blue)\n[![GitHub stars](https://img.shields.io/github/stars/Gerapy/Gerapy)](https://github.com/Gerapy/Gerapy/stargazers)\n![PyPI - Downloads](https://img.shields.io/pypi/dm/gerapy)\n![Docker Pulls](https://img.shields.io/docker/pulls/germey/gerapy)\n![PyPI - License](https://img.shields.io/pypi/l/gerapy)\n\nDistributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js.\n\n## Documentation\n\nDocumentation is available online at [https://docs.gerapy.com/](https://docs.gerapy.com/) and [https://github.com/Gerapy/Docs](https://github.com/Gerapy/Docs).\n\n## Support\n\nGerapy is developed based on Python 3.x. Python 2.x may be supported later.\n\n## Usage\n\nInstall Gerapy by pip:\n\n```bash\npip3 install gerapy\n```\n\nAfter the installation, you need to do these things below to run Gerapy server:\n\nIf you have installed Gerapy successfully, you can use command `gerapy`. If not, check the installation.\n\nFirst use this command to initialize the workspace:\n\n```bash\ngerapy init\n```\n\nNow you will get a folder named `gerapy`. Also you can specify the name of your workspace by this command:\n\n```\ngerapy init \u003cworkspace\u003e\n```\n\nThen `cd` to this folder, and run this command to initialize the Database:\n\n```bash\ncd gerapy\ngerapy migrate\n```\n\nNext you need to create a superuser by this command:\n\n```\ngerapy createsuperuser\n```\n\nThen you can runserver by this command:\n\n```bash\ngerapy runserver\n```\n\nThen you can visit [http://localhost:8000](http://localhost:8000) to enjoy it. Also you can vist [http://localhost:8000/admin](http://localhost:8000/admin) to get the admin management backend.\n\nIf you want to run Gerapy in public, just run like this:\n\n```\ngerapy runserver 0.0.0.0:8000\n```\n\nThen it will run with public host and port 8000.\n\nIn Gerapy, You can create a configurable project and then configure and generate code of Scrapy automatically. But this module is unstable, we're trying to refine it.\n\nAlso you can drag your Scrapy Project to `projects` folder. Then refresh web, it will appear in the Project Index Page and comes to un-configurable, but you can edit this project through the web page.\n\nAs for deployment, you can move to Deploy Page. Firstly you need to build your project and add client in the Client Index Page, then you can deploy the project just by clicking button.\n\nAfter the deployment, you can manage the job in Monitor Page.\n\n## Docker\n\nJust run this command:\n\n```\ndocker-compose up\n```\n\nThen it will run at port 8000. You can use the temp admin account (username: admin, password: admin) to login. And please change the password later for safety.\n\nCommand Usage:\n\n```\ndocker run -d -v \u003cworkspace\u003e:/home/gerapy -p \u003cpublic_port\u003e:\u003ccontainer_port\u003e germey/gerapy\n```\n\nPlease specify your workspace to mount Gerapy workspace by `-v \u003cworkspace\u003e:/app/gerapy` and specify server port by `-p \u003cpublic_port\u003e:\u003ccontainer_port\u003e`.\n\nIf you run Gerapy by Docker, you can visit Gerapy website such as [http://localhost:8000](http://localhost:8000) and enjoy it, no need to do other initialzation things.\n\n## TodoList\n\n- [x] Add Visual Configuration of Spider with Previewing Website\n- [x] Add Scrapyd Auth Management\n- [x] Add Gerapy Auth Management\n- [x] Add Timed Task Scheduler\n- [ ] Add Visual Configuration of Scrapy\n- [ ] Add Intelligent Analysis of Web Page\n\n## Communication\n\nIf you have any questions or ideas, you can send [Issues](https://github.com/Gerapy/Gerapy/issues) or [Pull Requests](https://github.com/Gerapy/Gerapy/pulls), your suggestions are really import for us, thanks for your contirbution.\n","funding_links":[],"categories":["Python","Apps","Python (1887)","Web Scraping \u0026 Crawling","Scrapy Distributed","🕸️ Web Scraping \u0026 Crawling"],"sub_categories":["Front-End Scrapy Managers","Tools"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGerapy%2FGerapy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGerapy%2FGerapy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGerapy%2FGerapy/lists"}