{"id":19059936,"url":"https://github.com/q-m/scrapyd-k8s","last_synced_at":"2025-08-15T22:41:57.547Z","repository":{"id":203336458,"uuid":"709350274","full_name":"q-m/scrapyd-k8s","owner":"q-m","description":"Scrapyd on container infrastructure","archived":false,"fork":false,"pushed_at":"2025-04-11T10:02:33.000Z","size":103,"stargazers_count":14,"open_issues_count":16,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-04T01:41:50.219Z","etag":null,"topics":["docker","kubernetes","scrapy","scrapyd"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/q-m.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-24T14:39:21.000Z","updated_at":"2025-04-11T10:02:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"95b8ba17-5077-4bff-8de8-0dc038630d06","html_url":"https://github.com/q-m/scrapyd-k8s","commit_stats":null,"previous_names":["q-m/scrapyd-k8s"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/q-m/scrapyd-k8s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/q-m%2Fscrapyd-k8s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/q-m%2Fscrapyd-k8s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/q-m%2Fscrapyd-k8s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/q-m%2Fscrapyd-k8s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/q-m","download_url":"https://codeload.github.com/q-m/scrapyd-k8s/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/q-m%2Fscrapyd-k8s/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266026236,"owners_count":23866031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","kubernetes","scrapy","scrapyd"],"created_at":"2024-11-09T00:12:13.829Z","updated_at":"2025-07-19T22:04:52.419Z","avatar_url":"https://github.com/q-m.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scrapyd for Kubernetes\n\nScrapyd-k8s is an application for deploying and running Scrapy spiders as\neither Docker instances or Kubernetes jobs. Its intention is to be compatible\nwith [scrapyd](https://scrapyd.readthedocs.io/), but adapt to a container-based\nenvironment.\n\nThere are some important differences, though:\n\n* _Spiders are distributed as Docker images_, not as Python eggs. This allows\n  to bundle spiders with dependencies, with all its benefits (and downsides).\n\n* _Each spider is run as a Docker instance or Kubernetes job_, instead of a process.\n  This gives good visibility within an already running cluster.\n\n* _Projects are specified in the configuration file_, which means this can not\n  be modified at run-time. On the other hand, scrapyd-k8s can be restarted\n  without affecting any running spiders.\n\nAt this moment, each spider job is directly linked to a Docker instance or\nKubernetes job, and the daemon will retrieve its state by looking at those\njobs. This makes it easy to inspect and adjust the spider queue even outside\nscrapyd-k8s.\n\nBy default, there is no scheduling, and all jobs created will be started immediately.\nThis can be changed by [setting `max_proc`](CONFIG.md).\n\n## Running\n\nTypically this application will be run on using a (Docker or Kubernetes) container.\nYou will need to provide a configuration file, use one of the sample configuration\nfiles as a template ([`scrapyd_k8s.sample-k8s.conf`](./scrapyd_k8s.sample-k8s.conf)\nor [`scrapyd_k8s.sample-docker.conf`](./scrapyd_k8s.sample-docker.conf)).\n\nThe next section explains how to get this running Docker, Kubernetes or Local.\nThen read on for an example of how to use the API.\n\n### Docker\n\n```\ncp scrapyd_k8s.sample-docker.conf scrapyd_k8s.conf\ndocker build -t ghcr.io/q-m/scrapyd-k8s:latest .\ndocker run \\\n  --rm \\\n  -v ./scrapyd_k8s.conf:/opt/app/scrapyd_k8s.conf:ro \\\n  -v /var/run/docker.sock:/var/run/docker.sock \\\n  -v $HOME/.docker/config.json:/root/.docker/config.json:ro \\\n  -u 0 \\\n  -p 127.0.0.1:6800:6800 \\\n  ghcr.io/q-m/scrapyd-k8s:latest\n```\n\nYou'll be able to talk to localhost on port `6800`.\n\nMake sure to pull the spider image so it is known locally.\nIn case of the default example spider:\n\n```sh\ndocker pull ghcr.io/q-m/scrapyd-k8s-spider-example\n```\n\nNote that running like this in Docker is not really recommended for production,\nas it exposes the Docker socket and runs as root. It may be useful to try\nthings out.\n\n\n### Kubernetes\n\n1. Adapt the spider configuration in [`kubernetes.yaml`](./kubernetes.yaml) (`scrapyd_k8s.conf` in configmap)\n2. Create the resources: `kubectl create -f kubernetes.yaml`\n\nYou'll be able to talk to the `scrapyd-k8s` service on port `6800`.\nNote that some features require additional Kubernetes configuration (scheduling, joblogs).\n\n### Local\n\nFor development, or just a quick start, you can also run this application locally.\n\nRequirements:\n- [Python 3](https://www.python.org/)\n- [Skopeo](https://github.com/containers/skopeo) available in `PATH` (for remote repositories)\n- Either [Docker](https://www.docker.com/) or [Kubernetes](https://kubernetes.io/) setup and accessible\n  (scheduling will require Kubernetes 1.24+)\n\nThis will work with either Docker or Kubernetes (provided it is setup).\nFor example, for Docker:\n\n```sh\ncp scrapyd_k8s.sample-docker.conf scrapyd_k8s.conf\npython3 -m scrapyd_k8s\n```\n\nYou'll be able to talk to localhost on port `6800`.\n\nFor Docker, make sure to pull the spider image so it is known locally.\nIn case of the default example spider:\n\n```sh\ndocker pull ghcr.io/q-m/scrapyd-k8s-spider-example\n```\n\n\n## Accessing the API\n\nWith `scrapyd-k8s` running and setup, you can access it. Here we assume that\nit listens on `localhost:6800` (for Kubernetes, you would use\nthe service name `scrapyd-k8s:6800` instead).\n\n```sh\ncurl http://localhost:6800/daemonstatus.json\n```\n\n\u003e ```json\n\u003e {\"spiders\":0,\"status\":\"ok\"}\n\u003e ```\n\n```sh\ncurl http://localhost:6800/listprojects.json\n```\n\n\u003e ```json\n\u003e {\"projects\":[\"example\"],\"status\":\"ok\"}\n\u003e ```\n\n```sh\ncurl 'http://localhost:6800/listversions.json?project=example'\n```\n\n\u003e ```json\n\u003e {\"status\":\"ok\",\"versions\":[\"latest\"]}\n\u003e ```\n\n```sh\ncurl 'http://localhost:6800/listspiders.json?project=example\u0026_version=latest'\n```\n\n\u003e ```json\n\u003e {\"spiders\":[\"quotes\",\"static\"],\"status\":\"ok\"}\n\u003e ```\n\n```sh\ncurl -F project=example -F _version=latest -F spider=quotes http://localhost:6800/schedule.json\n```\n\n\u003e ```json\n\u003e {\"jobid\":\"e9b81fccbec211eeb3b109f30f136c01\",\"status\":\"ok\"}\n\u003e ```\n\n```sh\ncurl http://localhost:6800/listjobs.json\n```\n```json\n{\n  \"finished\":[],\n  \"pending\":[],\n  \"running\":[{\"id\":\"e9b81fccbec211eeb3b109f30f136c01\",\"project\":\"example\",\"spider\":\"quotes\",\"state\":\"running\", \"start_time\":\"2012-09-12 10:14:03.594664\", \"end_time\":null}],\n  \"status\":\"ok\"\n}\n```\n\nTo see what the spider has done, look at the container logs:\n\n```sh\ndocker ps -a\n```\n\n\u003e ```\n\u003e CONTAINER ID  IMAGE                                          COMMAND                CREATED   STATUS              NAMES\n\u003e 8c514a7ac917  ghcr.io/q-m/scrapyd-k8s-spider-example:latest  \"scrapy crawl quotes\"  42s ago   Exited (0) 30s ago  scrapyd_example_cb50c27cbec311eeb3b109f30f136c01\n\u003e ```\n\n```sh\ndocker logs 8c514a7ac917\n```\n\n\u003e ```\n\u003e [scrapy.utils.log] INFO: Scrapy 2.11.0 started (bot: example)\n\u003e ...\n\u003e [scrapy.core.scraper] DEBUG: Scraped from \u003c200 http://quotes.toscrape.com/\u003e\n\u003e {'text': 'The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.', 'author': 'Albert Einstein', 'tags': 'change'}\n\u003e ...\n\u003e [scrapy.core.engine] INFO: Spider closed (finished)\n\u003e ```\n\n\n## Spider as Docker image\n\n- Spiders are distributed as Docker images.\n- One can run `scrapy crawl \u003cspider\u003e` in the container to run a spider,\n  without any additional setup (so set `SCRAPY_SETTINGS_MODULE`).\n- Each Docker image has specific labels to indicate its project and spiders.\n  * `org.scrapy.project` - the project name\n  * `org.scrapy.spiders` - the spiders (those returned by `scrapy list`, comma-separated)\n\nAn example spider is available at [q-m/scrapyd-k8s-example-spider](https://github.com/q-m/scrapyd-k8s-spider-example),\nincluding a [Github Action](https://github.com/q-m/scrapyd-k8s-spider-example/blob/main/.github/workflows/container.yml) for building a container.\n\n\n## API\n\n### `daemonstatus.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#daemonstatus-json))\n\nLists scrapyd jobs by looking at Docker containers or Kubernetes jobs.\n\n### ~~`addversion.json`~~ ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#addversion-json))\n\nNot supported, by design.\nIf you want to add a version, add a Docker image to the repository.\n\n### `schedule.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#schedule-json))\n\nSchedules a new spider by creating a Docker container or Kubernetes job.\n\n### `cancel.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#cancel-json))\n\nRemoves a scheduled spider, kills it when running, does nothing when finished.\n\n### `listprojects.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#listprojects-json))\n\nLists projects from the configuration file.\n\n### `listversions.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#listversions-json))\n\nLists versions from the project's Docker repository.\n\n### `listspiders.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#listspiders-json))\n\nLists spiders from the spider image's `org.scrapy.spiders` label.\n\n### `listjobs.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#listjobs-json))\n\nLists current jobs by looking at Docker containers or Kubernetes jobs.\nNote that `end_time` is not yet supported for Docker.\n\n### ~~`delversion.json`~~ ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#delversion-json))\n\nNot supported, by design.\nIf you want to delete a version, remove the corresponding Docker image from the repository.\n\n### ~~`delproject.json`~~ ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#delproject-json))\n\nNot supported, by design.\nIf you want to delete a project, remove it from the configuration file.\n\n## Configuration\n\nThis is done in the file `scrapyd_k8s.conf`, the options are explained in the [Configuration Guide](CONFIG.md).\n\n## License\n\nThis software is distributed under the [MIT license](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fq-m%2Fscrapyd-k8s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fq-m%2Fscrapyd-k8s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fq-m%2Fscrapyd-k8s/lists"}