{"id":13792396,"url":"https://github.com/orangain/scrapy-s3pipeline","last_synced_at":"2026-01-14T07:48:21.315Z","repository":{"id":27210339,"uuid":"111988794","full_name":"orangain/scrapy-s3pipeline","owner":"orangain","description":"Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.","archived":false,"fork":false,"pushed_at":"2022-03-18T17:30:37.000Z","size":55,"stargazers_count":76,"open_issues_count":0,"forks_count":12,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-30T02:08:59.423Z","etag":null,"topics":["aws","pipeline","s3","scrapy"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/scrapy-s3pipeline/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orangain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-25T08:33:05.000Z","updated_at":"2025-08-27T05:39:16.000Z","dependencies_parsed_at":"2022-08-07T12:15:39.105Z","dependency_job_id":null,"html_url":"https://github.com/orangain/scrapy-s3pipeline","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/orangain/scrapy-s3pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orangain%2Fscrapy-s3pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orangain%2Fscrapy-s3pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orangain%2Fscrapy-s3pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orangain%2Fscrapy-s3pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orangain","download_url":"https://codeload.github.com/orangain/scrapy-s3pipeline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orangain%2Fscrapy-s3pipeline/sbom","scorecard":{"id":711470,"data":{"date":"2025-08-11","repo":{"name":"github.com/orangain/scrapy-s3pipeline","commit":"1a11de83c51c3863b471d806583fa5f3a5a6464a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.5,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":1,"reason":"Found 1/7 approved changesets -- score normalized to 1","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yaml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/orangain/scrapy-s3pipeline/ci.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yaml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/orangain/scrapy-s3pipeline/ci.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yaml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/orangain/scrapy-s3pipeline/ci.yaml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/ci.yaml:26","Warn: pipCommand not pinned by hash: .github/workflows/ci.yaml:27","Warn: pipCommand not pinned by hash: .github/workflows/ci.yaml:28","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   3 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yaml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 27 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T08:18:37.095Z","repository_id":27210339,"created_at":"2025-08-22T08:18:37.095Z","updated_at":"2025-08-22T08:18:37.095Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","pipeline","s3","scrapy"],"created_at":"2024-08-03T22:01:11.786Z","updated_at":"2026-01-14T07:48:21.295Z","avatar_url":"https://github.com/orangain.png","language":"Python","funding_links":[],"categories":["Apps"],"sub_categories":["Data Processing"],"readme":"# Scrapy S3 Pipeline\n\n[![PyPI version](https://badge.fury.io/py/scrapy-s3pipeline.svg)](https://badge.fury.io/py/scrapy-s3pipeline) ![CI](https://github.com/orangain/scrapy-s3pipeline/workflows/CI/badge.svg)\n\nScrapy pipeline to store items into [Amazon S3](https://aws.amazon.com/s3/) or [Google Cloud Storage (GCS)](https://cloud.google.com/storage) bucket. Unlike built-in [FeedExporter](https://docs.scrapy.org/en/latest/topics/feed-exports.html#s3), the pipeline has the following features:\n\n* The pipeline upload items to S3/GCS by chunk while crawler is running.\n  * From Scrapy 2.3, built-in [FEED_EXPORT_BATCH_ITEM_COUNT](https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEED_EXPORT_BATCH_ITEM_COUNT) does almost the same thing.\n* Support GZip compression.\n\nThe pipeline aims to run crawler and scraper in different processes, e.g. run crawler process with Scrapy in AWS Fargate and run scraper process with lxml in AWS Lambda.\n\n## Requirements\n\n* Python 3.6+ (Tested in 3.9)\n* Scrapy 1.1+ (Tested in 2.4)\n* boto3 or google-cloud-storage\n\n## Install\n\n**For S3 users:**\n\n```shell-session\n$ pip3 install scrapy-s3pipeline[s3]\n```\n\n**For GCS users:**\n\n```shell-session\n$ pip3 install scrapy-s3pipeline[gcs]\n```\n\n## Getting started\n\n1. Install Scrapy S3 Pipeline with pip.\n\n    ```shell-session\n    $ pip3 install scrapy-s3pipeline[s3]\n    ```\n\n    or\n\n    ```shell-session\n    $ pip3 install scrapy-s3pipeline[gcs]\n    ```\n\n2.  Add `'s3pipeline.S3Pipeline'` to `ITEM_PIPELINES` setting in your Scrapy project.\n\n    ```py\n    ITEM_PIPELINES = {\n        's3pipeline.S3Pipeline': 100,  # Add this line.\n    }\n    ```\n\n3. Add `S3PIPELINE_URL` setting. You need to change `my-bucket` to your bucket name.\n\n    ```py\n    # For S3 users\n    S3PIPELINE_URL = 's3://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gz'\n\n    # For GCS users\n    S3PIPELINE_URL = 'gs://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gz'\n    GCS_PROJECT_ID = 'my-project' # Change to your project id\n    ```\n\n4. Setup AWS/GCP credentials.\n\n    **For S3 users:**\n\n    Setup AWS credentials via `aws configure` command or [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Alternatively, use Scrapy's settings [AWS_ACCESS_KEY_ID](https://docs.scrapy.org/en/latest/topics/settings.html#aws-access-key-id) and [AWS_SECRET_ACCESS_KEY](https://docs.scrapy.org/en/latest/topics/settings.html#aws-secret-access-key).\n\n    **For GCS users:**\n\n    Setup GCP credentials via `gcloud auth application-default login` command or environment variable [GOOGLE_APPLICATION_CREDENTIALS](https://cloud.google.com/docs/authentication/getting-started). Alternatively, you can set json string of service account's key file to `GOOGLE_APPLICATION_CREDENTIALS_JSON` settings.\n\n5. Run your spider. You will see items in your bucket after 100 items are crawled or the spider is closed.\n\n## Settings\n\n### S3PIPELINE_URL (Required)\n\nS3/GCS Bucket URL to store items.\n\ne.g.:\n\n* S3: `s3://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gz`\n* GCS: `gs://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gz`\n\nThe following replacement fields are supported in `S3PIPELINE_URL`.\n\n* `{chunk}` - gets replaced by a start index of items in current chunk, e.g. '0', '100', '200',....\n* `{time}` - gets replaced by a timestamp when the spider is started.\n\nYou can also use other spider fields, e.g. `{name}`. You can use [format string syntax](https://docs.python.org/3/library/string.html#formatstrings) here, e.g. `{chunk:07d}`.\n\nFile format is determined by a file extension in the URL. For example, if `S3PIPELINE_URL` ends with `.json` or `.json.gz`, JSON format is used. See Scrapy's built-in [FEED_EXPORTERS](https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEED_EXPORTERS) settings for supported formats. If the file extension is not available in `FEED_EXPORTERS`, JSONLines format is used by default.\n\n### S3PIPELINE_MAX_CHUNK_SIZE (Optional)\n\nDefault: `100`\n\nMax count of items in a single chunk.\n\n### S3PIPELINE_MAX_WAIT_UPLOAD_TIME (Optional)\n\nDefault: `30.0`\n\nWhen no new item is processed in more than `S3PIPELINE_MAX_WAIT_UPLOAD_TIME` seconds, it will be forced to upload a chunk.\n\n### S3PIPELINE_GZIP (Optional)\n\nDefault: `True` if `S3PIPELINE_URL` ends with `.gz`; otherwise `False`.\n\nIf `True`, uploaded files will be compressed with Gzip.\n\n## Page item\n\nFor convinience, Scrapy S3 Pipeline provides `s3pipeline.Page` item class to store entire HTTP body. It has `url`, `body` and `crawled_at` fields.\n\nThis make it easy to store entire HTTP body and run scraper in other process. It's friendly to server-less architecture which run scraper in AWS Lambda.\n\nExample usage of Page:\n\n```py\nfrom datetime import datetime, timezone\n\nimport scrapy\nfrom s3pipeline import Page\n\n# ...\n\nclass YourSpider(scrapy.Spider):\n\n    # ...\n\n    def parse(self, response):\n        # You can create Page instance just one line.\n        yield Page.from_response(response)\n\n        # Or, you can fill item fields manually.\n        item = Page()\n        item['url'] = response.url\n        item['body'] = response.text\n        item['crawled_at'] = datetime.now(timezone.utc).replace(microsecond=0).isoformat()\n        yield item\n```\n\nNote: Page's body is omitted when printed to logs to improve readbility of logs.\n\n## Development\n\n### Test\n\n```\n$ python3 setup.py test\n```\n\n### Release\n\n```\n$ pip install twine wheel\n```\n\n```\n$ python3 setup.py bdist_wheel sdist\n$ twine upload dist/*\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forangain%2Fscrapy-s3pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forangain%2Fscrapy-s3pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forangain%2Fscrapy-s3pipeline/lists"}