{"id":51344131,"url":"https://github.com/brightdata/bright-data-scraper-studio-python-project","last_synced_at":"2026-07-02T10:03:38.622Z","repository":{"id":359589954,"uuid":"1246712586","full_name":"brightdata/bright-data-scraper-studio-python-project","owner":"brightdata","description":"Bright Data Scraper Studio Python boilerplate code","archived":false,"fork":false,"pushed_at":"2026-05-22T15:19:58.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-22T19:42:58.342Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brightdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-22T13:26:55.000Z","updated_at":"2026-05-22T15:20:02.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/brightdata/bright-data-scraper-studio-python-project","commit_stats":null,"previous_names":["brightdata/bright-data-scraper-studio-python-project"],"tags_count":null,"template":true,"template_full_name":null,"purl":"pkg:github/brightdata/bright-data-scraper-studio-python-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightdata%2Fbright-data-scraper-studio-python-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightdata%2Fbright-data-scraper-studio-python-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightdata%2Fbright-data-scraper-studio-python-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightdata%2Fbright-data-scraper-studio-python-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brightdata","download_url":"https://codeload.github.com/brightdata/bright-data-scraper-studio-python-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightdata%2Fbright-data-scraper-studio-python-project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35042006,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T10:03:37.796Z","updated_at":"2026-07-02T10:03:38.613Z","avatar_url":"https://github.com/brightdata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bright Data Scraper Studio (Python)\n\nA minimal Python starter for running a [Bright Data Scraper Studio](https://brightdata.com/products/web-scraper/scraper-studio) collector via the Data Collection API: trigger a job with a list of URLs and download the results.\n\n[![Bright Data Promo](https://github.com/luminati-io/LinkedIn-Scraper/raw/main/Proxies%20and%20scrapers%20GitHub%20bonus%20banner.png)](https://brightdata.com/)\n\n\u003ca href=\"https://githubbox.com/brightdata/bright-data-scraper-studio-python-project?file=index.py\" target=\"_blank\"\u003eOpen in CodeSandbox\u003c/a\u003e, sign in with GitHub, then fork the repository to begin making changes.\n\n---\n\n## Table of contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Configuration](#configuration)\n- [How it works](#how-it-works)\n- [Examples](#examples)\n- [Output](#output)\n- [Security](#security)\n- [Support](#support)\n- [License](#license)\n\n---\n\n## Overview\n\n[Bright Data Scraper Studio](https://brightdata.com/products/web-scraper/scraper-studio) is a low-code IDE for building custom web scraping collectors on the Bright Data platform. Once a collector is published it exposes two HTTP endpoints:\n\n| Step | Endpoint | Purpose |\n| --- | --- | --- |\n| 1 | `POST /dca/trigger?collector=\u003cid\u003e` | Queue one or more inputs for the collector |\n| 2 | `GET  /dca/dataset?id=\u003csnapshot_id\u003e` | Download the collected data when ready |\n\nThis repository wraps those two calls in about 150 lines of Python so you can copy, paste and ship.\n\n---\n\n## Features\n\n- Trigger a Scraper Studio collector via the `/dca/trigger` endpoint\n- Poll `/dca/dataset` until results are ready\n- Env-var config via `.env` (no secrets in code)\n- Retry with exponential backoff for transient errors (5xx and network); fails fast on 4xx\n- Library helpers: `trigger_with_url`, `trigger_with_urls`, `run_scraper`\n- Saves the raw JSON response to a timestamped file\n\n---\n\n## Prerequisites\n\n- Python 3.8 or higher\n- A Bright Data account with an [API token](https://brightdata.com/cp/setting)\n- A published collector in [Scraper Studio](https://brightdata.com/cp/scrapers); copy its **Collector ID** (starts with `c_`)\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/brightdata/bright-data-scraper-studio-python-project.git\ncd bright-data-scraper-studio-python-project\npip install -r requirements.txt\ncp .env.example .env       # then edit .env with your token and collector ID\n```\n\n### Dependencies\n\n- `requests`: HTTP client for the Bright Data API\n- `colorama`: colored terminal output\n- `python-dotenv`: load `.env` files into `os.environ`\n\n---\n\n## Usage\n\n```bash\npython index.py\n```\n\nResults are written to a `scraper_studio_results_\u003ctimestamp\u003e.json` file in the project directory.\n\n---\n\n## Configuration\n\nTwo environment variables are required. Set them in `.env`, in your shell, or hardcode them in [`index.py`](index.py):\n\n| Variable | Where to find it |\n| --- | --- |\n| `BRIGHT_DATA_API_TOKEN`    | Bright Data dashboard, [Account Settings \u0026rarr; API Tokens](https://brightdata.com/cp/setting) |\n| `BRIGHT_DATA_COLLECTOR_ID` | Scraper Studio: open your collector, copy the ID from the URL (starts with `c_`) |\n\nYou can also tune the polling and retry behavior at the top of `index.py`:\n\n```python\nPOLL_INTERVAL_S   = 5    # delay between dataset checks (seconds)\nMAX_POLL_ATTEMPTS = 60   # give up after ~5 minutes\nMAX_RETRIES       = 3    # for transient HTTP failures\n```\n\nThe shape of `SAMPLE_URLS` must match the **input schema** you defined in Scraper Studio. The default sample assumes a single `url` field. If your collector uses different inputs (for example, `keyword`, `zip_code`, `category`), update the dictionaries accordingly.\n\n---\n\n## How it works\n\n```text\n       +-----------------+      POST /dca/trigger      +-------------------+\n       |  Your script    | --------------------------\u003e |  Scraper Studio   |\n       |  (index.py)     | \u003c-- { collection_id } ----- |  Collector        |\n       +-----------------+                             +-------------------+\n                |                                                |\n                |  GET /dca/dataset?id=\u003csnapshot_id\u003e             |\n                |  (poll every 5s, retry 5xx with backoff)       |\n                |  \u003c--- [ { ...record... }, ... ] -------------- |\n                v\n       scraper_studio_results_\u003ctimestamp\u003e.json\n```\n\nThe script polls `/dca/dataset` every five seconds for up to five minutes. A non-empty JSON array is treated as a finished snapshot. Transient errors (5xx and network) are retried with exponential backoff (1s, 2s, 4s); 4xx errors fail immediately so you fix the request rather than retry it.\n\n---\n\n## Examples\n\n### Run with your own URLs\n\nReplace `SAMPLE_URLS` in [`index.py`](index.py):\n\n```python\nSAMPLE_URLS = [\n    {\"url\": \"https://example.com/product/1\"},\n    {\"url\": \"https://example.com/product/2\"},\n]\n```\n\n### Custom input schema\n\nIf your collector expects something other than `url`, pass whatever fields it defines:\n\n```python\ninputs = [\n    {\"keyword\": \"wireless headphones\", \"country\": \"US\"},\n    {\"keyword\": \"standing desk\",       \"country\": \"DE\"},\n]\nrun_scraper(inputs)\n```\n\n### Use as a library\n\n`run_scraper`, `trigger_with_url`, `trigger_with_urls` and `save_results` are top-level functions:\n\n```python\nfrom index import trigger_with_urls, save_results\n\ndata = trigger_with_urls([\n    \"https://example.com/page-1\",\n    \"https://example.com/page-2\",\n])\nsave_results(data, \"my_run.json\")\n```\n\n---\n\n## Output\n\n- Results are saved as JSON files named `scraper_studio_results_\u003cISO timestamp\u003e.json`.\n- The file contains the raw collector output: one record per input URL by default.\n\n### Sample console output\n\n```\nBright Data Scraper Studio\n==============================\nStarting Scraper Studio collector...\nQueueing 3 input(s)\nJob queued. Snapshot ID: j_abc123\nPolling for results...\nAttempt 1/60 - building\nAttempt 2/60 - building\nAttempt 3/60 - building\nResults downloaded.\nSaved to scraper_studio_results_2026-05-22T10-30-45-123456.json\n\nDone.\n```\n\n---\n\n## Security\n\nNever commit your `.env` file. The shipped `.gitignore` blocks `.env` and `.env.local`.\n\nIf you accidentally commit a real `BRIGHT_DATA_API_TOKEN`:\n\n1. Rotate the token immediately at [brightdata.com/cp/setting](https://brightdata.com/cp/setting).\n2. Use [`git filter-repo`](https://github.com/newren/git-filter-repo) or [BFG Repo-Cleaner](https://rtyley.github.io/bfg-repo-cleaner/) to remove the secret from history.\n3. Force-push and notify anyone who may have cloned the leak.\n\n---\n\n## Support\n\n- [Bright Data Help Center](https://brightdata.com/help)\n- [Scraper Studio quickstart](https://docs.brightdata.com/api-reference/scraper-studio-api/Getting_started_with_the_API)\n- [Contact support](https://brightdata.com/contact-us)\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrightdata%2Fbright-data-scraper-studio-python-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrightdata%2Fbright-data-scraper-studio-python-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrightdata%2Fbright-data-scraper-studio-python-project/lists"}