{"id":23071951,"url":"https://github.com/scrapingant/scrapingant-client-python","last_synced_at":"2025-09-23T15:31:26.942Z","repository":{"id":41287061,"uuid":"339340694","full_name":"ScrapingAnt/scrapingant-client-python","owner":"ScrapingAnt","description":"ScrapingAnt API client for Python.","archived":false,"fork":false,"pushed_at":"2024-07-16T19:14:41.000Z","size":62,"stargazers_count":36,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-07T19:11:58.464Z","etag":null,"topics":["crawler","scraper","scraping","scrapingant","scrapy","webscraping"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/scrapingant-client/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ScrapingAnt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-16T09:07:30.000Z","updated_at":"2024-11-16T10:18:17.000Z","dependencies_parsed_at":"2024-07-16T22:28:44.622Z","dependency_job_id":null,"html_url":"https://github.com/ScrapingAnt/scrapingant-client-python","commit_stats":{"total_commits":37,"total_committers":2,"mean_commits":18.5,"dds":0.05405405405405406,"last_synced_commit":"e4060e06f72e34747ac7773621436f95bf801932"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScrapingAnt%2Fscrapingant-client-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScrapingAnt%2Fscrapingant-client-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScrapingAnt%2Fscrapingant-client-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScrapingAnt%2Fscrapingant-client-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ScrapingAnt","download_url":"https://codeload.github.com/ScrapingAnt/scrapingant-client-python/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233984692,"owners_count":18761311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","scraper","scraping","scrapingant","scrapy","webscraping"],"created_at":"2024-12-16T07:18:19.618Z","updated_at":"2025-09-23T15:31:21.581Z","avatar_url":"https://github.com/ScrapingAnt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScrapingAnt API client for Python\n\n[![PyPI version](https://badge.fury.io/py/scrapingant-client.svg)](https://badge.fury.io/py/scrapingant-client)\n\n`scrapingant-client` is the official library to access [ScrapingAnt API](https://docs.scrapingant.com) from your Python\napplications. It provides useful features like parameters encoding to improve the ScrapingAnt usage experience. Requires\npython 3.6+.\n\n\u003c!-- toc --\u003e\n\n- [Quick Start](#quick-start)\n- [API token](#api-token)\n- [API Reference](#api-reference)\n- [Exceptions](#exceptions)\n- [Examples](#examples)\n- [Useful links](#useful-links)\n\n\u003c!-- tocstop --\u003e\n\n## Quick Start\n\n```python3\nfrom scrapingant_client import ScrapingAntClient\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n# Scrape the example.com site\nresult = client.general_request('https://example.com')\nprint(result.content)\n```\n\n## Install\n\n```shell\npip install scrapingant-client\n```\n\nIf you need async support:\n\n```shell\npip install scrapingant-client[async]\n```\n\n## API token\n\nIn order to get API token you'll need to register at [ScrapingAnt Service](https://app.scrapingant.com)\n\n## API Reference\n\nAll public classes, methods and their parameters can be inspected in this API reference.\n\n#### ScrapingAntClient(token)\n\nMain class of this library.\n\n| Param | Type                |\n|-------|---------------------|\n| token | \u003ccode\u003estring\u003c/code\u003e |\n\n* * *\n\n#### Common arguments\n - ScrapingAntClient.general_request\n - ScrapingAntClient.general_request_async\n - ScrapingAntClient.markdown_request\n - ScrapingAntClient.markdown_request_async\n\nhttps://docs.scrapingant.com/request-response-format#available-parameters\n\n| Param               | Type                                                                                                                       | Default    |\n|---------------------|----------------------------------------------------------------------------------------------------------------------------|------------|\n| url                 | \u003ccode\u003estring\u003c/code\u003e                                                                                                        |            |\n| method              | \u003ccode\u003estring\u003c/code\u003e                                                                                                        | GET        |\n| cookies             | \u003ccode\u003eList[Cookie]\u003c/code\u003e                                                                                                  | None       |\n| headers             | \u003ccode\u003eList[Dict[str, str]]\u003c/code\u003e                                                                                          | None       |\n| js_snippet          | \u003ccode\u003estring\u003c/code\u003e                                                                                                        | None       |\n| proxy_type          | \u003ccode\u003eProxyType\u003c/code\u003e                                                                                                     | datacenter | \n| proxy_country       | \u003ccode\u003estr\u003c/code\u003e                                                                                                           | None       | \n| wait_for_selector   | \u003ccode\u003estr\u003c/code\u003e                                                                                                           | None       |\n| browser             | \u003ccode\u003eboolean\u003c/code\u003e                                                                                                       | True       |\n| return_page_source  | \u003ccode\u003eboolean\u003c/code\u003e                                                                                                       | False      |\n| data                | same as [requests param 'data'](https://requests.readthedocs.io/en/latest/user/quickstart/#more-complicated-post-requests) | None       |\n| json                | same as [requests param 'json'](https://requests.readthedocs.io/en/latest/user/quickstart/#more-complicated-post-requests) | None       |\n\n**IMPORTANT NOTE:** \u003ccode\u003ejs_snippet\u003c/code\u003e will be encoded to Base64 automatically by the ScrapingAnt client library.\n\n* * *\n\n#### Cookie\n\nClass defining cookie. Currently it supports only name and value\n\n| Param |  Type               | \n|-------|---------------------|\n| name  | \u003ccode\u003estring\u003c/code\u003e | \n| value | \u003ccode\u003estring\u003c/code\u003e |\n\n* * *\n\n#### Response\n\nClass defining response from API.\n| Param       | Type                       |\n|-------------|----------------------------|\n| content     | \u003ccode\u003estring\u003c/code\u003e        |\n| cookies     | \u003ccode\u003eList[Cookie]\u003c/code\u003e  |\n| status_code | \u003ccode\u003eint\u003c/code\u003e           |\n| text        | \u003ccode\u003estring\u003c/code\u003e        |\n\n## Exceptions\n\n`ScrapingantClientException` is base Exception class, used for all errors.\n\n| Exception                            | Reason                                                                                                                       |\n|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------|\n| ScrapingantInvalidTokenException     | The API token is wrong or you have exceeded the API calls request limit                                                      |\n| ScrapingantInvalidInputException     | Invalid value provided. Please, look into error message for more info                                                        |\n| ScrapingantInternalException         | Something went wrong with the server side code. Try again later or contact ScrapingAnt support                               |\n| ScrapingantSiteNotReachableException | The requested URL is not reachable. Please, check it locally                                                                 |\n| ScrapingantDetectedException         | The anti-bot detection system has detected the request. Please, retry or change the request settings.                        |\n| ScrapingantTimeoutException          | Got timeout while communicating with Scrapingant servers. Check your network connection. Please try later or contact support |\n\n* * *\n\n## Examples\n\n### Sending custom cookies\n\n```python3\nfrom scrapingant_client import ScrapingAntClient\nfrom scrapingant_client import Cookie\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\nresult = client.general_request(\n    'https://httpbin.org/cookies',\n    cookies=[\n        Cookie(name='cookieName1', value='cookieVal1'),\n        Cookie(name='cookieName2', value='cookieVal2'),\n    ]\n)\nprint(result.content)\n# Response cookies is a list of Cookie objects\n# They can be used in next requests\nresponse_cookies = result.cookies \n```\n\n### Executing custom JS snippet\n\n```python\nfrom scrapingant_client import ScrapingAntClient\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\ncustomJsSnippet = \"\"\"\nvar str = 'Hello, world!';\nvar htmlElement = document.getElementsByTagName('html')[0];\nhtmlElement.innerHTML = str;\n\"\"\"\nresult = client.general_request(\n    'https://example.com',\n    js_snippet=customJsSnippet,\n)\nprint(result.content)\n```\n\n### Exception handling and retries\n\n```python\nfrom scrapingant_client import ScrapingAntClient, ScrapingantClientException, ScrapingantInvalidInputException\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\nRETRIES_COUNT = 3\n\n\ndef parse_html(html: str):\n    ...  # Implement your data extraction here\n\n\nparsed_data = None\nfor retry_number in range(RETRIES_COUNT):\n    try:\n        scrapingant_response = client.general_request(\n            'https://example.com',\n        )\n    except ScrapingantInvalidInputException as e:\n        print(f'Got invalid input exception: {{repr(e)}}')\n        break  # We are not retrying if request params are not valid\n    except ScrapingantClientException as e:\n        print(f'Got ScrapingAnt exception {repr(e)}')\n    except Exception as e:\n        print(f'Got unexpected exception {repr(e)}')  # please report this kind of exceptions by creating a new issue\n    else:\n        try:\n            parsed_data = parse_html(scrapingant_response.content)\n            break  # Data is parsed successfully, so we dont need to retry\n        except Exception as e:\n            print(f'Got exception while parsing data {repr(e)}')\n\nif parsed_data is None:\n    print(f'Failed to retrieve and parse data after {RETRIES_COUNT} tries')\n    # Can sleep and retry later, or stop the script execution, and research the reason \nelse:\n    print(f'Successfully parsed data: {parsed_data}')\n```\n\n### Sending custom headers\n\n```python3\nfrom scrapingant_client import ScrapingAntClient\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\nresult = client.general_request(\n    'https://httpbin.org/headers',\n    headers={\n        'test-header': 'test-value'\n    }\n)\nprint(result.content)\n\n# Http basic auth example\nresult = client.general_request(\n    'https://jigsaw.w3.org/HTTP/Basic/',\n    headers={'Authorization': 'Basic Z3Vlc3Q6Z3Vlc3Q='}\n)\nprint(result.content)\n```\n\n### Simple async example\n\n```python3\nimport asyncio\n\nfrom scrapingant_client import ScrapingAntClient\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\n\nasync def main():\n    # Scrape the example.com site\n    result = await client.general_request_async('https://example.com')\n    print(result.content)\n\n\nasyncio.run(main())\n```\n\n### Sending POST request\n\n```python3\nfrom scrapingant_client import ScrapingAntClient\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\n# Sending POST request with json data\nresult = client.general_request(\n    url=\"https://httpbin.org/post\",\n    method=\"POST\",\n    json={\"test\": \"test\"},\n)\nprint(result.content)\n\n# Sending POST request with bytes data\nresult = client.general_request(\n    url=\"https://httpbin.org/post\",\n    method=\"POST\",\n    data=b'test_bytes',\n)\nprint(result.content)\n```\n\n### Receiving markdown\n\n```python3\nfrom scrapingant_client import ScrapingAntClient\n\nclient = ScrapingAntClient(token='\u003cYOUR-SCRAPINGANT-API-TOKEN\u003e')\n\n# Sending POST request with json data\nresult = client.markdown_request(\n    url=\"https://example.com\",\n)\nprint(result.markdown) \n```\n\n## Useful links\n\n- [Scrapingant API doumentation](https://docs.scrapingant.com)\n- [Scrapingant JS Client](https://github.com/scrapingant/scrapingant-client-js)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapingant%2Fscrapingant-client-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrapingant%2Fscrapingant-client-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapingant%2Fscrapingant-client-python/lists"}