{"id":25201587,"url":"https://github.com/zfc-digital/cf-clearance-scraper","last_synced_at":"2025-10-23T21:27:04.765Z","repository":{"id":239673319,"uuid":"800214950","full_name":"ZFC-Digital/cf-clearance-scraper","owner":"ZFC-Digital","description":"This library was created for testing and training purposes to retrieve the page source of websites, create Cloudflare Turnstile tokens and create Cloudflare WAF sessions.","archived":false,"fork":false,"pushed_at":"2025-02-21T11:41:48.000Z","size":1886,"stargazers_count":446,"open_issues_count":6,"forks_count":92,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-01T10:06:45.616Z","etag":null,"topics":["bypass","cf-clearance","cloudflare","cloudflare-scrape","cloudflare-scraper","scraper-api"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZFC-Digital.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T23:10:24.000Z","updated_at":"2025-04-01T04:12:41.000Z","dependencies_parsed_at":"2024-05-14T00:52:06.615Z","dependency_job_id":"d0c72526-cd8f-411e-ac52-fc65418e15ef","html_url":"https://github.com/ZFC-Digital/cf-clearance-scraper","commit_stats":null,"previous_names":["zfcsoftware/cf-clearance-scraper","zfc-digital/cf-clearance-scraper"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFC-Digital%2Fcf-clearance-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFC-Digital%2Fcf-clearance-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFC-Digital%2Fcf-clearance-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFC-Digital%2Fcf-clearance-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZFC-Digital","download_url":"https://codeload.github.com/ZFC-Digital/cf-clearance-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247829490,"owners_count":21002995,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bypass","cf-clearance","cloudflare","cloudflare-scrape","cloudflare-scraper","scraper-api"],"created_at":"2025-02-10T05:17:11.759Z","updated_at":"2025-10-23T21:27:04.748Z","avatar_url":"https://github.com/ZFC-Digital.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!WARNING]\n\u003e This repo will no longer receive updates. Thank you to everyone who supported it.\n\n# CF Clearance Scraper\n\nThis library was created for testing and training purposes to retrieve the page source of websites, create Cloudflare Turnstile tokens and create Cloudflare WAF sessions.\n\nCloudflare protection not only checks cookies in the request. It also checks variables in the header. For this reason, it is recommended to use it with the sample code in this readme file.\n\nCookies with cf in the name belong to Cloudflare. You can find out what these cookies do and how long they are valid by **[Clicking Here](https://developers.cloudflare.com/fundamentals/reference/policies-compliances/cloudflare-cookies/)**.\n\n## Sponsor\n\n[![ScrapeDo](src/data/sdo.gif)](https://scrape.do/?utm_source=github\u0026utm_medium=repo_ccs)\n\n## Installation\n\nInstallation with Docker is recommended.\n\n**Docker**\n\nPlease make sure you have installed the latest image. If you get an error, try downloading the latest version by going to Docker Hub.\n\n```bash\nsudo docker rmi zfcsoftware/cf-clearance-scraper:latest --force\n```\n\n```bash\ndocker run -d -p 3000:3000 \\\n-e PORT=3000 \\\n-e browserLimit=20 \\\n-e timeOut=60000 \\\nzfcsoftware/cf-clearance-scraper:latest\n```\n\n**Github**\n\n```bash\ngit clone https://github.com/zfcsoftware/cf-clearance-scraper\ncd cf-clearance-scraper\nnpm install\nnpm run start\n```\n\n## Create Cloudflare WAF Session\n\nBy creating a session as in the example, you can send multiple requests to the same site without being blocked. Since sites may have TLS protection, it is recommended to send requests with the library in the example.\n\n```js\nconst initCycleTLS = require('cycletls');\nasync function test() {\n    const session = await fetch('http://localhost:3000/cf-clearance-scraper', {\n        method: 'POST',\n        headers: {\n            'Content-Type': 'application/json'\n        },\n        body: JSON.stringify({\n            url: 'https://nopecha.com/demo/cloudflare',\n            mode: \"waf-session\",\n            // proxy:{\n            //     host: '127.0.0.1',\n            //     port: 3000,\n            //     username: 'username',\n            //     password: 'password'\n            // }\n        })\n    }).then(res =\u003e res.json()).catch(err =\u003e { console.error(err); return null });\n\n    if (!session || session.code != 200) return console.error(session);\n\n    const cycleTLS = await initCycleTLS();\n    const response = await cycleTLS('https://nopecha.com/demo/cloudflare', {\n        body: '',\n        ja3: '772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,23-27-65037-43-51-45-16-11-13-17513-5-18-65281-0-10-35,25497-29-23-24,0', // https://scrapfly.io/web-scraping-tools/ja3-fingerprint\n        userAgent: session.headers[\"user-agent\"],\n        // proxy: 'http://username:password@hostname.com:443',\n        headers: {\n            ...session.headers,\n            cookie: session.cookies.map(cookie =\u003e `${cookie.name}=${cookie.value}`).join('; ')\n        }\n    }, 'get');\n\n    console.log(response.status);\n    cycleTLS.exit().catch(err =\u003e { });\n}\ntest()\n```\n\n## Create Turnstile Token with Little Resource Consumption\n\nThis endpoint allows you to generate tokens for a Cloudflare Turnstile Captcha. It blocks the request that fetches the page resource and instead makes the page resource a simple Turnstile render page. This allows you to generate tokens without having to load any additional css or js files. \n\nHowever, in this method, the siteKey variable must be sent to Turnstile along with the site to create the token. If this does not work, you can examine the token generation system by loading the full page resource described in the next section.\n\n```js\nfetch('http://localhost:3000/cf-clearance-scraper', {\n    method: 'POST',\n    headers: {\n        'Content-Type': 'application/json'\n    },\n    body: JSON.stringify({\n        url: 'https://turnstile.zeroclover.io/',\n        siteKey: \"0x4AAAAAAAEwzhD6pyKkgXC0\",\n        mode: \"turnstile-min\",\n        // proxy:{\n        //     host: '127.0.0.1',\n        //     port: 3000,\n        //     username: 'username',\n        //     password: 'password'\n        // }\n    })\n})\n    .then(res =\u003e res.json())\n    .then(console.log)\n    .catch(console.log);\n```\n\n## Creating Turnstile Token with Full Page Load\n\nThis example request goes to the page at the given url address with a real browser, resolves the Turnstile and returns you the token.\n\n```js\nfetch('http://localhost:3000/cf-clearance-scraper', {\n    method: 'POST',\n    headers: {\n        'Content-Type': 'application/json'\n    },\n    body: JSON.stringify({\n        url: 'https://turnstile.zeroclover.io/',\n        mode: \"turnstile-max\",\n        // proxy:{\n        //     host: '127.0.0.1',\n        //     port: 3000,\n        //     username: 'username',\n        //     password: 'password'\n        // }\n    })\n})\n    .then(res =\u003e res.json())\n    .then(console.log)\n    .catch(console.log);\n```\n\n## Getting Page Source from a Site Protected with Cloudflare WAF\n\nWith this request you can scrape the page source of a website protected with CF WAF.\n\n```js\nfetch('http://localhost:3000/cf-clearance-scraper', {\n    method: 'POST',\n    headers: {\n        'Content-Type': 'application/json'\n    },\n    body: JSON.stringify({\n        url: 'https://nopecha.com/demo/cloudflare',\n        mode: \"source\"\n        // proxy:{\n        //     host: '127.0.0.1',\n        //     port: 3000,\n        //     username: 'username',\n        //     password: 'password'\n        // }\n    })\n})\n    .then(res =\u003e res.json())\n    .then(console.log)\n    .catch(console.log);\n```\n\n## Quick Questions and Answers\n\n### Does It Open A New Browser On Every Request?\nNo, a new context is started with each request and closed when the job is finished. Processes are executed with isolated contexts through a single browser.\n\n### How Do I Limit the Browser Context to Open?\nYou can do this by changing the process.env.browserLimit value. The default is 20\n\n### How Do I Add Authentication to Api?\nYou can add authorisation by changing the process.env.authToken variable. If this variable is added, it returns 401 if the authToken variable in the request body is not equal to the token you specify.\n\n### How Do I Set The Timeout Time?\nYou can give the variable process.env.timeOut a value in milliseconds. The default is 60000.\n\n## Disclaimer of Liability\nThis repository was created purely for testing and training purposes. The user is responsible for any prohibited liability that may arise from its use.\nThe library is not intended to harm any site or company. The user is responsible for any damage that may arise. \nUsers of this repository are deemed to have accepted this disclaimer. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzfc-digital%2Fcf-clearance-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzfc-digital%2Fcf-clearance-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzfc-digital%2Fcf-clearance-scraper/lists"}