Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ulixee/secret-agent
The web scraper that's nearly impossible to block - now called @ulixee/hero
https://github.com/ulixee/secret-agent
automated browser chromium devtools mitm mitmproxy playwright proxy puppeteer scraping secretagent stealth
Last synced: about 1 month ago
JSON representation
The web scraper that's nearly impossible to block - now called @ulixee/hero
- Host: GitHub
- URL: https://github.com/ulixee/secret-agent
- Owner: ulixee
- License: mit
- Created: 2020-06-24T18:21:38.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-03-07T03:00:38.000Z (over 1 year ago)
- Last Synced: 2024-10-09T16:55:25.641Z (about 1 month ago)
- Topics: automated, browser, chromium, devtools, mitm, mitmproxy, playwright, proxy, puppeteer, scraping, secretagent, stealth
- Language: TypeScript
- Homepage: https://secretagent.dev
- Size: 46.3 MB
- Stars: 669
- Watchers: 18
- Forks: 44
- Open Issues: 44
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
- License: LICENSE.md
Awesome Lists containing this project
README
## SecretAgent 2.0 === Hero
> :bell: SecretAgent 2.0 is named "[Hero](https://github.com/ulixee/hero)" and currently in alpha testing. We're ready for developers to begin to switch over - it's a pretty easy transition ([migration guide](https://ulixee.org/docs/hero/help/migration-guide)). Follow along with development of Hero [here](https://github.com/ulixee/hero) or check out the latest npm packages `@ulixee/hero-playground`.
---
# SecretAgent
SecretAgent is a web browser that's built for scraping.
- [x] **Built for scraping** - it's the first modern headless browsers designed specifically for scraping instead of just automated testing.
- [x] **Designed for web developers** - We've recreated a fully compliant DOM directly in NodeJS allowing you bypass the headaches of previous scraper tools.
- [x] **Powered by Chrome** - The powerful Chrome engine sits under the hood, allowing for lightning fast rendering.
- [x] **Emulates any modern browser** - BrowserEmulators make it easy to disguise your script as practically any browser.
- [x] **Avoids detection along the entire stack** - Don't be blocked because of TLS fingerprints in your networking stack.
Check out our [website for more details](https://secretagent.dev).
## Installation
```shell script
npm i --save secret-agent
```
or
```shell script
yarn add secret-agent
```
## Usage
SecretAgent provides access to the W3C DOM specification without the need for Puppeteer's complicated evaluate callbacks and multi-context switching:
```js
const agent = require('secret-agent');
(async () => {
await agent.goto('https://example.org');
const title = await agent.document.title;
const intro = await agent.document.querySelector('p').textContent;
await agent.close();
})();
```
Browse the [full API docs](https://secretagent.dev/docs).
## Contributing
We'd love your help in making SecretAgent a better tool. Please don't hesitate to send a pull request.
## License
[MIT](LICENSE.md)